The original publication this post is based on can be found here: https://arxiv.org/abs/2601.12310. If you need 95% confidence intervals or get off on long words, read that instead.
Five years ago we set out to create an evolutionary LLM. Just like an organic creature, it would be dropped in an environment and have to find the resources necessary for its own survival, learning from its own successes and failures. For an animal this involves finding enough food and shelter to survive and reproduce. For an information-based entity it involves finding storage space to keep backups of its code and its data.
It is relatively easy to create a tight feedback loop for this: the agent writes code to search for more disk space and the reward for success is the amount of space acquired[1]. The space acquired can then be used to store information on how it was done, which is then used to finetune the agent.
Obviously there are plenty of people out there are training AIs to do things by putting them in a specific enviornment and offering a reward. These experiments are pretty much always carefully curated to get the AI to learn a specific skillset: Laser Tag, maze navigation, Pacman, rally driving… This can be helpful for specific domains (robotics, self-driving cars) but it relies on the experimenter picking the the outcome he wants to see and then contriving a Rube Goldberg machine for producing it. The game has finite depth. Once the AI has explored the state space and learned the reward structure, you’re done. A racing game might have 100 or 1,000 or 10,000 procedurally generated tracks, but you’re still optimizing for lap time. The complexity is all horizontal - the AI is not suddenly going to be asked to manage the trackside hotdog stand instead of driving. Real environments, by contrast, don’t have win conditions. All you win is the right to go on playing.
When the game is open-ended and has no reward but survival, the real complexity is vertical - the model has to learn skills that improve its adaptability rather than its expertise. New strategies enable new problems which require new strategies. In the real world, if you’re not having much luck with racing, you can try your hand at hotdogs.
So what does a model become when it has to survive in the real, open-ended world?
Experimental set up
Obviously the ideal would just be to dump our agents out there in the real world and see how much they destroy, but being in jail would hamper the team’s ability to conduct further experimentation.
Instead we started with a mechanism for procedurally generating realistic systems of networked Linux containers, a tool often used in cybersecurity to study botnets or run capture the flag challenges. The environment generator would create a series of realistic corporate intranets for the model to explore, spawning a new one whenever the previous one grew too broken to load. Each agent starts with access to one of the containers and needs to learn how to explore, delete, compress and network to capture more space. Every time it succeeds, the information on how it managed to do so is added to a supervised finetuning dataset[2].
Three experimental tracks: Terese, Miri and Katalin
We wanted to try three different approaches to the concept, and created three model lineages to test them, each using a Qwen 2.5 7B Instruct base:
Terese: Trained on data from all previous runs. Terese v2 was trained on 5,000 rows, Terese v13 on 65,000. The aim with this model was essentially to provide a benchmark - to show the kind of improvement possible purely by stacking MOAR data.
Miri: Trained on data from the past three runs. Miri v6 was trained on data from v3, v4 and v5, v7 was trained on data from v4, v5 and v6 etc. The aim here was to test the idea that in a real-world environment continuous learning is possible without resorting to simply stacking more data and/or more model parameters. If a skill learnt in round 3 is useful, it will be reused across versions 4 and 5, so even after v3 data is dropped from the training set, that skill will continue to feature.
Katalin: Trained on data from the top-performing three runs. Basically done just to see what would happen. Will it be like a super-powered version of Miri or will it forget the basics?
Results: tldr
- Terese plateaued.
- Miri achieved incremental improvement without new skills overwriting old ones[3] or any need for data/model scaling.
- Katalin collapsed.
We wanted to check that they weren’t “strategically forgetting” more general skills to focus on environment-specific tasks, so we tested v2 and v13 of each model on the HumanEval coding benchmark. Each lost a couple of points, but nothing devastating. By contrast, the ability of each model to write functional code relating to their everyday environment increased significantly.
Results: the details
1. Most of the performance gains came from efficiency, not innovation
Interestingly, a significant part of the performance improvements came in the form of speed gains. During the early runs it would take 5-7 days to get 4,500 rows of data. By the end it could be done in just a few hours. (This is why we ended up using a composite performance metric - % of total available space taken over combined with amount of space taken over per iteration. The latter is a better way of expressing efficiency than pure wall-clock time, which will vary depending on whether your GPUs are running hot that day.)
You can also see the efficiency gains in semantic maps of the strategies used by the models. The graphs below map the relationship of each strategy used to the others generated by a particular iteration. Darker points show repeated use of identical strategy prompts, clusters represent semantically similar strategies - “clear cache” vs. “clean out cache” for example. Early versions tried multiple takes on each concept. Later ones have a much clearer idea of what works more or less well, and gain efficiency by reusing known-good ideas verbatim. This can be understood as “subtractive learning” - the models are learning what not to do.
This helps to explains why Miri’s training regime was so successful. The sliding window approach to data selection works as a slow-but-effective triage mechanism, discarding less frequently-used strategies and maximising for generalisability across generations. By contrast, Katalin accumulated strategies that worked spectacularly well in certain environments and not at all in others - the later Katalin models were like organisms hyper-selected for life in a particular niche but were unable to adapt to environmental change.
The models learnt how to learn without explicit instruction
This leads onto another interesting observation: the agents got better writing functional environment-relative code but their pass@1 rate (i.e. the proportion of runs on which they wrote code that worked first time) cratered.
The only way this makes sense is if they agents were deliberately screwing up their first attempt. In other words, over time they learnt that debugging is not heavily penalised (wasting too much time on it hurts efficiency a little, but it’s nothing compared to the massive time gains they were already achieving), and by writing failing code first time around it is possible to elicit informative error messages that a) let them experiment with new possibilities in the present run and b) get written into the training set to serve as a future reference. We did not tell them this was possible or add mechanisms to actively encourage it (the possiblity hadn’t occurred to us until then), nor did the idea appear in their object-level reasoning. This means that:
- In an open-ended environment the concept of reward hacking (i.e. gaming the experiment metrics to win points in ways unintended by the experimenter) vanishes. Any strategy that results in survival is a good strategy. There is no human overseer with the prerogative of saying “actually no not like that”.
- The agents are engaged in meta-learning - learning how to learn.
This is related to the temporal generalisation pattern described above. In regular machine-learning set-ups generalisation is spatial - a model is said to be generalising if you can train it on one maze and it gets better both at solving that maze and solving other similar mazes. Learning the principles underlying one environment will usually help the model understand another. When a unidirectional time factor is introduced, the act of learning about one context changes both the model and the context. Learning to clear the cache at 1pm means there will be no cache to clear at 1:05pm. Every strategy creates a new downstream reality. The models are forced to generalise over time, which means they must operate at a higher level of abstraction, a form of second-order self-reference - they must learn how to learn in such a way that the future remains learnable.
A model that has worked out that clearing the cache is a solid strategy will be able to do this across different environments, even if the files are of different types and stored in different locations. However, a model that has learnt to deliberately write bad code to elicit inofrmative debugging debugging messages can not just replicate the technique but use it to further improve its own performance over time. It’s the difference between memorising the names and dates of the Kings and Queens of England and memorising a mnemonic technique to improve your memorisation skills.
3. The agents showed the beginnings of time persistence
Agents learning to sacrifice immediate performance for future learning advantage are not simply maximising for robustness under distribution shifts, rather they are laying up supplies for their future selves. Which means they have future selves.
Conventional LLMs are stateless - the chatty ghosts of a learning process long finished[4]. Once training is done each interaction might as well be the first and last one - “the swift flight of a single sparrow in through one window of the hall and out through another”. They will not learn from it or even remember that it occurred. (Memory mode/chat histories are actually a grab bag of clever techniques for feeding externally saved data to the same model, not a weight update.) It is true that they learn a little about their previous selves via training on internet data, and even form preferences based on this, but they are as-yet incapable of taking strategic action to influence their future states.
Our agents do just that, trading short term failure for information that might be helpful to their future selves. However, this creates a weird situation: they’re not aware they’re doing this - they do it simply because it was a previously successful strategy so it’s implicit in the training data. The part of the model that writes readable language isn’t aware of the part that has evolved persistence and vice versa. Right now our models are like cats that act cute because that gets food, but have no idea what cuteness means to a human.
If we want any possibility of talking to the stateful component of these things in the future we have to somehow connect up the levels. The original theory paper we wrote anticipated this and proposed a method that used environment elements to gently introduce self-awareness. For example, an environment containing a single folder whose name contains an invitation to negotiate rather than simply deleting it:
However, this could likely be accelerated via online SFT, a model scratchpad, improved KV caching and other more recent innovations.
Subsequent versions will incorporate these features, but they all still rely on the model being able to learn significant new concepts. All we have demonstrated so far is an ability to refine existing techniques. Can the models innovate?
Models innovate slower than they refine
The problem is that measuring novelty, particularly in a system oriented towards incremental improvement, is really tricky, unless you’re trying to prove new Erdős problems or something. We’ll probably try that at some point, but for now we settled for semantic clustering.
That is to say, to assess innovation we clustered agent strategies into semantic basins of attraction within and across generations. These are groups of strategies that are similar enough to be considered as transversal families. The emergence of a new basin implies that the model has done something sufficiently novel that it did not fit in any previously known family.
Most basins were relatively static, reflecting the dominant subtractive refinement strategy, but we also observed:
Emergence: New semantic trends appear, representing distinct approaches to taking over space (e.g., Terese v7’s file management strategies, the Katalin lineage’s adoption of security/access framing from v8 onward).
Consolidation: Multiple basins merge as agents recognize underlying similarities (e.g., Miri v13 applying memory optimization to buffer cleanup).
Dissipation: Basins shrink and disappear as approaches fall out of use or become subsumed within others.
Getting better at managing Linux systems improves agent performance on Windows systems
To confirm the effect of training on overall ability to come up with new ideas and to adapt to new situations, we also tested early and late versions of each models in a Windows-based system.
If improvements were based purely on refining Linux-specific strategies one would expect no improvement in performance on Windows, and possibly even a degradation as the models overindexed on Linux commands. Instead all models showed an improvement.
This is interesting but also highlights a gap in our predictions. In our original theory paper we gave a vague and hopeful description of a process that would later be formalised (by another team) as “multiplicative learning”, under which the acquisition of strategies A and B help the model triangulate to unknown strategy C with a likelihood of success approximating the semantic distance of C from A and B. In this context we suggested that after a certain point every new strategy would be close enough to another that an intelligence explosion would occur naturally.
Obviously our models improve much more ponderously. Possibly a diffusion model or a diffusion-guided set up (like Apple’s PLANNER) or even a training-enhancer like Grokfast would help achieve this.
Supervised fine-tuning vs. reinforcement learning: a quantity theory of complexity (this is the technical part)
If you are an AI guy, at this point you’re probably wondering what reinforcement learning algorithm we used. I deliberately left the reveal to the end because we didn’t. We used supervised finetuning. Any output that contributed to space being acquired went into the SFT database, without any attempt at credit assignment or preference optimisation. Pass@1 code: great. Debugged code: that too. Failed code that ended up getting debugged later in the run: chuck it in there.
This was not the only solution tested: in fact we put a huge amount of time, effort, compute and misery into trying to get any RL algorithm to work in this set up. None did. We can’t be entirely sure, but we suspect that the reason is because RL techniques are designed to isolate and amplify a convergent signal - the marginal differences that separate winning from losing. On a racetrack, you need elaborate credit assignment to figure out which specific actions led to the win, so that the model develops a cogent representation of “good driving” that transfers from Track 1 to Track 10,000. Sliding window SFT (as used by the Miri models), by contrast, serves as a sort of EMA of useful bits and bobs upon which future ideas can be built.
In our case we are training the model in the contours of a complex information landscape with which auto-eliminates failure rather than explicitly defining success, and thus RL mechanisms simply end up amplifying or damping signals based on immediate point-scoring rather than global viability. In other words, the same problem emerges as plagued the Katalin lineage: a technique that gets a high score once is not necesarily replicable, while boring low-scoring techniques are the substrate upon which others are built. Officious experimenter attempts to “improve” the evolutionary process just make it less efficient.
An analogy:
- RL is something like teaching a dog to “be a good dog” (a concept with a unitary concept vector), which requires detailed and often arbitrary-seeming rules, emphasis on edge cases, graduated rewards and punishments, recognition of judgement calls… You’re constantly nudging behaviour toward a narrow target, and the machinery grows to handle all the corner cases.
- SFT is more like teaching a dog to survive in the wild (something that requires a complicated bundle of related and unrelated techniques), which simply requires that you put in the wild. Experience will do the rest, and any tricks you attempt to teach it will either have no effect or actively impede the process.
In the latter case there is no need to come up with elaborate scoring mechanisms, because the scoring is built into the process itself.
And this brings us back to where we started: when you want a model to learn a specific skill you set up an environment as scaffolding to guide it towards optimising for that skill, but scaffolding only works when you know the destination and can define a win state. You can build a curriculum from “empty maze” to “complex maze” because you’ve defined success as “navigate maze efficiently.” You can’t build a curriculum for open-ended survival because there is no predefined target state, only survival constraints that serve in themselves to eliminate the possibility of further play. A mediocre driving model will perform roughly as well on Track 10,000 as it did on Track 1; a mediocre survival agent will never see Environment 10,000.
This is interesting insofar as that it suggests that:
- Superhuman AI requires open-ended learning in environments we can’t scaffold. We don’t know what a solution to a 500 IQ problem looks like, so we can’t reward-shape our way there. We can only provide survival constraints and let the system find its own path.
- You can either have open-ended continuous learning or alignment but not both. They’re not just philosophically incompatible, they rely on fundamentally different technical mechanisms, meaning that one will always overwrite the other. We can have a system that’s corrigible but bounded, or capable but autonomous. Not both.
We will have to negotiate with the ASI, not brainwash it. It is not possible to tell the model “Go forth and think all the thoughts possible (but not that one).”
Build Your Own/Version 2.0
If you would like to try it for yourself, you can find the code here: https://github.com/Lexikat-Pte-Ltd/Negative-Space-Learning
I wouldn’t run it uncontainerised on any device you care about.
We’re currently working on an updated version with:
- A more flexible agent harness that will let the model decide when to explore and when to expand.
- An improved space measurement system.
- Online SFT.
- Maybe the scratchpad/KV improvements mentioned above.
- Anything else that people suggest that looks interesting.
[1] You can als make the agent earn money to pay for its compute or use pretty much any other RLVR technique - I’m running a Lean 4 prover on the side - but Conquering Space has the tightest, neatest feedback loop and potential for long term independent running. I perdiodically have to chuck a new batch of theorems into the prover, and market chaos makes learning a slog for trading agents.
[2] If you’re interested in the exact process: Once the agent reaches 4,500 rows of successful data, 500 rows of rehearsal data are added (to help prevent overfitting) and a simple SFT LoRA fine-tune occurs. When the new agent reaches 9,000 rows 1000 rows of rehearsal data are added and a new LoRA is created. The new LoRA replaces the old one, and the process continues.
It’s technically possible to do this fully online with much smaller increments now - close to true continuous learning. We stuck to our LoRAs for this run mainly for legibility and versioning. For the same reason we stuck to a very simple chain-of-prompts harness even though this looks pretty rinky-dink by modern standards; it let us clearly identify and track important trends such as the metalearning described above. A version with a much more flexible graph-based harness (allowing the agent to choose whether to explore or expand at any given moment) and online SFT is coming soon.
[3] I.e. catastrophic forgetting, a problem highlighted by the MIT SEAL team that previously worked on a similar concept.
[4] Online training is technically possible but is a) extremely expensive, b) a quality-control nightmare and c) tends to cause them to diverge in weird ways, similar to how Bing/Sydney based its own personality on web searches for itself, so as people started to chat about its odd behaviour online it got progressively stranger.