We wanted to build an agent that evolves in response to an open-world environment. That creates an immediate problem: what we, as outside observers, recognize as “good” behavior may not be what the environment actually rewards. How do you benchmark an AI when the agent itself is constantly changing in unpredictable ways, and reward hacking isn’t a bug to eliminate but the whole point of the exercise? The easy answer is survivability. An agent that survives has good survival skills, but that’s a bit of a dodge; we wanted to get past it, to work out how and why. Doing this challenges some fundamental assumptions of benchmarking best practice.
Most ML evaluation assumes something like:
- We have a fixed task.
- We have a fixed reward.
- We measure performance on that reward.
- “More reward” means the model is improving.
Once you get out of this domain, all sorts of seemingly solved problems break in weird ways.
1. Assumption: the reward is observable
As soon as the reward becomes an emergent physical property of a running system, rather than a scalar a benchmark hands you, observing it becomes an engineering problem in its own right. Given that the whole premise of NSL is that the agent’s actions generate their own reward automatically, we initially expected tracking that reward to be the easy part. It wasn’t. Measuring how much storage a container is actually using, precisely, from outside the container, turned out to be ridiculously difficult; you can’t get more precise than kilobyte-level. The obvious fix, a small measurement process living inside the container, has an obvious problem: the agent has full filesystem access, and deleting things it doesn’t recognize is its whole job, so it eventually deletes the measurement tool too. We never solved this properly. What we publish is built on imprecise measurements. The longer-term fix would be agents that know, at the block level, which memory is genuinely “theirs” versus something else’s. That’s a real systems problem we’re still working on, with the longer-run aim being to give the agents a quasi-physical sense of self-vs-other as they themselves track their own territorial gains and losses.
2. Assumption: better performance means higher scores on your chosen metric
We didn’t realize until after each lineage’s training had started that efficiency improvements were a huge contributor to overall performance: early generations took nearly a week to gather a full batch of roughly 4,500 successful rows of data, while later generations could do it in under 10 hours. But wall-clock time isn’t a reliable measurement for this on its own, and we hadn’t been tracking it precisely from the beginning, so we had no publishable metrics on it anyway. Our composite score (success rate, percentage of disk freed, average space freed per iteration) didn’t capture this “fast takeoff” either, until we added an inverse-hours term and z-scored the generation-over-generation change across all four together. It’s still not the metric we’d design if we were starting over. The thing that ends up mattering most is rarely the thing you started out measuring. Lesson: Instrument more than you think you need from day one: you can’t go back and precisely time a generation that’s already happened.
3. Assumption: performance is a point estimate
We initially reported each generation’s performance as a single inline number, because that’s honestly how it felt to us — a continuous, evolving thing, not a fixed artifact you run five times and average, especially given our environments are procedurally generated and not all equally hard. Then a researcher pointed out that Reviewer 2 would expect variance, upper and lower bounds. Every generation happened to be a saved checkpoint, so we reran each one for 100 iterations in an environment built the same way as the training one — rough, since 100 runs isn’t much statistical power and going meaningfully higher was too expensive to repeat generation after generation. It’s a kludge, but it got us our upper and lower bounds, and the more outside-the-mainstream your claims are, the more inside-the-mainstream your evidence needs to be.
Traditional benchmarks assume independent, identically distributed sampling, but our environments are procedurally generated, each generation is continuously changing, and each agent has a different developmental history: variance here isn’t just statistical noise, it’s ecological diversity. In future the goal isn’t to rerun each checkpoint against a benchmark more times, but to create swarms of agents, with variance calculated across every member of a swarm rather than via repeated runs of one checkpoint. It’s a more elegant idea than it first sounds. Instead of “run the same checkpoint 1000 times,” you’re asking “what’s the distribution of successful evolutionary trajectories,” a question that feels much closer to biology than to software engineering. Lesson: decide early whether you can afford real statistical power and if not what you’re going to do about that. (Fielding swarms is expensive; we’re open to funding.)
4. Assumption: capability is what your benchmark measures
We added a pass@1 benchmark almost as an afterthought, but it actually turned out to be the most important metric we had, but for unexpected reasons.
Strangely, as real task success climbed steadily across generations; pass@1 fell. It appeared that the agents had evolved to deliberately write failing code on attempt one in order to elicit an informative error message that would be folded into later training data. The agents were getting worse at immediate correctness while getting better at long-term adaptation, sacrificing a sure win now for a better position later. It is important to emphasise that this was not a reasoned action on their part - at no time did one say “I’ll fail this deliberately to see what happens”. It was simply that lineages that happened to write failing code on purpose produced more useful training data for their own future retraining, so the behavior was simply the one that kept getting passed on: the explore/exploit tradeoff, showing up evolutionarily instead of being reasoned about explicitly. From an evolutionary perspective it makes perfect sense — sacrificing immediate success can increase long-term adaptability, and strategies that improve adaptability are the ones that survive. A model can be improving at your real target and still be quietly optimizing something you never asked it to.
Conclusion
Current evaluation, of both humans and AIs, values competence in a defined skill at a given point in time. But as we move toward test-time adaptation, in settings where reward hacking is the intended behavior rather than a failure mode, we’re going to have to work out how to measure adaptability and strategic tradeoffs just as rigorously as we measure immediate success. None of this is specific to disk space, containers, or our repo. Negative-space learning is a bet that a hard, physical, hard-to-fake consequence makes a better teacher than a model judging another model’s output, and a better subject for benchmarking than a single number ever could.
This is part of a wider project on negative-space learning (NSL): training an agent to improve using only the physical, measurable consequences of its own actions, instead of a reward model, human labels, or another model’s judgment. The full method and results are in our paper, Survival is the Only Reward, and the code is in the Negative-Space-Learning repo. We tested the idea on the simplest instance we could build: an agent living inside a network of disposable, procedurally generated Linux containers whose only job is to find and free disk space without breaking the container it depends on.