Eigenform dynamic benchmark: Ground Truth: Dynamic Benchmarking in Highly Specific Fields
Release notes: Release notes
How do you benchmark model performance when every deployment has its own corpus?
Obviously if you’re interested in getting a model to do maths you bencmark on IMO results, if you want it to learn to code you use HumanEval, and so on and so forth? But if you want to test its ability to navigate geological reports from a specific region of Western Australia – to pick an example out of the air – that’s much more difficult.
Sure, you can track its general geological skills, but that’s not very satisfying if you’re working on a continual fine-tuning project aiming to get it to expert status on this one specific topic.
Most benchmarks measure capability but we’re measuring knowledge acquisition, which is a much more fluid entity. Capability asks, “What can this model do?” Knowledge acquisition asks, “What did this model learn after reading this corpus?” Less “what are sedimentary rock” than “explain everything you have learnt about sedimentary rock formation”. A high schooler’s answer might be two sentences, a geology professor’s could be 20 pages. A good knowledge acquisition benchmark should be able to measure progress along this route rather than mere correctness.
This is where many current benchmarks fall down. Current benchmarks tell us whether Claude or Qwen knows geology in general. They do not tell us whether a local model has actually learned anything from our specific geological reports. And it’s not a pure epistemological nicety. A mining company doesn’t care whether the model knows about gold pathfinders sub speciae aeterinitatis. They care whether it learned them in Cue, WA.
Our solution to this is dynamic benchmarking. What exactly does this mean in practice?
Imagine training on:
- one exploration licence
- one oil basin
- one state’s geological survey
- one mine’s historical reports
You cannot realistically hand-author a benchmark for every new corpus. So instead of creating benchmarks you need to create a benchmark generator: your local show pony model acts as the candidate while a panel of frontier models with web and corpus access acts as the exam board: identifying what should be tested, writing questions, constructing grading rubrics, and only then evaluating responses.
It is important to emphasise that this is not the LLM-as-a-judge paradigm. Exam boards are not smarter than every candidate to face them, and they are structured in such a way as to account for the fact. They have better information access, an awareness of the abstractions being tested, and an ability to operate on a consensus basis, and they leverage that to set universal standards even for candidates who may have a far higher IQ than any of their members.
In our case we wanted to track a model’s ability to explore a data corpus, discover knowledge about a specific topic – in our case the Cue Victory gold field in Western Australia – and integrate that knowledge into its own weights via continual learning.
This is tricky to assess. All models have a certain level of knowledge of this region just by virtue of the fact that mining companies exploring in the area have been publishing reports on it for years and some of this information has got into the pre-training data. On the other hand, a skilled-but-naive data analysis model would be able to read through the corpus data and outperform a basic pre-trained model without corpus access.
In other words, there are several different things being measured here:
- Innate, weight-level knowledge of the Cue Victory site.
- Data analysis skills.
- Ability to navigate the Cue Victory dataset in particular (effectively a combination of the other two).
General benchmarks often conflate prior knowledge, reasoning ability and adaptation. For continual learning, we care specifically about the third. To accurately track our models’ improvement (or decline) we developed a dynamic benchmarking system that we are now open-sourcing. The basics of this system can be found in the skill.md document published to our Github, but the tldr is as follows:
- Read the corpus.
- Build an internal understanding.
- Identify concepts worth testing.
- Generate questions.
- Generate reference answers.
- Generate grading rubrics.
- Verify every answer against source evidence.
- Hold out the benchmark.
Obviously it’s not always that simple – you need to develop a relatively sophisticated and coherent schema to assess a “vague but basically ok” answer against a “detailed but with a couple of flubs” answer. You can check out the Github to see how we did this in practice.
You can then apply whatever scoring metric you like to the results as long as it’s reasonably rigorous. We like to compare finetuned models pairwise against the foundation model and see which one the panel of examiners prefers, but ymmv.
We’re releasing this as a geology benchmark because that’s the domain we work in. But we don’t think the underlying idea is geological. Any field with a specialist corpus should be able to generate an examination tailored to what the model was actually supposed to learn.
MMLU is an asset. GPQA is an asset. Humanity’s Last Exam is an asset. People build them once and then use them until question-answer pairs get into the training data and are replaced. We’re arguing for a formalisation of this process: not lists of questions but reliable pipelines for producing lists of questions.
The process gets stronger the more people adapt it for their own domains. Make your own here.