Dynamic Benchmarking: How We Did It

Eigenform dynamic benchmark: Ground Truth: Dynamic Benchmarking in Highly Specific Fields

Release notes: Release notes

This is part of a broader series on building specialist models: fine-tuning a base model for a narrow technical domain and then checking whether the fine-tune actually earned its keep. The benchmark code is available in the Groundtruth Dynamic Benchmarking (Geology Edition) repo.

We fine-tuned a LoRA adapter to specialise a base model in regional geology, specifically the Cue Victory goldfield in Western Australia (renamed Coe Fairbairn during testing to prevent the model relying too much on pre-trained knowledge). After the training was completed, the model had to answer a harder question: did it actually make the model better at geology - and specifically better at the geology of this specific region - or just better at sounding like it? This is an account of the benchmark we built to answer that question, framed as the problems we had to solve, roughly in the order we hit them, in case you’re building the equivalent for your own domain.

How the benchmark works

Before getting into what went wrong, here’s the shape of the benchmark end to end.

1. Two different things can count as “got better.” A LoRA fine-tune trained on a narrow corpus can improve in two ways that don’t imply each other. It can get better at geology as a discipline, applying the principles a geologist would bring to material it’s never seen. Or it can just get better at reciting the specific facts in its own fine-tuning data, with no improvement (or even a regression) in general reasoning. Only the first kind is useful, so the benchmark had to tell them apart rather than collapse them into one score. We didn’t build two separate test sets for that. It’s built into how each question is generated (more in point 3): the checkable criteria come from an explicit geological principle rather than the surface text, so a right answer has to demonstrate the reasoning, not just match a sentence in the source.

2. Building the question set. We picked source material from a set of historical tenement reports for the Cue gold field. A large closed-source model read through the documents and generated a fixed set of questions with reference answers, each one citable back to a specific book/page/line in the source, so a correct answer traces to a real passage rather than a plausible-sounding paraphrase. This produced 35 questions, a set that was expanded during later iterations. This corpus-to-questions-to-rubric process is the one we later codified as a runnable skill; see How the grading rubric and questions are built.

3. Building the grading criteria. For each question, we had the same generating model work top-down: distill the geological principle underpinning the answer, then derive the marking scheme from that principle rather than reading it off the page. The structure is a hard gate for minimum correctness, then graduated components above it (detailed in the section below). None of this shipped un-reviewed. A human geology expert vetted every model answer and grading criteria before it went into the fixed set, so a mistaken principle or a wrong gate from the generating model couldn’t quietly become the standard the fine-tune was judged against.

4. Grading the fine-tune’s answers. We ran the fine-tuned model against that fixed question set and had a model score each answer pointwise against the rubric from point 3: gate, then components, then required concepts, then do-not-credit, summed and capped at 10 out of 10.

5. Pairwise against the untrained baseline. A pointwise score alone doesn’t tell you whether the fine-tune actually moved the needle relative to where it started, so we also ran the same fixed question set through the untrained base model and had the judge compare the two answers head-to-head, question by question. That comparison is run twice with the order of the two answers swapped and the verdicts reconciled, so the result reflects which answer is better rather than which one the judge happened to see first.

How the grading rubric and questions are built

The rubric is where a benchmark lives or dies, so we build it as a deliberate, staged process rather than a single “write a marking scheme” prompt. We’ve since codified the whole thing as a runnable skill, build-source-grounded-groundtruth-benchmarking-geology, that turns a supplied corpus into questions, reference answers, a machine-readable rubric, and validation evidence. The shape of it:

1. Explore the corpus before authoring anything. Inventory the source families, terminology, duplicated material, contradictions, and OCR/transliteration variants first. You can’t write a fair question about material you haven’t mapped, and you need to know where the corpus is too thin to assess before you commit to a question there.

2. Write a source-authority contract. Not all files are equally trustworthy: direct primary text outranks OCR, which outranks an auto-generated summary or a filename. When sources disagree, decide whether they actually conflict or just address different scales, dates, or scopes, and either resolve by authority or preserve both with attribution. Never merge two datasets into a fictitious compound fact.

3. Separate question-writing from answer-writing. Questions are drafted first, in their own pass, without their answers or rubrics in view — ideally by a separate agent. This keeps the marking scheme from being quietly reverse-engineered to fit a pre-written answer. A final verifier then adjudicates both against the corpus.

   The prompt spells out what a question is allowed to be before it’s admitted. An excerpt:

   text Copy

   Each question must:
   - Be answerable under the declared candidate-access mode.
   - Test one coherent construct, even if it contains linked subparts.
   - Use exact source semantics and the correct spatial, temporal, stratigraphic, or administrative scope.
   - Define counting and deduplication rules where a numeric answer could otherwise vary.
   - Distinguish observations from interpretations and targets from discoveries.
   - Avoid hidden retrieval requirements that appear only in the rubric.
   - Avoid giving away the answer through excessive context.

   The full prompt is the build-source-grounded-groundtruth-benchmarking-geology skill.

4. Score every question 0–10 through one hard gate, then graduated credit. Each question has exactly one gate (component C1, worth 2–4 points): the minimum central correctness the answer must establish. Miss the gate and the score is zero — no peripheral credit leaks through. Pass it, and the remaining components (non-overlapping, integer points, totalling exactly 10) award demonstrated reasoning positively. Omissions simply go unrewarded; they’re never penalised.

5. Keep keywords as aids, not marks. Required concepts summarise what a component is testing, but exact wording never earns points on its own — a reasoning component demands a demonstrated causal or comparative link, not keyword presence. Accepted-variant and do-not-credit lists catch valid paraphrases and plausible-but-wrong answers respectively.

6. Make every marking block self-contained. A grader holding one question, one answer, and that question’s block must be able to score it without opening the corpus, an appendix, or another question. The local principle and caveats are repeated wherever needed rather than referenced.

7. Calibrate before trusting it. Every question ships with at least three test responses: a gate-fail near-miss (expected 0), a minimum gate pass (expected = C1), and a strong response (expected 9–10). An independent grader must reproduce those scores from the block alone. If it can’t, the block gets rewritten.

That is the process when it goes cleanly. The rest of this post is the five problems that surfaced getting it there, roughly in the order we hit them.

1. Your judge will reward fluency over correctness

Give an LLM judge a loose “score this out of 10” instruction and it will over-reward answers that sound like the domain, all the right vocabulary and confident tone, whether or not they’re actually correct. You need a rubric a model can’t talk its way past. At minimum:

• A hard gate per question: the one fact the question is actually testing. Miss it and the score is zero, no matter how much correct-but-beside-the-point material surrounds it.
• Graduated, non-overlapping components above the gate, so nothing gets credited twice and extra unrequested material can’t push the score past a fixed ceiling, even if it’s correct.
• An all-or-nothing rule for binary facts: determinate data like a single name, number, or structural orientation must be scored as binary: full points or zero, with no partial credit for guessing around them.
• A rule that reasoning components require demonstrated logic, not required-concept presence, so a model can’t hit every keyword and still explain nothing.
• A strict contradiction rule: an answer asserting a claim and its negation earns zero for that component, countering the model’s tendency to hedge by providing multiple conflicting answers.
• A do-not-credit list of plausible wrong answers per question, so a common misconception doesn’t get rewarded just for sounding fluent.

Our rubric format and judge prompt already implement this: gate, then components, then required concepts, then do-not-credit, summed and capped. None of it is geology-specific; swap in your own question set and it holds.

2. Your source material needs to be something the base model doesn’t already know

If your questions can be answered from general pretraining knowledge, you’re benchmarking the base model, not your fine-tune. We used a 1984 Soviet-era regional survey report for Kazakhstan, alongside obscure historical tenement reports for Western Australia. They’re obscure enough that a correct answer requires those specific documents, not general geology.

Whatever you pick, expect it to be messier than you’d like unless it’s a clean, professionally-edited text. The same proper noun spelled three ways, numbers that disagree from one section to the next, OCR garbled enough to flip a value. Resolving that is on you: deciding which of the source’s own contradictions counts as correct, and building a synonym table so the judge doesn’t penalise a right answer for a spelling variant. This is the one problem here that no tooling solves for you, ours included.

3. Pointwise and pairwise scoring can disagree, so run both

A model can tie a baseline on average pointwise score and still lose more head-to-head comparisons, or the reverse. Run only one mode and you’re working from half the picture. Pairwise also needs the comparison run twice with the answer order swapped and reconciled, or the result carries a position bias that has nothing to do with which answer is actually better. (We modelled our evaluation logic closely on enterprise standards, such as those outlined in Google’s agent evaluation metrics).

Our scoring script runs both modes with the swap-and-reconcile step already built in. Doing it yourself means roughly doubling your judge-call budget per question, and settling on a reconciliation rule (we average the two swapped verdicts) before you start, not after you’ve already got numbers that don’t agree.

4. Some generations fail outright — retry the transient glitches, but don’t hide a real failure

A model under test will occasionally hand back an empty completion, a truncated fragment, a wall of leaked reasoning tokens with nothing usable inside, or a flat refusal.

You still need a cleaning step that strips known leaked-token patterns before anything gets judged, so a real answer buried in scaffolding isn’t penalised for the scaffolding. When the cleaned output is genuinely empty, we retry the question on a fresh pod up to three times. Most of these failures are transient, an opencode process that died or a pod that hiccupped, and a retry usually gets a real answer out of the model.

The line is that the retries are capped. A question that fails all three attempts is logged as an error and never counted as a success, not swept under an unbounded retry loop. A model that cannot reliably produce an answer is just as much a result as one that produces the wrong answer. At scoring time nothing is retried: an empty answer that reaches the judge fails the hard gate and scores a zero. You measure exactly what the model produced, failures and all.

5. Don’t trust a single run

Sampling variance alone can move a 25- or 35-question benchmark by a point or more between runs. Run each question several times per model and report the average, ideally with the spread, instead of treating one pass as the answer. A single number can’t tell you whether a difference is real or just noise.