If you gave an AI the same messy exploration archive you would give a consulting geologist, how much would you trust what came back?
Traditional AI benchmarks are basically an exam paper everyone has had access to for years. Eventually you don’t really know whether a model has learned the subject or learned the exam. This isn’t the only problem when it comes to working out which model is best for geological work, however.
Suppose you’re an exploration manager with 15 years of reports, logs, maps, assays and consultant studies sitting in SharePoint. Someone wants you to spend $100,000 integrating an AI system. The vendor tells you that Model X scored 83% on some academic geology benchmark. You say: who cares?
Because every terrain is so unique, it can be actively damaging to rely too much on general knowledge. Sure, bismuth is often a gold pathfinder, but if you’re an explorer you want to know whether it’s a pathfinder in your specific geology.
That’s why we made the Groundtruth Dynamic Benchmark. Rather than asking a standard set of questions to test the sector knowledge of a model, Groundtruth generates a fresh set of questions and scoring rubrics from your own dataset, letting you compare models and agent harnesses on the data that matters to you.
How it Works
If you want to test the skill and versatility of a new model or agent harness, we provide three diverse test sets free: a set of Wamex reports from a gold mining property in the Murchison, four historical USGS reports on the Southwestern US, and four Canadian technical reports. If you want to compare model or agent skills on your own data, the system will generate a custom question set and scoring rubric based purely on that dataset. The generation process is set up such that some will require finding facts. Others will require reconstructing sequences of geological events, distinguishing competing interpretations or deciding what the evidence actually supports.
Can it find the evidence? “Which formations make up this succession?”
Can it reconstruct what happened? “What happened first: metamorphism, deformation or gold mineralisation?”
Can it reconcile geologists who disagree? “One report says there is a major shear. Another says there isn’t. What does the evidence actually establish?”
Does it know when the evidence runs out? “Two reports use different names for apparently similar greenstone belts. Can the model resist simply declaring them identical?”
Once you have your data and your question set, you’re ready to test. Pick a model and add in a custom agent harness (if you want one) then let it run. Once complete, the grading AI will check the answers - making sure that they match the grading rubric rather than simply being persuasive-sounding verbiage.
So Who Won?
Kimi K3.
But that’s not the full story.
While Kimi got the highest one-off score, statistically there was enough overlap between the three top models that they can more or less be treated as equally good. This being so, the question of price is an important one.
If we compare price and accuracy, we end up with four models offering roughly equivalent ROI: Deepseek V4 Pro, GPT 5.6 Sol, Claude Sonnet 5 and Kimi K3… at least until Ox Alpha was released, which shifted the Pareto frontier drastically.
What it Means
This means that the exploration manager from the opening paragraphs can now run:
- Commercial models + vanilla retrieval
- Commercial models + their proposed RAG system
- Commercial models + an agentic search harness
- Whatever the vendor is proposing
against a custom benchmark derived from their own documents.
Then six months later, when someone says “we should migrate to Model Y because it’s 40% cheaper,” they can rerun the same frozen company benchmark and find out whether that change actually degrades performance on their work.
Want to see the full leaderboard? Start with the Groundtruth benchmark.