Why pipelines are the new prompts, and fixable is better than perfect
Research harnesses are here to stay. Our partners at NextMaps just released an improved version of their exploration analytics report pipeline, and our Geocluster agent harness is now fully open-source on Github, among many others. Obviously we would love you to use ours off the peg, but what if you need a custom implementation?
The temptation is to spot a mistake and insert a reminder not to do it again directly into the prompt (or the CLAUDE.md file or whatever you’re using). Even Anthropic can’t resist the lure:
Modern long-context models are big boys, after all. They can take it.
Up to a point. The problem is that if you’re building a pipeline or a harness, contexts get looooooooong quickly, and if you don’t have a great memory management/prompt cacheing system the string that is actually sent to the model pretty soon starts to look like insane and self-contradictory glossolalia.
Adding prompt instructions is the equivalent of yelling the entire employee handbook at your new intern on his first morning. Sure, he will make more of an effort to comply than if you’d said nothing, but he will eventually get flustered and start forgetting bits, which makes you yell corrections, which only confuses him further.
If you just need one quick fix, this might get you there, but if you want a replicable process you’re going to have to deal with the problem at a higher level.
Error-Correcting Harnesses
The first thing to bear in mind is that because AI models do not produce deterministic whitebox outputs, it does not matter how good your prompt is, if you run it 100 times it will not give 100 perfect outputs. This does not mean you should resign yourself to failure and inaccuracy, however. Instead of trying to make it perfect, you need to build a system in which imperfections are unimportant. How? Here are our top five principles of harness design.
- Shorter is almost always better.
The shorter and simpler a task is, the less likely a model is to mess it up. Split your tasks into sub-tasks and have separate agents work on each. Not only does this reduce the likelihood of errors, they can be great at spotting when another agent has got stuck in a loop or failed to save an artifact and reminding it to do its job. If agent B needs agent A’s output to start its own task, it will demand that agent A do a proper job. If both tasks are being done by a single agent and task A failed, it is much more likely that the agent will simply hallucinate an output and continue.
- Fixable is better than perfect.
Modern LLMs are the direct descendents of Claude Shannon’s work on error-correcting code - the idea that it is impossible to be entirely sure that a message will ever be transmitted perfectly, but that it is possible to add checks to make sure that mistakes will be noticed and - ideally - fixed. Every time a verbal instruction is given there is no 100% sure method to ensure that no parts are garbled or omitted, or that the receiver has not misheard or misunderstood the content. However, it is very easy to add failsafes to spot it when it happens. Adding “MAKE NO MISTAKES” to a prompt might reduce the number of mistakes an agent makes, but submitting its work to a second QC agent with a list of common mistakes to check for definitely will.
- Design for the smallest model.
This is a good idea purely for money-saving purposes, but it also makes your system much less fragile. If your pipeline can only run with the very latest model from GPT or Anthropic this means that a) you are highly vulnerable to guardrail or system-prompt edits beyond your control and b) your pipeline design is probably very bad. Given a good harness, current open source models are easily capable of matching SoTA Claude; it’s your job to put them in a position to show what they can do.
- It’s ok to fail, it’s not ok to fail silently.
“The best lack all conviction, while the worst are full of passionate intensity.” It is tempting to read increasingly confident responses as a sign of better performance. In fact they are more likely to be a reflection of a prompt that encourages simulations of certainty rather than admissions of uncertainty. A good way to deal with genuine failure to resolve a definite answer is to set up a tagging system: agents can output responses tagged as [confirmed_data], [inferred_information], [necessary_assumption], [code_results] or whatever else is required to create a visible reasoning process. When we set up our Geocluster agent we deliberately make sure that these outputs were colour-coded in the user interface to give a reader instant awareness of where the numbers they were seeing had come from.
- Chat history is a variable.
The Claude, GPT and Gemini web portals give a great chat history experience because there is a vast suite of infrastructure in the back end to manage the historical information being fed into every response - everything from assistant persona to inferred user personality to prior conversation directories. If you’re not actively managing agent memory locally rather than simply relying on a passive chat history system your agents will never perform as well as an in-browser model. It is almost always better to feed an agent the specific information you want it to respond to as a variable than to rely on it being somewhere in prior conversation turns. So if agent A needs to reflect on a data analysis exercise it conducted in a previous turn, give it only the output and an explanation of what it should do with it, not the entire conversation that led to it producing the output in the first place.
A Case Study: Glenburgh Gold
When NextMaps wanted to stress-test its analysis pipeline, it deliberately picked some very tricky ground: E 09/1325, a gold exploration licence in Western Australia, and the 56 open-file WAMEX reports covering it, lodged between 1995 and 2022.
The trap was the point. The licence sits beside several major gold deposits, and because exploration boundaries have been redrawn repeatedly over three decades, many of those historical reports discuss both the ground being assessed and discoveries that now fall outside its modern borders. A basic RAG system would retrieve all of those genuine references to gold and conclude they applied to E 09/1325. They do not.
The NextMaps harness does not ask the model to work that out from the prose. Every named place and every drillhole arrives already tagged with its true distance from the licence boundary, computed from coordinates before the model sees a word. A separate verification pass then re-reads the finished report against the same ground truth. The model’s job is to write. Deciding what is inside the fence is not left to it.
That mattered here more than usual, because the Glenburgh project the licence forms part of contains roughly a million ounces of identified gold, and almost all of it sits on the neighbouring mining leases rather than on this licence. The easiest answer was also the most tempting one: point at the million ounces next door and call E 09/1325 highly prospective.
It rated the licence highly prospective, and then explicitly refused to use the neighbours to justify it: “The tenement therefore merits a High prospectivity rating based on its own data, not on the adjacent Apollo/Icon/Tuxedo deposits.”
The case it built instead came from evidence it could place inside the boundary: a gold-silver occurrence at Thunderbolt, 322 drillholes already sunk in the SW Area, and a best intercept of 18.8 g/t Au whose coordinates fall within the licence. It also volunteered what it could not support, noting that the published project resource figures span several tenements and cannot be attributed wholly to this one, and that the northeast of the licence has seen only seven holes. That is what you want from an AI when cash money is riding on the answer: not confidence, but a clear line between what is established and what is not.
There is a postscript. Thunderbolt, the occurrence the pipeline placed inside the boundary from the public record alone, with nothing in its stack newer than 2022, is the same target Benz Mining now describes in its investor material as the largest source of projected upside in the Glenburgh exploration target: undrilled by Benz, historically tested to under 100 metres, with three rigs mobilised and systematic drilling underway this year. The pipeline put infill drilling on that trend at the top of its recommendations and flagged that its depth continuity was still open. It did not know about the rigs.
Want to try it for yourself? You can visit NextMaps. Want to try building your own? Our analysis agent is open sourced and editable on GitHub.