Rewriting the harness code beats a hand-tuned baseline by 7.7 points

It isn't only the brain that matters, but the wiring around it. How Meta-Harness automates the job of building an LLM harness.

Not only the brain, but the wiring around it

When we talk about AI we almost always talk about the models themselves: size, data quality, architecture, speed. But in real applications there is another layer that decides how useful the whole thing ends up being — the harness. It is the code wrapped around the model that decides what to carry over from earlier steps, how to pull up the right pieces of information, what exactly to put in front of the model at each moment, and how to run a multi-step process.

The authors of Meta-Harness: End-to-End Optimization of Model Harnesses open with an observation most engineers know from experience: on one and the same benchmark, the gap between a good harness and a bad one can be several times over, even with the model held fixed. The problem is that harnesses are normally tuned by hand. That is slow, expensive and scales badly: you pick apart the failures, invent heuristics, test hypotheses — and then do it all again.

The obvious move is to reach for prompt optimization methods. And that is where the central conflict appears: most of them run on very compressed feedback. One optimizer sees only a scalar score, another a short summary, another a brief trace of what happened. For a harness that is not enough: decisions about memory and context retrieval show their effect not immediately but many steps later. A bad context choice early on can come back at the end of the trajectory, and without detailed traces it is hard to tell what actually went wrong.

The Meta-Harness idea: search over executable code, not text — and keep everything

Meta-Harness is a system that searches over and improves the harness code rather than individual turns of phrase. At its center is a coding agent that can read the project, edit files and propose a new harness. The important detail: the agent is given access not to brief summaries but to the entire history of experiments, through the file system. For every candidate it stores the source, the metrics and the execution traces — literally a record of how the system lived: which prompts were assembled, which tools were called, how state was updated, where it fell into loops or failed outright.

Instead of trying to cram all that experience into the context window, the agent decides for itself what to look at: it uses the ordinary tools, grep and cat, compares candidates against one another, finds recurring failures and writes the fixes. In practice, the authors note, in one of the most demanding settings the agent reads dozens of files per iteration and regularly leans on many earlier attempts — that is, it behaves like an engineer investigating causes and testing hypotheses.

Three testbeds: classification, retrieval-augmented math, and agentic coding

The authors test the approach on three tasks where the harness genuinely determines how the system behaves.

In online text classification the LLM receives examples one at a time, adds to its memory, and then classifies new cases. Here Meta-Harness finds strategies that are not merely more accurate but also cheaper on context: the best version beats ACE, a strong hand-built baseline, by 7.7 accuracy points while using roughly four times fewer context tokens. That is the telling part: the gain comes not from pouring more examples into the prompt, but from a smarter policy about which examples to retrieve and how to lay them out for the model.

In the second scenario — retrieval-augmented math reasoning — the discovered harness improves accuracy on 200 IMO-level problems by an average of 4.7 points, across five held-out models at once. Meta-Harness finds more structural solutions: routing by problem type (geometry or combinatorics, say), different selection and ranking rules, and even a choice of when to deduplicate aggressively and when to leave the raw neighboring examples alone.

The third testbed is agentic coding on TerminalBench-2. Here too the result reads like an engineering find: one of the winning moves is to take a snapshot of the environment before the main loop starts and add it to the initial context.

Why a rich memory of past runs turned out to be decisive

The ablations are worth a look on their own: give the agent only the numbers (the scores), or even the scores plus short summaries, and quality drops sharply. Full access to the raw traces, by contrast, produces clear progress. In complex LLM systems a great deal breaks at the level of process — a bad retrieval, or one extra rule in the prompt that sends the run into a loop 20 steps later. Summaries tend to swallow exactly the details you need for the diagnosis.

In the end, Meta-Harness looks like an attempt to automate harness engineering through a realistic workflow: accumulate artifacts, investigate causes, make careful edits to the code, and check the result on the tasks. And judging by the results, the approach really is capable of managing context well.