A harness trained on past runs lifts Terminal-Bench from 0.722 to 0.806

When the problem isn't the model but the harness

In conversations about AI agents, almost all the attention goes to models. Which LLM is stronger, whose code is better, who sits higher on the leaderboard. In practice, what decides the outcome is often not only the model but how exactly it is packaged into an agent: what context it gets, which tools it can call, how its steps are structured, what stays in memory, how the answer is checked.

The authors of MemoHarness: Agent Harnesses That Learn from Experience aim straight at that point. Instead of touching up a prompt or rearranging a pipeline, they propose training the agent's harness itself on the experience of past runs. And not on average across everything — with adaptation to the specific task at the moment it runs.

That is an important shift, because most AI agents today live with a single global configuration for every occasion. One template. One way of calling tools. One step schema. But tasks differ: one needs a short precise answer, another a long stretch of terminal work, another careful handling of output format, another search and verification. The same configuration inevitably suits one task and gets in the way of the next.

MemoHarness is aimed at exactly this problem.

What an agent harness is and why it matters

Put plainly, the agent harness is the control layer wrapped around the LLM. It decides:

This sounds like technical detail, but details like these often move the final success rate by tens of percentage points. The same model performs noticeably better or worse purely because of how its input was assembled, when it was given search access, how the answer length was capped, or how the final output was checked.

The authors make a fair point: the industry already has plenty of methods that automatically improve prompts, instructions or workflows. But almost nobody optimizes the entire agent control system as a whole. Rarer still is making that system change per test case without new labels and without a separate search at run time.

That is precisely what MemoHarness sets out to do.

How MemoHarness works

The core of the work is the idea that a harness need not be treated as one monolithic object. It can be broken into six controllable dimensions.

The MemoHarness scheme: search for a good global harness during training, then adapt it to the specific test case.

Here are the six parts:

This is a sound engineering move. Instead of "magically improving the agent", the system can pinpoint where the failure actually happened. Not enough context? Tool used wrong? Needed a multi-step plan rather than a single call? Output broke on the final formatting?

Then comes the second key element: an experience bank. It has two layers.

That is less a run log than a system's memory of its own successes and failures.

Training without gradients: a guided search

MemoHarness does not fine-tune the model in the usual sense. It runs a search over the space of harness configurations.

It starts from a very plain baseline configuration: no demonstrations, no memory, no tools, no complex orchestration. From there the system proposes new variants step by step. Each variant is run against labeled tasks, scored on quality, and its execution traces are written into the experience bank.

One detail matters here: selection follows a correctness first, cost second rule. Solving the task correctly is the primary criterion. Tokens and cost enter only as a tiebreaker when variants score the same on quality.

This sounds almost obvious, but against the current fashion for "cheap and fast" the emphasis is useful. The authors clearly do not want the system drifting toward cheap but wrong configurations.

After several rounds, what gets picked is not the luckiest peak on the test set but a final global harness selected on validation. Another healthy choice: less risk of overfitting to one small set of tasks.

What happens when a new task arrives

The interesting part starts at test time. MemoHarness does not apply the same final harness to every task in a row. It takes the globally found variant and nudges it toward the specific input.

How exactly:

Crucially, all of this happens with no feedback at test time. The system does not know the correct answer, gets no reward and starts no new search loop. It simply uses the experience it has already accumulated. That makes the approach more realistic for production use.

In effect, the paper argues for treating an agent not as a fixed product but as a system that can recall: "on tasks like this, memory is usually where I break" or "in cases like this it is better to switch on the multi-step schema and check the output format more strictly".

What the experiments showed

The authors tested MemoHarness on three very different kinds of task:

The headline result: on Terminal-Bench the system reached 0.806 average success against 0.722 for the strongest baseline. For that setting, a noticeable gain.

Comparison against baseline systems on Terminal-Bench: MemoHarness posts the best average result.

What matters is that this is not a comparison against a deliberately weak opponent. The strongest baseline is already a system specialized for terminal work. The gain was not taken from a straw man; it came out of a fairly demanding setup.

The trajectory over the course of the search is more interesting still. On all three benchmarks the final version beat the starting one, but the shape of the improvement differed.

Harness quality at different stages of the search: the final result beats the baseline on all three benchmarks.

FinanceAgent saw the largest gain: roughly 0.600 to 0.767. LiveCodeBench improved too, but less, because the base model there was already close to the ceiling. That is a telling detail. MemoHarness helps most where the task is longer, harder and demands more control over the process. Where the model already solves the task in one shot, there is less headroom.

The per-iteration chart says the same.

On FinanceAgent additional search iterations keep paying off, while on LiveCodeBench the gain saturates quickly.

On FinanceAgent extra search rounds kept helping almost to the end. On LiveCodeBench everything flattened out fast. The conclusion is simple: the more multi-step work, tools and chances to go off course a task involves, the more a good harness matters.

Does it transfer to other task sets and models

One of the most useful questions: did the system merely learn local tricks for one set of tasks?

The authors checked this two ways.

First, transfer to other task sets. The picture here is not magical but normal and plausible: transfer happens, and it is selective. The harness found on Terminal-Bench produced gains on several external sets, including comprehension tasks, refusal of harmful requests, and software engineering. But not everywhere. On saturated sets, where the baseline is already near maximum, there was almost no movement.

That is a good sign. The paper promises no universal incantation. It does show that some of what the search found outlives one particular benchmark.

Second, transfer to other models. This is probably one of the most interesting results. The harness was searched on GPT-5.3-Codex and then run on six other models with no new search. All of them improved. The average gain was about +0.098.

The jump for GLM-5 stands out: 0.500 to 0.733. GPT-4.1 gained less, but still gained. That hints that MemoHarness learns more than per-model tricks — it learns more transferable rules for organizing an agent's work.

This is not yet conclusive proof of generality, of course. But as a practical signal it is a strong one.

What about cost

A scheme like this carries an obvious risk: if you constantly pull past experience into the context, everything can get expensive.

The authors checked this on Terminal-Bench. MemoHarness does consume more input tokens. But most of that context is cached, so the final dollar cost stays competitive. In their accounting the system came out cheaper than some strong commercial baselines while scoring higher.

There is a caveat: much depends on how well the retrieved experience caches. If a real deployment gets less out of the cache, the economics change. The authors admit this openly.

And that is one of the paper's stronger traits: no attempt to sell the result as absolute truth. There is a clear engineering idea, a noticeable gain and carefully stated limits.

What actually matters in this work

The paper's main value is not only in the specific numbers. It is that MemoHarness offers a different view of how AI agents get better.

Progress today is usually reduced to three levers:

Here a fourth lever appears: accumulate execution experience and turn it into an adaptive control system for the agent.

It reads like the field growing up. Instead of "what is our best prompt?", the question becomes "how does the agent learn to work better from its own past actions?". That is closer to real engineering of complex systems than to fishing for a lucky phrase.

Takeaways

An AI agent can be improved beyond the model and beyond the prompt — the whole external control layer is fair game.

The approach pays off most on hard tasks that need many steps, tools, memory and format control. That is where an adaptive harness wins the most. On simple tasks already near the quality ceiling the effect is smaller — which is also what you would expect.

It also matters that the system does not learn blind. It stores experience from successful and failed runs, extracts the patterns that recur, and uses them to adjust to a new case with no feedback at test time.

Which leads to a fairly practical conclusion: the next visible jump in AI agents may come not only from bigger models but from systems that remember, diagnose and rewrite their own harness better. Not one setting for every occasion, but a live configuration that adapts to the task. For agent systems, that looks like one of the most promising paths in the near term.