Rewiring the agent harness cut cost 41% with no drop in quality

When the bottleneck isn't the model, it's the harness

There's a reflex in the AI industry: when an agent underperforms, it gets more tokens. A longer prompt. More steps. More tools. More replaying of the history. More thinking. On paper this often looks like progress. On the infrastructure bill it looks like a disaster.

A new paper with a fitting title, The Harness Effect, goes straight at that sore spot. Its central claim is simple: the cost and speed of an AI agent are set not only by the model but by the orchestration layer around it. How the system assembles context, when it calls tools, how it stores history, what it replays on every turn and what it doesn't — all of that can move the economics more than the choice between the models themselves.

And this isn't theory. The authors took six models and 22 enterprise tasks and ran something close to a lab experiment: the models and the tasks stayed the same, only the agent's harness changed. The result is uncomfortably clear for the whole industry: 38% fewer tokens, 41% lower cost and 44% less time — with no noticeable loss in quality.

That matters for the market. We're used to arguing about which LLM is smarter. This paper is a reminder that sometimes the right question is how smart the thing running that LLM is.

What the authors call token escalation

The authors introduce a new term: token escalation. It's the case where a system's usefulness grows more slowly than the number of tokens per task. Put bluntly, you buy a little quality at the price of a lot more spend.

With AI agents this happens naturally. One user request isn't one model call, it's a whole chain. The system prompt. Tool schemas. Documents from retrieval. Intermediate answers. Tool outputs. And if the system is built naively, all of it goes back to the model in full at every next step.

Hence the core problem: cost grows not linearly but sometimes almost quadratically. The more steps the agent takes, the longer the tail of past messages it drags along.

Why the token bill for AI agents balloons so fast: naive history replay grows close to quadratically, while careful orchestration stays nearer to linear.

This is probably the paper's most useful conceptual contribution. The authors show that tokens are consumed not only by the model itself and not only by its taste for long answers. A huge share of the bill is created by orchestration — the code around the model. It decides what lands in the context, what gets cached, what needs compressing and what can live outside the current window.

For the past two years the market has mostly hunted for optimizations inside a single model call: shorten the prompt, speed up inference, cap the length of the reasoning. This paper points at a different level — between calls. At how many calls add up to a working agent.

How the experiment was set up

The authors didn't compare "our good system" against "someone else's bad system" in the abstract. They made a controlled swap of the orchestration layer.

What exactly was compared:

On one side was an ordinary production agent loop: a large system prompt, full replay of the history, sloppy handling of context overflow, waiting by polling and the other compromises the industry is used to. On the other side, the Writer Agent Harness — a stricter and more frugal execution system.

The tasks aren't toys either. They're an enterprise profile: retrieval and grounding in documents, content creation, running workflows, working with external tools, presentations, voice, images. The things companies actually buy AI agents for.

How quality was measured matters too. The authors say plainly that with 22 tasks you cannot draw loud conclusions from a small gain in quality. So they call the final 0.78 versus 0.81 not an improvement but parity. That's a good sign. The paper makes no attempt to sell a statistically shaky plus as a breakthrough.

What actually changed in the orchestration

The most interesting part of the work isn't the numbers, it's the engineering mechanisms that produced them.

The first key technique is a two-zone prompt. Everything that can stay stable between turns goes into a fixed prefix: the tool catalog, the system instruction, the durable part of the conversation. Everything that changes at every step goes into a volatile tail.

Why bother? Caching. If the start of the prompt is byte-for-byte identical, model providers will often serve it from cache at a steeply reduced price. The authors show one call in which 99.9% of the prompt tokens came from cache.

The two-zone prompt: a stable prefix for caching and a volatile tail for everything that changes from step to step.

You can't simply hope the cache will somehow do its job. The prompt has to be structured so that nearly everything long and expensive stays unchanged.

The second mechanism is structured compaction of the history. Instead of blindly truncating the middle of the context, the system builds checkpoint summaries: what has been decided, what the constraints are, what was tried and why it was dropped, which files were touched, what to do next. And not from scratch each time — incrementally.

The third is moving heavy context outside the window. Large tool outputs don't have to sit inside the model's context window. They can be written to files, passed along as short excerpts and pulled back in by reference when needed. Subagents work the same way: a child agent can run a broad search inside its own local context and hand the parent only a short summary.

The fourth is waiting without tokens. When an agent is waiting for human approval or for a long background job to finish, it shouldn't sit there polling the system and burning tokens. It should sleep and wake on an event.

The fifth is spend control on failure. Retries, loops, repeated failing tool calls are a hidden black hole in the budget. Good orchestration has to tell error types apart and stop the agent from buying the same failure three times in a row.

Compressed into one line, it comes out as a very practical principle: as many tokens as possible should either come from cache, or carry new useful information, or be spent only on steps that are genuinely needed.

The main results: savings almost everywhere and for everyone

Now to the numbers. Pooled across all six models and 22 tasks, swapping the orchestration layer gave:

The paper's headline result: with the same model, the new orchestration noticeably cuts cost, latency and token spend.

That alone is a strong result. But something else is more interesting: the effect turned out to be almost independent of the model.

Cost fell for all six models, in a range from 33% to 61%. So the win didn't go to one lucky provider or one particular class of models. It went to the execution architecture itself.

Cost and latency drop for every model: the effect comes from the orchestration layer, not from any particular LLM.

This is perhaps the most unwelcome finding for teams that argue endlessly about model choice and think little about their agent's infrastructure. The authors say it outright: on their task set, the orchestration layer moved cost more than switching from the most expensive model to the cheapest one.

A sobering thought. You can spend weeks choosing between providers to save tens of percent on the price of a token, and lose more than that to a carelessly built agent loop.

What about quality?

Here the paper is more careful, and rightly so. On average, quality didn't drop. Better than that: across the 48 model × capability cells there were more gains than losses — 30 improvements, 11 unchanged, 7 regressions.

But those regressions aren't randomly distributed. Almost all of them land on the weaker models and on the scenarios where orchestration loads the model hardest: complex workflows, working with external tools, presentations.

From this the authors draw an interesting idea: stronger models extract more from good orchestration. They call it the orchestration leverage effect. The same infrastructure layer can give a powerful model both savings and a quality gain, while a weak model gets only the savings — and sometimes extra cognitive load on top.

That's an important nuance. The paper doesn't say good orchestration automatically makes any model better. It says something else: good orchestration almost always makes the system cheaper, but the quality on top depends on whether the model can use that structure at all.

For practitioners this is very useful. You can count two things separately:

Why this matters for the AI market right now

The work has a clear business moral. While the industry celebrates falling prices per token, teams start spending even more tokens. It's the classic it-got-cheaper-so-we-use-more effect.

Which is why a lower price per token doesn't save you on its own. If an agent answers every new capability with a longer context and more steps, the total bill still climbs.

Hence the main practical takeaway: the maturity metric for an AI agent should be not just quality but task completions per million tokens. Or, more simply, how much useful work you get per unit of spend.

The authors give a good example: completions per million tokens rose from 54.9 to 92. That isn't cosmetic optimization, it's a change of economic regime.

For companies with high task volume the gain becomes material fast. The authors estimate that at a million agent tasks a month the difference between the two approaches is on the order of $90,000 a month. Even treating those estimates cautiously, the scale is clear: orchestration stops being an internal engineering detail and becomes a line in the P&L.

The takeaway

We're used to thinking of an AI agent as a model plus tools. The authors make a convincing case that there is one more critical layer between the model and the product: orchestration. It largely decides whether your agent spends tokens sensibly or burns them without thinking.

If you want a cheaper and faster AI agent, don't rush to change the model — first look at how its working loop is built.

The work has limits, of course. The task set isn't huge. The authors work for the company whose system is being tested. The comparison with outside frameworks is mostly architectural rather than experimental. But even with those caveats, the result looks serious.

And it may be one of the most useful pieces written about AI agents in a while, because it's a reminder of something simple: a breakthrough in AI isn't only a new LLM. Sometimes it's knowing how not to pay twice for the same context.