Writing a spec before code lifts coding agents' test pass rate by 21%

Understand first, then write

AI agents are already decent at fixing code, filling in functions and finding their way around someone else's repository. Ask them to build a program from scratch and the picture changes sharply. Especially when there are no sources at all — just a README and a working binary you can run as a black box. On tasks like these, even the strongest models available today fail more often than not.

The authors of SpecFirst propose something straightforward: before writing any code, work out what the program is actually supposed to do. Not superficially, not on the fly, but in a separate phase. Behavior specification first, implementation second.

And that changes the result considerably more than you would expect from such a simple step.

What goes wrong with ordinary agents

Today's standard coding-agent pipeline works like this: it reads the documentation, runs the binary, tries something, immediately starts writing code, checks something again, writes more, fixes again. All of it interleaved.

On paper this looks reasonable. In practice it breaks in several places at once.

The authors demonstrate this on the ProgramBench benchmark. The task there is unforgiving: reconstruct a program from scratch given only a text description and an executable with no sources. Hidden tests then check how closely the new implementation reproduces the original's behavior.

The result is sobering: even very strong models fully solve less than 1% of the tasks. So this is not a case of models needing "a little more intelligence". The problem is the way they work.

What SpecFirst proposes

The SpecFirst idea is simple: split one big task into two separate ones.

First a dedicated specification agent goes to work. Its only goal is to understand the program's behavior. It reads the documentation, runs the binary in different ways, invokes commands with different flags, checks errors, edge cases, output format. It writes up what it finds in a structured `SPEC.md` file.

Then the coding agent takes over. It gets the documentation, the binary and a finished behavior specification. Only then does it start writing the implementation.

The SpecFirst scheme: a separate specification agent sits on top of the ordinary pipeline and gathers the requirements for the program's behavior first.

In the ordinary mode the agent has to explore the system and build a copy of it at the same time. SpecFirst pulls those roles apart. One agent is responsible for understanding, the other for code.

Why this helps:

How the agent builds the specification

The authors did not stop at the general idea of "let the agent poke at the binary for a while". They spell out exactly how it does that.

The specification agent works through an ordinary shell. It runs the program with different arguments, feeds input through standard input, reads `stdout`, `stderr` and the exit code. For interactive programs it uses a virtual terminal.

The probing patterns are what matter here. The agent does not jab at commands at random — it aims at the areas documentation almost always describes badly.

What comes out is not a raw transcript of the session but a compact structured document with six sections: overview, flags, input, output format, errors, edge cases.

That matters, because the next agent does not need a pile of observations — it needs a working manual for how the program behaves.

The authors explicitly forbid shortcuts. The agent may not look for sources online, download the package from a registry, run decompilers or analyze the binary from the inside in any way. Black box only: run it, watch it, write down the conclusions. Otherwise the task loses its point.

How it was tested

The evaluation covers all 200 ProgramBench tasks. These are real command-line programs from open source projects: from Go and Rust utilities to larger tools like SQLite, FFmpeg and the PHP interpreter.

The comparison is fair: the same base agent, the same models, the same conditions. Only one thing changes — whether there is a separate specification phase before the code gets written.

Four models were tested:

The headline metric is the share of hidden tests the reconstructed program passes. That is a sensible choice: an exact match with the original is rare here, so partial progress needs to be visible too.

What came out of it

SpecFirst comes out ahead on all four models. The gain in the average share of tests passed ranged from 6.9% to 21.3%.

With GPT-5.5-high the picture is interesting beyond the average score. SpecFirst markedly raises the share of near-perfect solutions:

So the approach does not merely nudge the average up. It more often carries a solution to the point where the program reproduces nearly all of the original's behavior.

Why it works

The authors show that this is not about "an extra step" but about a change in how the agent behaves while it works.

First, SpecFirst really does explore the program more widely. To measure this they tracked exploration coverage: what share of the lines in the original program the agent touched at all through its calls to the binary. The agent obviously never sees the sources, but the researchers instrumented the reference programs and watched which parts of the code were actually executed during probing.

SpecFirst wins here too: final exploration coverage is 9.4–18.5% higher depending on the model.

Where that gain comes from matters too. It is the specification agent that produces it. The coding agent then explores less, because it already has something to lean on.

Size of the reconstructed codebase over the course of a session: with a specification up front, the agent starts writing code earlier and stays in the implementation phase longer.

Second, the rhythm of the work changes. When the agent has `SPEC.md`, it starts writing code earlier and keeps at it longer, without constantly dropping back into reconnaissance mode. Put crudely, an ordinary agent thrashes between "let me check one more thing" and "time to actually write code". SpecFirst removes that thrashing.

The authors measured this through the growth in codebase size over a session. With a specification up front the growth curve rises earlier and the final implementation comes out larger. Final codebases were 7–29% bigger depending on the model. That looks less like pointless bloat and more like a fuller implementation of the features.

An example where the ordinary approach gets lost

The paper has a telling case involving the `gomplate` utility. It is a command-line templating tool with a pile of namespaces, functions, flags and behavior variants. The README describes the general idea but cannot list every detail: signatures, errors, format quirks, flag compatibility.

On tasks like this an ordinary agent often latches onto the examples in the documentation and goes no further. It might explore `env` and `data` while barely touching the other namespaces. Or it sees an important piece of information once and then simply forgets about it dozens of turns later.

SpecFirst in the same situation first builds a map of the behavior and records it in `SPEC.md`: which functions exist, how the flags work, which combinations are forbidden, which errors get printed and where. After that the coding agent is handed a working specification rather than a vague description.

In four examples with GPT-5.5 you can see that without a separate specification the agent explores for longer and moves to implementation later.

Where the problems remain

For all the improvement, the task is still far from solved. The authors went through 50 failures by hand and found several distinct error types.

The most common case is the final step breaking on implementation anyway. Knowing how a program behaves is not enough — that knowledge still has to be carried into code carefully and in full.

That is a clear pointer to where the work goes next. A single specification phase is not enough to close the task. What is missing is more reliable machinery for checking an implementation against its specification, for self-checking and for self-correction.

What it costs

The improvement has to be paid for. SpecFirst is almost always more expensive than the ordinary mode, because it adds another agent and another inference phase.

On average, cost per task rose by 48% to 130% depending on the model. For GPT-5.5-high the difference is especially pronounced: a lot of the money goes into the specification phase.

But there is a nuance. The baseline agent was not running into the turn limit. It almost always finished on its own, ahead of time. So the problem was not a shortage of budget as such. Simply giving an ordinary agent more steps is not the same thing as giving it a separate phase for understanding the task.

Why this matters

In programming from scratch this is close to obvious: requirements first, then code. With AI agents we somehow spent a long time doing it the other way round, hoping the model would keep everything in its head through a long session.

SpecFirst points at a more general principle. When a task is long, ambiguous and requires exploring an environment, it pays to move the understanding of that task into a separate artifact. Not hold everything in context, but produce an external document you can lean on afterwards.

That could be useful well beyond reconstructing programs from a binary.

Takeaway

If you want an AI agent to write a program from scratch, handing it a README and terminal access is not enough. It needs a separate phase in which it first establishes what exactly the program is supposed to do and writes that down explicitly.

SpecFirst shows that this decomposition delivers a consistent win: broader exploration coverage, an earlier start on implementation, a larger and fuller codebase, a higher share of tests passed. And the effect holds across different models and across tasks of different difficulty.

On long coding tasks an AI agent needs more than code — it needs memory in the form of a specification. Without one it gets tangled in its own steps. With one it has an anchor to build the implementation from.