Coding agents skip looking at the app when the task gets long

When the app is the spec

A coding agent is usually evaluated the simple way: hand it a task, a bug report or a test suite, then check whether it fixed the code. A new paper, ProgramDistill, proposes a setup much closer to real work. You have a working reference app. You also have a broken or incomplete version of it. The agent never sees the reference's source. It has to walk the interface itself, work out how everything is supposed to behave, and restore that behavior in the editable app.

This is a test of the ability to extract a spec from a live system. For web development that framing is natural. It is how people often work: open an old version of the product, a competitor's service, a prototype or an internal demo, then reproduce the behavior you need in code.

The authors turn the idea into a full benchmark for coding agents — and they build it almost entirely automatically, not by hand.

The ProgramDistill idea: the agent studies a working reference app, edits the code of the current app, then checks whether the required behavior is back.

What ProgramDistill is

ProgramDistill is a benchmark where the task is posed not through a written description but through watching a running application. The paper works with interactive web apps: kanban boards, email, résumé builders, CRMs, calendars, finance tools and other services.

The setup is simple:

🟠 There is a reference app that works correctly.

🟠 There is a current app with part of the functionality removed.

🟠 A coding agent observes the reference's behavior through a browser.

🟠 It then edits the current app's code.

🟠 After that, the system automatically checks whether the required behavior is back.

The interesting part is elsewhere. The authors do not treat an app as one big "restore everything" task. They break it into chains of dependent actions. For example: log in → create a board → create a card → open the card details → edit a comment. Each action depends on the ones before it, so difficulty becomes a dial: from a small local fix to a long multi-step restoration.

How one app became thousands of tasks

The paper's main technical contribution is an automated mine–craft–patch pipeline. It is essentially a task factory built out of live web apps.

The mine–craft–patch pipeline: the system first finds reproducible actions, then hides their implementation in the code, then asks the agent to restore the behavior from the reference.

It runs in three steps.

Step 1. Find reproducible behavior

First, separate AI agents explore the app through a browser, looking for meaningful user actions: create an object, edit a record, open a form, drag a card, import a file and so on. When an action succeeds, the system records a trace: the sequence of browser steps and the observable results expected from them.

The key point is that this behavior is then replayed automatically. If a trace cannot be reliably reproduced from a clean state, it does not make it into the benchmark. Fragile, flaky and hard-to-verify scenarios get cut.

Across 26 apps, that yielded:

🟣 1,975 verified behaviors

🟣 2,862 atomic tasks restoring a single piece of logic

🟣 1,201 cumulative tasks that require bringing back several dependent features at once

🟣 4,063 tasks in total

The scale, briefly:

🟠 Tasks come from the running app itself

🟠 Only reproducible scenarios are kept

🟠 The benchmark holds both local fixes and long dependency chains

How the benchmark grew: from the mined behaviors the system assembled more than four thousand tasks of varying depth and type.

Step 2. Break the feature properly

Once the behaviors exist, the system has to turn them into real repair tasks. It does that by deleting the implementation of a specific feature from the source.

But there are many ways to delete something. You can honestly cut out the core logic. Or you can cheat: flip a flag, add an early return, hide a button, leave nearly all the code in place. The authors go out of their way to prevent these shallow breaks.

They use two kinds of masking:

🟠 Logic only — the interface stays, but the internals stop working.

🟠 Logic and interface — both the logic and the UI elements the feature drives are removed.

The system then verifies that:

🟠 the app still starts;

🟠 earlier steps in the chain still work;

🟠 the target feature is genuinely broken;

🟠 the "gold patch" puts everything back and restores the behavior.

That makes for a clean setup. The agent does not have to fix everything around it. It gets a controlled break and has to restore exactly what was removed.

Step 3. Hand the task to the agent

Now the test itself. The agent gets:

🟣 a repository with the functionality removed;

🟣 a short description of the feature;

🟣 browser access to compare the current app with the reference;

🟣 tools to read, search and edit code.

What it does not get is the important part: the reference's source, the ready-made patch and the verification traces.

Scoring is by behavior replay. If the recorded chain of actions runs after the patch the way it runs on the reference, the task is solved. If the agent restored only part of a long chain, that can be scored separately.

What the experiments showed

The authors ran nine frontier models on ProgramDistill, among them GPT-6 Astra, Claude Opus 5, GPT-5.6 Sol, Gemini, Grok and others.

The headline result: models handle local fixes markedly better than long dependent restorations.

Agent quality drops as restoration depth grows: individual features are easier to fix than long chains of dependent behavior.

For partial restoration of an app:

🟠 GPT-6 Astra solved 84.3% on average

🟠 Claude Opus 5 — 68,7%

🟠 GPT-5.6 Sol — 60,7%

But look past the average to chain depth and the picture gets worse fast. On depth-1 tasks, where a single piece of behavior has to come back, the best models are almost flawless. At depth 8 it is already hard:

🟣 GPT-6 Astra falls from 100% to 64%

🟣 Claude Opus 5 — from 96% to 32%

🟣 GPT-5.6 Sol — from 92% to 32%

The results, briefly:

🟠 On short tasks the best models are close to 100%

🟠 On long chains quality collapses

🟠 The problem is composing several fixes without breaking dependencies

Another important detail: tasks that require restoring both the interface and the logic are noticeably harder than logic-only ones. Even for the leader, the gap between the two modes is wide.

Why agents fail on long tasks

The more interesting part of the paper examines not just scores but the agents' own behavior.

The authors looked at how the workload changes as tasks get harder. As restoration depth grows:

🟠 the amount of code that has to come back grows roughly 9.3×

🟠 the number of target browser actions grows roughly 10.7×

But the effort agents spend on each individual feature being restored goes down. Observation sags hardest:

🟣 observation steps on the reference per target fall about 75%

🟣 checks of the agent's own app per target also fall about 75%

🟣 code edits per target fall less, but still noticeably

As task depth grows, the amount of work grows faster than the agent's effort per feature; observing the reference and self-checking fall off most sharply.

Put plainly: the longer the task, the more the agent starts economizing on understanding and verifying behavior. It looks at the reference less, re-checks its own result less, and then leaves part of the needed code unrestored.

The numbers show it clearly:

🟠 There is 9–11× more work

🟠 Observation and self-checking per target drop about 75%

🟠 The agent misses required details of the behavior more often

On a short task you can study the interface carefully and run the scenario several times. On a long one the agent slides sooner into a "close enough, I'll just fill it in" mode. That is exactly where the errors pile up.

The differences in strategy between models are interesting too. GPT-6 Astra turned out to be the most observant: it looked at the reference and at the current app more often than the rest, while making fewer code edits. Better results here come not only from code generation but from how the agent splits its effort between observing, editing and checking.

Rebuilding the whole app

The authors also tested a harsher setup: not a partial fix to an existing app but full reconstruction from a minimal template. Here you essentially have to build the product again from scratch, with only the working reference app to look at.

The results are markedly worse:

🟠 GPT-6 Astra: 58.98% on atomic behaviors and 49.15% on full cumulative scenarios

🟠 Claude Opus 5: 42.03% and 28.81%

🟠 GPT-5.6 Sol: 33.39% and 21.07%

Even the best models, in other words, are far from reliably reconstructing a real interactive product.

Why does this happen? The authors went through almost a thousand failures and found something simple: most often the agent never saw the behavior in the reference at all. That is the largest error category — 59.2% of all failures. The rest are cases where the behavior was noticed but not implemented, implemented in the wrong form, or implemented with the wrong state.

The main cause of failure in full reconstruction is that the agent simply did not notice the behavior in the reference; the remaining errors come from wrong implementations and weak self-checking.

Failures, briefly:

🟠 The main cause of error is that the agent under-explores the reference

🟠 That is 59.2% of all failures

🟠 The problem often starts before any code is written

This breaks the popular intuition that what holds today's agents back is only the code. The bottleneck often appears earlier: the agent did not explore the system enough.

Why it matters

ProgramDistill shows that current coding agents do reasonably well on isolated fixes and worse wherever they have to derive the spec themselves, hold a long chain of state, and check the result carefully.

For the industry this matters for several reasons.

🟣 Benchmarks move closer to real development. In practice you are rarely handed a perfect task description; you are handed a working product, a mockup or an old version.

🟣 Compositional difficulty becomes measurable. Not just "fixed the bug" but "held eight dependent steps in a row".

🟣 A new axis for comparing models appears: how they observe, how they check themselves and how they allocate attention.

🟣 A benchmark like this suits fine-tuning. You can turn it into a curriculum, from simple fixes to long scenarios, including through reinforcement learning.

And one more idea that matters: the working app here is not merely the object under test. It becomes a source of specs, tasks and checks. That is a practical idea for future data pipelines around coding agents.

The takeaway

ProgramDistill is an attempt to test coding agents in a more lifelike mode: when the required behavior is not written down anywhere but hidden inside a working interface. Instead of a text task the agent gets a live reference, and the question is whether it can understand the functionality and reproduce it in code.

The main finding: the longer and more dependent the scenario, the faster quality falls. The best models still hold up on short fixes but start slipping badly at depth 6–8. The cause is not only the code. Agents often observe too little, verify their own solution too little, and miss important details of the behavior.

If you want to build useful agents for web development, the ability to write code is not enough. Three more things are needed: exploring the reference, holding long dependencies and verifying the result strictly. ProgramDistill measures exactly that trio.