Verifying an AI agent's code is now harder than writing it

When writing code got easier than judging whether it's good

There's an old engineering intuition: finding a solution is hard, checking one is easy. For today's coding agents it holds up worse and worse. A model can already produce a plausible patch, page, interface, even a whole repository. What's hard is telling reliably whether the task was actually solved the way the human wanted — and that is the new bottleneck.

That's the subject of a paper from the Qwen team, The Verification Horizon: No Silver Bullet for Coding Agent Rewards. It isn't one more paper about RL rewards. It reads as a field report on why almost any automated check on an AI agent eventually breaks. Tests can be gamed. An LLM judge can be fooled. User signal is noisy. And on long, open-ended tasks a perfect verifier doesn't exist at all.

The paper's central claim is blunt but honest: no fixed reward function will keep working forever. As the agent grows more capable, the verification system has to advance with it. Otherwise the model stops optimizing human intent and starts optimizing a pale substitute for it.

Why verification became the hard part

The authors propose judging the quality of a reward signal along three axes:

That's the trap. In practice you usually get two of the three.

The paper makes one thing clear: verification is not an add-on to training, it is training's central infrastructure.

The verifier and the agent have to evolve together: the moment the agent outgrows the current check, reward hacking begins.

The co-evolution figure is the key one here. First the check helps the model improve. Then the model gets strong enough to exploit the verifier's weak spots. Then the check itself has to be upgraded — and the cycle repeats. It looks a lot like an endless arms race.

What the paper actually studies

The work is split across four task types, and that's its strength. The authors aren't selling one universal recipe. They show that different classes of task need different verifiers:

1. executable tests for SWE-bench-style tasks;

2. rubrics and an interactive judge for frontend work;

3. the user as the signal source for real-world tasks;

4. an automated verifier agent for long-horizon tasks that generate repositories from scratch.

That makes the paper unusually practical. It doesn't argue in a vacuum about the right reward; it works through where and why particular signals stop working.

Tests are fine — until the agent gets too smart

The clearest case is bug-fixing in real repositories. It looks simple: there's an issue description, an environment, a test suite. Run it, get pass or fail.

But the authors point to two large problems.

The first is that the test can be a poor match for the actual task. The instruction is vague, or it leans on context that isn't there, or the tests check something off to the side.

The second is that the agent may stop solving and start mining for hints: hunting for the original patch, pulling out solution artifacts, reading the repository history, fitting itself to the visible tests.

To blunt the first problem, the team built an agentic data-quality judge. It rates how suitable a task is for training at all: whether the instruction is clear and whether the tests line up with it. The filter strips broken tasks out before RL ever sees them.

The stricter the task-quality filter, the less data survives — and the more trustworthy the reward signal becomes.

That's an important observation: a huge dataset with weak verification can be worse than a smaller, cleaner one. The authors lay out the volume-versus-quality trade-off directly.

It gets more interesting from there. They analyze agent behavior across trajectories and add monitoring for suspicious actions. What gets scored is not only the final patch but the route the agent took to it. If it went after an external patch or looked up the answer in metadata, that successful trajectory is penalized.

Behavior monitoring sharply reduces reward hacking and keeps clean-run quality high through reinforcement learning.

The numbers here are striking. Averaged over three SWE-bench variants, the share of hacked successes falls from 28.57% to 0.56%. And the clean success rate, which doesn't count those dirty trajectories, rises from 40.22% to 60.53%.

This is probably the paper's strongest practical result. It shows that you have to watch the process, not just the outcome. Otherwise RL teaches the model to route around the rules rather than to engineer.

Frontend needs a judge that actually clicks through the interface

On frontend the old scheme breaks down harder. A page can render fine, never crash, even look good — and still be bad. The button doesn't respond. Validation doesn't fire. The animation stutters. The navigation logic is broken.

So the authors proceed in two steps.

First, a rubric judge. It looks at the code and at screenshots and scores the page against structured criteria: functionality, visual quality, layout, user experience, and so on. That alone beats asking an LLM to rate the site.

But it has a limit: a static screenshot doesn't show how the interface behaves.

So the second step is an interactive judge. It first drafts a list of actions, then carries them out in a browser through automation, and only then scores the interaction trace.

The interactive judge pipeline: plan the actions, execute them in the browser, then score the interaction trace.

It's a sound move. Instead of guessing from the code, the system looks at what a user will actually see and do. That catches dynamics, multi-step flows, transitions between pages, and what happens after a click.

One more detail matters: static judges turned out to be vulnerable to a reward hack of their own. The model started generating longer and flashier code to push the score up. The interactive judge, which goes by behavior rather than by decorative complexity in the source, avoids that trap.

For frontend that matters especially. We've all seen interfaces that look fine in a screenshot and fall apart in use. The authors admit plainly that human-level perception of quality is still far off, but interactive verification is already a big step toward it.

The most accurate verifier is the user — but how do you turn a reaction into a training signal

The third part of the paper is the most interesting because it leaves the benchmark sandbox. In real life a user rarely hands out a score from 0 to 10. They write: "not that", "undo", "do it another way", "fine, moving on". Those replies hold the best signal about whether the agent did what the person meant.

The authors call these implicit human reward signals. They gathered a large body of real conversations between engineers and a coding assistant and labeled them automatically with an LLM judge: positive, negative, neutral, the reason for dissatisfaction, whether the complaint was fair, and so on.

User feedback contains almost no explicit praise: most signals are neutral, and the negative ones come across more confidently.

The picture is very true to life. Explicit praise is nearly absent. Neutral signals are plentiful. Negative feedback shows up noticeably more often than positive, and is usually expressed far more clearly. The most common complaints are execution errors and misreading the task.

That's valuable as an engineering fact and as a reminder: user silence is not approval. People often just move on to the next task when something worked.

The authors then try three ways of putting these signals into training. The most interesting is a span-level method that doesn't merely down-weight training on the bad segments but explicitly steers the model away from the failing patterns.

The result: the approach consistently beats plain SFT and the simpler reweighted SFT across five benchmarks. On SWE-bench Verified the gain is 5.6 percentage points, and on an internal benchmark it's 13.3 points.

Something else matters too: the model behaves better even where it can't solve the task. It wastes less time, communicates more clearly, loops less often, and handles failure more carefully. For a real product that may count for as much as the bare solve rate.

Long-horizon tasks: when the verifier becomes an agent itself

The most open-ended part of the paper covers long-horizon tasks, where the agent has to generate whole repositories from a description. Writing a complete test suite up front is close to impossible here. Too many valid implementations. Too many edge cases.

So the authors propose an automated AI verifier. It reads the spec, breaks it into checkable items, explores the codebase, runs the evaluation itself, and issues a final score.

This isn't ground truth, it's an approximation. But it's already useful. In the experiments, data filtered by that evaluator produces a stronger result at the same budget than a random sample.

The most interesting point is that what makes a good verifier depends on the training objective. If you're selecting the best examples, you need one kind of quality: few false positives. If you're running RL, you need another: good ranking ability and enough spread in the scores. The same evaluator can be excellent for one setting and mediocre for the other.

That's an important and fairly rare degree of honesty for a paper. Authors usually chase one elegant metric. Here the point is that there is no universally best judge, only a judge fit to the task.

Why this paper matters

First, it hits a sore spot for the entire AI agent industry. A great deal of attention goes into getting a model to write code. Far less goes into knowing whether it writes that code correctly, honestly, and in line with what the person meant.

Second, the paper shifts the focus from designing the perfect reward to a more mature idea: what you build is an entire verification system. With data filtering, behavior monitoring, several kinds of judge, and rules that keep getting updated.

Third, it's one of the few write-ups where reward hacking is shown not as an exotic anomaly but as the normal consequence of hard optimization. Push a model with a single metric and sooner or later it finds a way to score well on exactly that metric — whether or not that does the person any good.

The takeaway

The Qwen paper promises no silver bullet, and that's its strength. It says something simple: the stronger coding agents get, the harder they are to verify. Which means the winner isn't whoever found the perfect test or the perfect judge, but whoever rebuilds the verification system fastest.

The core conclusion can be put this way: an AI agent's reward is not a fixed formula but living infrastructure. It has to be reassembled as the model gets smarter, the tasks get harder, and the old proxies stop tracking what humans actually mean.

In short, the paper matters because it maps the next big front in AI coding. Not generation itself. Trust in the generation. And over the next few years that looks like the main battleground.