Reading the full score distribution makes an LLM a better verifier

When an LLM learns to check, not just to answer

Large language models have an odd weakness. They keep getting better at generating solutions, but they are still not very good at telling which solution is actually right. That is not a small gap. If your AI agent writes code, works in a terminal, drives a robot arm or handles medical data, the question that matters is not "can it produce a candidate?" but "can we reliably pick a good one out of several?"

That is the premise of LLM-as-a-Verifier, a paper from researchers at Stanford, Berkeley and Nvidia Research. The authors argue that checking should be treated not as a support function but as a scaling axis of its own. Models can be improved not only through pretraining, fine-tuning and a larger inference budget, but also through smarter verification.

It sounds simple. The idea underneath it is not: if a model can verify reasonably well, it can choose better among its own attempts. System quality then rises with no additional training at all.

What goes wrong with ordinary LLM judges

The popular approach today looks like this: ask an LLM to act as a "judge" and score a solution, say from 1 to 5 or from 1 to 10. Then take the best candidate.

The trouble is that this is too coarse. When two solutions are both almost right, the model often hands them the same score. Ties pile up. And when the task is long and complicated — an agent spending 20 steps editing code, installing packages, running tests and fixing errors — a coarse scale starts to break down.

The authors propose moving from the idea of a judge to the idea of a verifier. A judge delivers a final verdict. A verifier tries to estimate carefully how close a solution is to being right, and where the signs of success or failure sit inside it.

The key move here is elegant. Instead of taking the single most likely score token, the authors look at the whole probability distribution over score tokens and compute its expected value. The model does not just say "this is a 4 out of 5"; it produces a smoother signal, something like "this is a 4.37."

That alone separates candidates far more finely.

One scheme across modalities: the verifier gives fine-grained feedback for code, video, medicine and reinforcement learning.

What the authors actually propose

Their scheme is called LLM-as-a-Verifier. It requires no additional training. That matters. The authors do not train a separate reward model for each domain; they take an off-the-shelf LLM or vision-language model and use it as a general-purpose checking module.

The method rests on three ideas.

The first is a finer scoring scale. Not 1–5 but up to 20 levels. And what matters is not only the number of levels but the fact that the probability distribution across them is used.

The second is repeated checking. A single pass can be noisy. Several independent scores cut the variance.

The third is criterion decomposition. Instead of one question — "is this correct?" — the model looks separately at things like:

This mirrors a sane human review process. When we assess complicated work, we do not eyeball a grade either. We break it into parts.

Verification accuracy improves along three directions at once: a finer scoring scale, repeated checks and criterion decomposition.

The results look convincing. On Terminal-Bench V2, pairwise comparison accuracy rose:

This is not a cosmetic improvement. When the job is picking the best trajectory, even a few points on top is a lot.

Why this matters in practice

The paper's central message is this: many models already hold quality in reserve. They can often solve a task in at least one of several attempts. What the system lacks is a good mechanism for recognizing that lucky attempt and picking it.

The authors demonstrate this on Terminal-Bench. If a perfect "oracle" existed that always picked the best answer from a set of candidates, accuracy would climb to almost 99%. That is an enormous reserve. It means the bottleneck is often not generation itself but selection.

To keep selection from getting too expensive, the authors add one more component: a candidate-ranking algorithm that works under a bounded checking budget. They call it a probabilistic tournament with anchor candidates. The idea is simple: rather than comparing everything against everything, which is costly, first sift out the strong candidates quickly, then spend the bulk of the budget on comparisons around the leaders. That drops the cost from quadratic to nearly linear in the number of candidates, with a small constant factor.

The probabilistic tournament: how to pick the best candidate out of many without comparing every pair head to head.

For AI agents this matters especially. A coding agent can usually make several runs at the same task. Once you have a reliable verifier, the system starts working in a "generate several solutions, then take the best" mode. That is a very practical route to higher quality.

Results: code, robotics, medicine

The strongest part of the paper is the breadth of the experiments. The authors did not stop at one benchmark in one domain. They tested the method on code, on video from robotics tasks and on medical scenarios.

The final numbers look like this:

Final results: the method sets the current best result in coding, robotics and medical tasks alike.

The RoboRewardBench case is especially telling. There the model has to judge which of two robot video trajectories is making better progress toward the goal — that is, read physical progress off the frames. And here the proposed approach beats not only an ordinary LLM judge but also specialized reward models trained specifically on robotics data.

That is an important point. We are used to assuming that a domain like this goes nowhere without narrow, dedicated training. The authors show that a strong general-purpose verifier can sometimes do better.

Not just picking an answer, but tracking progress

Another interesting part of the paper is the idea of using the verifier's signal as a progress estimate along a long trajectory.

If an agent solves a task step by step, you can feed the verifier not only the final result but intermediate prefixes of the trajectory. Then you can see whether the agent is moving toward the goal or wandering.

The authors give a nice example with a task about getting MNIST inference running. The successful trajectory follows a coherent chain of steps, and the verifier's scores climb steadily. The failed one veers off, installs packages it does not need, runs out of disk space, and its scores are visibly lower.

The verifier's score rises as a successful trajectory advances, which helps separate real progress from wandering.

This is no longer just a tool for picking the best of five answers. It is close to a dashboard for an AI agent. You can watch when it is genuinely making headway and when it is stuck. For long coding-agent runs that can be very useful: stop the process early, roll back bad actions, switch to another candidate.

What this gives reinforcement learning

The most unexpected piece of the work is using the verifier inside reinforcement learning.

One of the perennial problems there is sparse reward. The agent usually learns it did well only at the very end. Working out which steps actually led to that success is hard.

The authors use the verifier's signal as a dense reward: not just success or failure, but a smoother estimate of progress at intermediate stages.

In experiments this sped up training:

A dense reward from the verifier speeds up reinforcement learning and reaches the same quality level sooner.

This does not look like a magic jump, but it does look like a very practical thing. If you can get a meaningful progress signal without training anything extra, that takes real cost out of tuning a system.

Where the approach is weak

The work is strong, but it has limits.

First, the method wants a model that will hand back token log probabilities. Many closed APIs will not. The authors offer a workaround: one model writes the reasoning, and a second, more open one turns that into a continuous score. It works, but it is a crutch all the same.

Second, the checking criteria here are often written by hand. For code that is still fine: requirements, output, errors. But new domains will need their own decompositions worked out from scratch. The obvious next step is for the model itself to build criteria dynamically.

Third, repeated checks and a finer scale improve things but do not remove the model's systematic biases. If the verifier reasons wrongly about some class of task, plain averaging will not save it.

Even so, none of these limits spoil the paper's main conclusion.

The takeaway

The paper offers a well-timed shift of focus. We are used to arguing about how to make generation stronger. The authors remind us that teaching the system to check well matters just as much.

And this is not philosophy but a workable engineering idea. Instead of a coarse judge, a probabilistic verifier with a continuous score. Instead of one score, scaling along three axes: granularity, repetition, criteria. Instead of an expensive full tournament, a leaner way to surface the best candidates.

The main thing is that all of it already pays off in practice: better quality in coding, in robotics, in medicine, and even more efficient reinforcement learning.

Put briefly, the paper's thought is this: the next gain in AI agent quality may come not only from how they think, but from how they check themselves. And that looks like one of the more useful ideas going for systems that have to act reliably, not just talk well.