The team behind ALTK-Evolve has published a method for a number that almost no agent benchmark prints: how often an agent succeeds every single time, rather than on average. On AppWorld's test_normal split, a ReAct agent running on GPT-4.1 completes tasks in 77.4% of runs across five repeats. Across all five runs it completes only 53.0% of them. The 24.4-point difference between those figures is what the authors call the consistency gap, and their new consistency guidelines cut it to 12.0 points. The full method and evaluation are in a technical report on arXiv.
The distinction the work rests on is small in notation and large in consequence. Mean@k runs a benchmark k times and averages the share of successful completions; k is usually 3, sometimes 1. That is the number that reaches leaderboards and becomes the phrase "77% accuracy." Pass^k is the share of tasks the agent completed successfully in every one of the k runs. It is not Pass@k, the familiar optimistic metric that asks whether at least one of k attempts worked — useful when you can check a result and retry, useless when you cannot. The ordering always holds: Pass^k is at most Mean@k, which is at most Pass@k.
For a demo, a failed repeat is awkward. For a deployed system it is a reliability problem: a process that succeeded once may fail on a repeat of the same request. On a financial reconciliation, or a contract check for the presence of an obligation, that stops the process entirely. Almost a quarter of the AppWorld benchmark consists of tasks this agent solves sometimes and not other times, with the task itself unchanged between runs.
The cause is not capability. Every time an agent picks an API call, an argument, or decides whether to retry, the model samples a next token from a probability distribution, and the shape of that distribution decides whether the choice is stable. When probability mass concentrates on one token and the rest trail behind, the same decision comes out every run. When several tokens sit close together, the winner is nearly arbitrary. Floating-point behavior on GPUs, request batching and other platform-side effects nudge the numbers slightly — not enough to unseat a clear leader, easily enough to flip a near tie. An agent trajectory contains dozens of decisions, so a small per-step flip probability compounds into a high probability that at least one run diverges.
Decoding settings do not fix this, and the authors are explicit about why: greedy decoding and a fixed seed govern how a distribution becomes a token, not what the distribution is. On a served endpoint the probabilities drift a little between calls, so the same prompt to the same model at temperature 0 can resolve a close contest one way today and another tomorrow. The ReAct agent in these experiments ran at temperature 0.0. The variability was not ordinary sampling noise.
The diagnostic tool, Consistency Analyzer, takes one recorded trajectory and re-examines each decision point, measuring how much the model's answer moves. It costs one extra model call per step, run once offline, with a sampling parameter that requests k candidate responses — five by default. It reuses the recorded context: no tool calls, no environment, no end-to-end rerun, no reference answer. Each step gets a stability score, and the scored table marks the decisions likely to change on the next run. The method is black box — no logits, no model internals, no extra instrumentation beyond the trajectory you already have. Flagged steps then become candidate consistency guidelines in ALTK-Evolve's existing format, so they are stored and retrieved through the pipeline that already injects guidelines at inference time.
The example the authors show was generated by GPT-4.1 from a trajectory of the AppWorld task "How many actions have been completed in my to-do list according to the SimpleNote note?" The first guideline: when counting checkbox markers in note content, use a regular expression anchored to the start of the line rather than a plain substring count, because note headers often repeat the marker character in an explanatory line. The second: always verify note search results — confirm that several matches were found and identify the right note before continuing. Neither is specific to that task. Line-counting errors and unverified search results show up as unstable decision points across many AppWorld tasks. Because the analyzer looks for instability rather than failure, it can flag a step the agent happened to get right and would probably get wrong next time. In a two-minute demo, five parallel runs split three to two on the choice of counting strategy; with the guidelines in context, all five agreed.
The evaluation covers 168 AppWorld test_normal tasks, with guidelines generated from a single source trajectory per task and then tested on five fresh runs. Aggregate Pass^5 rose from 53.0% to 69.0% and Mean@5 from 77.4% to 81.0%, halving the gap from 24.4 to 12.0 points. Nearly a third of the previously unstable tasks began passing on every run. Medium tasks gained 22.9 points (+44% relative), hard tasks 14.3 points (+45%), easy tasks 12.2 points, where there was least room. Holding Mean@5 was treated as a requirement rather than a bonus — raise Pass^5 by lowering Mean@5 and you have only moved the instability elsewhere — and average accuracy held or improved at every difficulty level.
Two transfer results matter more than the headline. Applied to a different task in the same AppWorld scenario, the guidelines raised Pass^5 by 13.0 points, only 3 points short of the gain on the source task. On the weaker gpt-oss-120b, Pass^5 on the same task rose 6.0 points, from 10.1% to 16.1%, and on a similar task by 8.7 points — more than on the source task, which suggests the guidelines capture recurring failure patterns rather than memorizing one trace.
Here is where I would push back. The transfer evidence is real but narrow: the strongest generalization shown is to another variant of the scenario the guideline came from. Every reported gain depends on already having run something close to the task at least once. That is per-scenario tuning with a retrieval layer on top, not a general reliability property, and the evaluation never reports what happens when a request arrives with nothing related in the store — which is the case the authors themselves say motivates the work, since production tasks often cannot be repeated end to end even once. The gpt-oss-120b figures deserve the same scrutiny: 16.1% Pass^5 is a much better number than 10.1% and still not a system anyone should reconcile a payment with. The gap is a separate axis from capability, as the authors argue, but a narrow gap on a weak model is not reliability either.
There is also something odd about where the fix lands. The causes named in the analysis — GPU floating point, batching, drift on a served endpoint — all live on the serving side, below anything a developer controls. The remedy is text added to the context window. It works, and it works because it sharpens the distribution at the specific points where it was flat, but nothing here stops those distributions from being flat in the first place. Consistency Analyzer is a way to find out which of your agent's decisions your model provider might quietly change for you tomorrow.
The practical advice is short. Publish Pass^k alongside Mean@k, because an average cannot distinguish a reliable agent from a lucky one, and even k=3 will expose a gap you did not know you had. Expect the gap to widen with difficulty; on the hardest tasks a single averaged number misleads most. Do not reach for a larger model first — a stronger model raises Mean@k without necessarily closing the gap. Consistency Analyzer and consistency guideline generation are now in the open-source ALTK-Evolve toolkit at github.com/AgentToolkit/altk-evolve.
That first recommendation is the one with teeth. Every agent accuracy figure in circulation today is the optimistic one, and the authors are asking an industry that sells reliability to start publishing the number that makes its systems look roughly 24 points worse.