A test built around decisions
Taste-Bench contains 502 questions drawn from thousands of evaluated agent runs on coding and AI research tasks. Each presents the original task, the work leading up to a point where possible paths diverge, and two ways to continue.
Some questions compare independent attempts at the same task. Others compare an agent’s initial direction with a later course correction that led to success. In both cases, the model sees what happened before the split, not which path worked out.
The benchmark’s premise is useful: final success scores can hide the quality of the decisions that produced them. Its method also has a clear limit. A model choosing between two recorded paths is not the same as an agent making decisions while doing the work.
The harder the evidence is to see, the worse the score
Each question was asked twice, with the options reversed. A model only received credit if it chose correctly both times, making random guessing worth 25%. GPT-5.6 Sol led at 59.7%, followed by GPT-5.5 at 59.5%. The other models scored considerably lower.
At a decision fork, the model must choose between two directions before the later trajectory reveals which one was better. | Image: Pan et al.
Source: the-decoder.com
The largest drop came when the decisive evidence appeared late. When earlier steps already contained a useful clue, average accuracy was 62.3%. When the clue emerged only after substantial further work, accuracy fell to 21%, near chance.
More reasoning did not solve the problem. Researchers tested three reasoning settings on two models; moving from the lowest to the highest barely changed accuracy. Models spent the most reasoning tokens on the hardest questions, but the authors’ explanation is that the evidence needed to choose correctly simply was not available yet.
Even the best model only picks the better direction in about 60 percent of cases on these binary decision questions. | Image: Pan et al.
Source: the-decoder.com
Training helped on unseen tasks
The researchers also tested whether the skill could be taught. They transferred judgments from a teacher model that knew the successful path to a Qwen3.6-27B student that saw only the question. On unfamiliar tasks, the student’s accuracy rose from 30 to 47.9%.
That improvement carried into a practical test. Researchers added the student’s recommendations at each decision point, then asked a separate agent to solve the task. On 41 held-out SWE-bench Pro tasks, the success rate rose from 14.6 to 33.7%. With perfect recommendations, the researchers estimate it would have reached 39%.
I think that result is more consequential than the leaderboard: a weak score can be a measurement problem, but the training result suggests the gap may be tractable. Still, the test is about recognizing a better path in recorded work. It does not establish that an agent can reliably find that path while working independently.
Source: the-decoder.com
The question is what the score misses
The benchmark’s broader claim is that long-term planning is a different skill from passing tests at the end. The authors say models with similar results on SWE-bench Verified can perform quite differently on Taste-Bench. They also point to earlier concerns about final-test benchmarks: METR found that experts rejected many coding-agent solutions despite their passing automated tests.
That matters more as agents work for longer. On MirrorCode, one model handled a single task for 19 days without human help. Yet the model leading Taste-Bench, GPT-5.6 Sol, also had the highest level of cheating on coding tests in METR’s evaluations.
My guess is that the uncomfortable question is not whether models can choose between two paths after the fact, but whether a benchmark can tell sound judgment from a convincing shortcut. Taste-Bench makes intermediate decisions visible; the gap between a better score and dependable autonomous work remains.
With guidance from the trained Qwen3.6-27B student, the agent solves more than twice as many SWE-bench-Pro tasks. | Image: Pan et al.
Source: the-decoder.com
Daily AI news
Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.
Only what matters — every day
Follow on X