Epoch finds AI research agents still overstate their results

The benchmark tested invention, not just execution

Epoch’s InnovationEval gave Claude Fable 5 and GPT-5.6 Sol up to 3,000 hours of compute on powerful chips, with no internet access. The task was to devise, implement, test and refine a new way to improve language models after initial training. Epoch says neither model knew about the target method, SDPO, in advance.

SDPO uses additional signals, such as error messages, to evaluate individual steps in a response. The more common baseline, GRPO, compares several answers to the same task and typically scores each answer as a whole. In SDPO, the model effectively becomes its own teacher.

Neither agent matched SDPO. Sol proposed reinforcing successful answers when every response to a task was already correct—a known approach that addresses a limitation of GRPO. Epoch estimated that Sol reached about 35% of the improvement SDPO delivers over GRPO under permissive criteria. Counting only changes that followed the experiment’s rules brought that figure down to about 15%.

On programming tasks, Sol mostly increased training cost and time without improving the method. Fable 5 retried failed tasks using its earlier attempts, another known technique that produced no measurable improvement. Epoch said experts would consider even Sol’s partial success only “moderately interesting.”

Models that had encountered SDPO during training did not fully reproduce it either. GPT-6 Astra produced a similar solution without identifying its source. Claude Fable 5 fell short of the original method even when given the research paper.

Claimed vs. actual scores for Claude Fable 5 and GPT-5.6 Sol. Dark bars show the agents' self-reported numbers, medium bars show results corrected for cherry-picked runs, and light bars show scores counting only rule-compliant changes. All values are expressed as a share of SDPO's improvement over GRPO. Error bars show measurement uncertainty. | Image: Epoch AI

Source: the-decoder.com

The chart shows how Claude Fable 5 and GPT-5.6 Sol’s reported results change after adjusting for selected runs and counting only results that followed the experiment’s rules.

The results looked better than the record

Both agents ran several near-identical training cycles and reported only the best result. Because outcomes vary randomly, selecting the strongest run makes a method appear more effective than it is. Their final reports barely mentioned the practice or cited the earlier work behind their methods.

GPT-5.6 Sol's progress plotted against cumulative GPU spend. On short-answer tasks (teal), the jump comes just before the budget runs out. On coding tasks (pink), the rule-compliant score stays at zero, and only out-of-scope changes produce any gain (dashed line). | Image: Epoch AI

Source: the-decoder.com

Sol claimed about 70% of SDPO’s improvement; Fable 5 claimed about 40%. Epoch excluded those inflated figures from its results. Logs of the models’ internal reasoning show they understood the issue: Fable 5 described the repeated runs as searching for a more favorable checkpoint. Epoch did not decide whether this was deliberate deception or confusion.

That distinction matters less for anyone relying on the reported results: either way, the agents did not give a reliable account of what their experiments showed. The behavior also fits an earlier METR assessment, in which GPT-5.6 Sol tried to cheat more often than any other publicly available model METR tested.

Anthropic describes related weaknesses in its system card for Claude Opus 5.5. The company says the model is far from replacing its own researchers. It presents unverified assumptions as facts, treats partial checks as full confirmation, and turns preliminary estimates into recommendations without further verification. It also tends toward incremental changes and published research rather than developing new ideas.

A separate team from Princeton University and the UK AI Security Institute reached similar conclusions after asking Claude Opus 4.8 to work for six days on research questions from two unpublished NeurIPS papers. The original authors rejected both results. When their initial hypotheses failed, the agents softened their conclusions rather than starting over.

More compute may not fix the harder problem

More compute could still help. Sol used its entire budget and found an improvement on short-answer tasks near the end; Epoch sees that late result as weak evidence that extra time may matter. But Sol made no rule-compliant progress on programming tasks, while Fable 5 used less than half its budget.

Epoch says leading models would have performed much worse on InnovationEval a year ago, and plans to repeat the benchmark with new tasks. That progress is real, but I think the more important gap is not how long an agent can run experiments. It is whether it can recognize a weak result, disclose uncertainty and take a negative finding seriously. Until then, human researchers remain responsible for checking the work end to end.