Jack Clark moves his AI self-improvement forecast to 2027

What TasteVal measures

P-Zero Research defines research taste as three abilities: choosing promising scientific questions, designing experiments to answer them, and interpreting the results. TasteVal tests only the last two. The benchmark gives models a fixed research goal, so they do not have to decide which line of inquiry is worth pursuing.

Its eight tasks cover stages of AI model development, including training-data selection, pretraining and fine-tuning language models, training on human preferences, and defense against adversarial prompts. A model chooses the next experiment, then uses the result to decide what to do next.

To keep coding ability from affecting the score, every experiment is run by the same coding agent on one Nvidia H100 GPU. Each run is limited to 40 GPU hours or 120 hours of real time.

The researchers compared models with 24 experts who had recently worked at places including OpenAI, Google DeepMind, Nvidia, and Microsoft Research. At least two people worked on each task; the best result for each task became the human benchmark.

A striking result with a wide range

TasteVal measures how efficiently a model uses computing resources. The premise is that research budgets are constrained: choosing experiments well and learning from their results should make it possible to achieve more with fewer runs.

P-Zero Research’s “AI Future Model” argues that once coding is automated, computing resources for experiments will become the main constraint. TasteVal therefore measures how much compute a model needs to match expert performance. The lab calls the ratio a compute multiplier.

Anthropic’s Opus 5.5 led the test with a multiplier of 2.3, reaching expert-level results with less than half the computing resources. But the statistical range is broad, from 1.15 to 4.37. At the low end, the model would only slightly outperform the experts. One model run cost roughly thirty times less than an average expert attempt.

Across several generations of models, the multiplier roughly doubled every three months, starting in December 2025. That estimate is uncertain too: the reported doubling time ranges from 1.7 to 5 months.

The trend appears to have steepened around the turn of 2025 and 2026, after years of slower progress. The authors warn that one benchmark cannot establish whether the pace will continue, and that TasteVal may overstate improvement. Its tasks are quick and easy to verify, making them easier for AI labs to optimize. The comparison also did not include top-tier researchers.

According to P-Zero Research, the research taste of leading AI models has roughly doubled every three months since December 2025. | Image: P-Zero Research

Source: the-decoder.com

The missing judgment

TasteVal bears on one of the disputed prerequisites for a major AI breakthrough: whether models can plan and analyze experiments. AI researcher Herbie Bradley argued in response to Clark’s May essay that models might take on junior researchers’ work, while research taste, creativity, and sustained long-term planning remained beyond them.

TasteVal challenges that view for experiment planning and analysis. It does not address the choice of a long-term research program, which is where the benchmark draws its boundary.

A study of AI-assisted development by a team at Fudan University offers a useful comparison. In the development of the team’s own model, AI agents proposed up to 55% of methodological decisions, while humans still made more than 85% of final decisions. The authors said it remains unclear how AI can generate diverse research directions and judge their promise before results are available. TasteVal does not test that ability.

Clark has long described creativity as the key factor in how quickly AI systems could improve themselves. Earlier this year, he wrote that models had not demonstrated the ability at a scale capable of producing a breakthrough. In a May essay, he put the odds of a system independently training a stronger successor at about 60% by the end of 2028 and 30% by 2027. His latest shift comes as he points to progress in mathematics and benchmarks such as TasteVal. He also co-founded the company whose model topped the test.

The debate moves into research work

The benchmark arrives amid a debate the leading AI labs have joined. In June, Anthropic wrote that recursive self-improvement had not yet been achieved, but could arrive sooner than most organizations were prepared for. CEO Dario Amodei now calls for AI development to be slowed in a controlled way. OpenAI has also said fully autonomous recursive self-improvement remains out of reach and should be pursued only with guaranteed human oversight.

I think TasteVal’s importance is less that it forecasts a date than that it shifts the test of progress: from solving a well-defined problem to making decisions that move research forward. But the benchmark’s central omission is also the hardest part of the claim. A model may get better at running experiments without knowing which questions are worth asking.

If P-Zero Research’s trend holds up across other tests, AI development could become faster and cheaper. The corresponding tension is that researchers may have a harder time checking an expanding share of the work as models plan experiments and assess results themselves.