Nonprofit METR has updated its chart measuring how long AI models can spend on tasks. New tests show that models are still improving quickly on tasks with verifiable results, such as programming and mathematics, but continue to lag behind people when judgment is required. In particular, none of the models produced an AI evaluation that METR staff considered useful for research. The newest models have come close to an acceptable level on this task, however. How durable that progress will be remains unclear: open-ended tasks are difficult both to design and to evaluate.
What METR’s chart measures
In the original version of the chart, METR evaluated models on a large set of tasks that people had completed in advance. The difficulty of each task was based on how long human testers took to complete it. Models were then evaluated on the duration of tasks they could handle. The results showed that every four months, the maximum duration of tasks successfully completed by leading models doubled.
The chart circulated widely, but was often misunderstood: the measurements METR used were complex. The organization has now added tasks that people are estimated to spend much longer on, as well as assignments with less clear-cut outcomes.
The difference matters for training models. On a verifiable task, it is possible to determine automatically whether a system succeeded. Researchers run the model many times, strengthen the connections associated with successful answers, and weaken those that lead to errors. This is how reinforcement learning works. The approach has helped models achieve strong results in programming and mathematics, where solutions can be checked automatically.
It is harder to apply this approach to tasks where success cannot be clearly defined or evaluation takes a long time—for example, choosing a new research direction. Such tasks are difficult both to learn and to evaluate. That is why they were absent from METR’s original chart, even though these are the kinds of abilities that may be needed to automate AI research and development.
New tests of judgment
The question of automating AI research became especially salient after another METR report, published earlier this month with extensive redactions. It claimed that AI could double the pace of AI development at Anthropic. The report raised concerns that the company, which recently completed a major initial public offering, could soon reach full recursive self-improvement: AI systems designing their own successors without human involvement.
For its new set, METR included tasks that require judgment and allow for different answers. In one test, models must come up with new AI evaluations that METR could use in its research. The organization’s staff then review the results. Although the most advanced models already outperform nearly all people at software development, they have not yet succeeded at this task: METR staff did not consider any model-generated evaluation useful for their research.
Another test examines how models assess philosophical arguments. METR uses it to study models’ ability to make sound judgments, though staff disagree about how important the task is to automating AI research and development. Here, too, models lagged behind people.
Computer scientist Sayash Kapoor, who is preparing to become a professor at the University of California, Berkeley, considers the move to more varied tests useful. In his view, evaluating AI requires more creative approaches. Kapoor was not surprised that models lag on open-ended tasks such as developing an AI evaluation: in a preprint published in August, he and his co-authors reported that models cannot yet independently reproduce human research in computer science.
On Wednesday, OpenAI also said in a blog post that its models perform worse on real-world tasks set by company employees than on the simpler, more verifiable tasks in METR’s original set.
What the results mean
There is no consensus on how to interpret the gap between verifiable and open-ended tasks. Kapoor believes the bottleneck to automating AI research will be “research taste”—the ability to choose promising questions, hypotheses, and experiments. Others think that training on tasks with easily verifiable results will eventually enable models to handle open-ended assignments as well. Progress on verifiable tasks has not yet slowed.
METR’s new data shows that models can improve on open-ended tasks, too. Some of the newest models, such as Opus 5.5, typically fall into the “just above passing” category when developing AI evaluations. METR does not yet consider these results suitable for its research, but a year ago all models were in the “clearly below passing” category.
The report includes few open-ended tasks, however: they are difficult to create and evaluate. Linguist and computer scientist Nayoung Kim of Boston University says the observed increase reflects the current state of models, but it is not yet clear whether it can be used to assess future progress.
METR technical staff member Matt Putz also notes that forecasting model development on tasks requiring judgment or allowing ambiguity will be difficult. In his view, it is harder to get clean and reliable trends for these tasks than for assignments with clear criteria.
The challenges also stem from how METR estimates task duration. For the original chart, researchers measured the time by asking people to complete each task. In the new report, durations are mostly estimates: paying people for hundreds of hours of work or finding the few specialists able to complete a particular task would take too much time and money.
Individual tasks have their own limitations, too. For example, most philosophical arguments were evaluated by one person, and models were compared with that person’s ratings. The results could have changed if more experts had been involved, but that would have taken additional time.
The experts interviewed for this article consider adding open-ended tasks to METR’s new chart a step in the right direction. Still, there are no definitive answers yet about whether AI research and development can be automated, or how quickly that might happen. Kapoor calls the transition promising, but stresses that much more work is needed to understand the situation fully.
Dataist’s take: The updated chart gives a broader picture of what models can do, but it does not yet provide a reliable basis for forecasting the automation of AI research. In particular, results on open-ended tasks rely on duration estimates and evaluations with limitations. For now, progress in this area cannot be confidently compared with gains in programming and mathematics.
Daily AI news
Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.
Only what matters — every day
Follow on X