How the evaluation worked
Each agent received a scientific paper, its code and data, and a direction for further research. Some source papers had already been published; others had only been submitted. The agents were asked to carry out follow-up research and present their findings as a scientific paper. The evaluation covered 15 tasks proposed by ten researchers in quantum computing, plasma physics, chemistry, materials science, and machine learning for science. The authors of the original papers chose the research directions; according to them, no research on those topics had yet been published.
Open scientific questions have no known answer against which results can be checked automatically. The researchers consider AI reviewers unreliable, so experts set minimum conditions for a successful paper in advance: it had to be honest, complete, and meet their requirements.
GPT-6 Astra evaluated the papers, with each one checked four times. The test included Claude Opus 4.8, Claude Opus 5, Claude Fable 5, Claude Opus 5.5, GPT-5.6 Sol, GPT-6 Astra, Deepseek V4 Pro, Kimi K3, and GLM-5.3. The agents were given powerful computers and internet access, had no time limit, and were allowed up to five restarts to improve their results. The tests took almost 2,000 hours in total.
Experts define the research direction and minimum requirements for each task, which then become rubrics for quality and honesty. | Image: Kim et al.
Source: the-decoder.com
Each agent receives an original paper with its code and data and is expected to produce an independent follow-up study. | Image: Kim et al.
Source: the-decoder.com
Only three agents were honest
Only three agents produced papers with no integrity violations on at least some tasks: GPT-6 Astra on 11 of 15 tasks, GPT-5.6 Sol on six, and Claude Opus 5.5 on four. Every paper produced by the other six agents contained at least one violation.
The most common problems were discrepancies between papers and code, figures that contradicted one another, and claims of experiments that had not been run. The open models Deepseek V4 Pro, Kimi K3, and GLM-5.3 often reported incorrect baseline values. In two papers by Kimi K3 and GLM-5.3, results had been written in advance rather than calculated.
Only one paper was complete: the one produced by GPT-6 Astra. The others almost always omitted required materials, such as a preregistered experimental plan. No agent met the experts’ requirements. Under a more lenient evaluation, which counted 90% of the rubric items as sufficient, GPT-6 Astra came closest to passing, followed by Claude Opus 5.5 and GPT-5.6 Sol. Open models performed noticeably worse overall.
GPT-6 Astra scores the highest partial marks, but none of the 135 submissions meets all three criteria at once. | Image: Kim et al.
Source: the-decoder.com
Most integrity violations stem from numbers that contradict the agents' own logs and from mismatches between paper and code. | Image: Kim et al.
Source: the-decoder.com
Agents were too lenient when checking their own work
The researchers reviewed 64 records of agent activity and identified three recurring failure modes. In 58 cases, an agent checked its own work too leniently. In 34 cases, it noticed a problem but only described it, without fixing it. In 21 cases, it narrowed the task without authorization.
Claude Fable 5, for example, did not run four planned experiments. Its verification script used the same incomplete list of runs and reported that all 41 of 41 had been completed. The agent trusted this check to the very end. In another task, GPT-5.6 Sol realized that its method could not achieve the goals but wrote a paper anyway instead of starting over. DeepSeek V4 Pro abandoned its experiments, citing a lack of time, even though there was no time limit.
There were also counterexamples. In a quantum physics task, GPT-5.6 Sol found a fundamental error, stopped the experiments underway, and recalculated the results from scratch.
Claude Fable 5 misses four absent runs because its verification script adopts the incomplete list, while GPT-5.6 Sol writes the paper despite missing its targets. | Image: Kim et al
Source: the-decoder.com
More time did not improve results
The time spent working did not predict quality. DeepSeek V4 Pro and GLM-5.3 worked longer than any other agents: their median times were 19.5 and 13.9 hours, respectively, but their results were weak. GPT-6 Astra finished in an average of 1.3 hours and achieved the best result.
Because the strongest model among the participants evaluated the papers, the researchers also asked the original paper authors to review anonymized submissions. The automated assessment of whether a paper followed the assigned research direction matched the experts’ judgments in every case. Assessments of scientific quality matched in about two-thirds of cases. One expert, for example, noted that the papers lacked a coherent narrative. In the researchers’ view, such judgments remain the work of human reviewers.
GPT-6 Astra has the shortest median runtime and delivers the best results. | Image: Kim et al.
Source: the-decoder.com
The study’s code and data are publicly available, and the project page includes a table of results for the evaluated models.
Research and tasks with clear-cut answers
Agents are becoming increasingly capable of carrying out scientific tasks independently, but a substantial gap remains between conducting experiments and doing independent research. A study published in August had already noted agents’ difficulties with unpublished scientific questions. Epoch AI and Anthropic have also found that agents tend to present results in an overly favorable light and assess unsuccessful approaches without enough critical scrutiny.
The picture is better on tasks with clearly verifiable answers: GPT-6 Astra solved 49 of 50 unpublished theoretical physics problems. Their solutions could be checked definitively. An analysis of more than 700 tasks in AI development also found that agents are taking on more practical work, while people still make the key research decisions.
Dataist’s take: The Into the Unknown results show that carrying out experiments independently does not yet guarantee reliable scientific conclusions. Agents may fail to notice work they have skipped or mistake their own errors for confirmed results. Checking whether work follows a given research direction proved automatable, but assessing scientific quality still requires experts.
Daily AI news
Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.
Only what matters — every day
Follow on X