Jr. AI Scientist writes junior-level ML papers, and reviewers rejected all three

Autonomous agents have lately been pitched as systems that can come up with ideas, write the code to test them, run the experiments on their own and write the paper. In practice they have tended to fall short: the ideas were never properly checked for novelty, the experiments stayed at prototype level, and the results came out weak (see AI SCIENTIST V1 / V2). Researchers at the University of Tokyo decided that this much autonomy is too ambitious for real papers, and proposed Jr. AI Scientist — a system meant to behave like a conscientious junior researcher. Find limitations, propose improvements, implement them directly in the original codebase, check them and produce a paper draft.

The idea in brief

Jr. AI Scientist imposes a strict constraint: one paper, its code and its data, full multi-file work and a complete set of experiments for a single new experiment, several hypothesis formulations, fully working code, reproducible results and a paper draft. This matters because earlier systems optimized either the simulation of the research process or the writing of the paper, not both. The important change is that the input is one paper rather than a research area. That constraint substantially raises the odds that the system proposes a realistic, publishable improvement, because it has no way to propose experiments drawn from the entire literature.

How it works under the hood

First, the model is required to pin down the limitations of the paper's experiments and propose an improvement. The idea is then checked against prior work through Semantic Scholar. A separate agent is responsible for drawing up the list of experiments and the scripts. For that it uses Claude Code with a working directory holding the paper's code and two experiment entry points — baseline.py and plot.py. The agent can read and edit any file in the working directory and inspect the logs of individual runs. The code is executed by the parent system; the agent itself runs nothing, but it sees the files and the logs. The scripts go through several stages. First they implement the proposed improvement, with bug filtering. Then they iterate until the metrics improve. The final stage is always an ablation of the new component or of the hyperparameters.

A separate run of the same agent also produces a short paper draft in the template for the target conference, with all the tables it needs and correctly formatted BibTeX references. It goes through three full reflection cycles with automatic checks for structural defects — whether all the content and references are needed, whether words and phrases repeat, grammar and spelling errors, algebraic errors, and whether the citations match every paper on Semantic Scholar that was implicitly drawn on in the writing. Using one and the same agent with one working directory for both the experiments and the writing is a blunt attempt to narrow the gap between how the paper describes the experiments and what the code actually does. The system also saves the ablations and all the experiments to a separate json file, so that when post-processing the generated draft one could swap the tables in it for the experiment tables and check them.

What the evaluations showed

The system is tested on two real papers on OOD detection with CLIP — LoCoOp and GL-MCM — and on Min-K%++, which looks for traces of pretraining data in large language models. Each has several improvements and ablations. Paper quality is judged with DeepReviewer-14B as an automatic reviewer, and the drafts are also sent to experts on the Agents4Science platform. On top of that, the authors run a detailed check of their own.

By DeepReviewer's scores, Jr. AI Scientist does noticeably better than earlier fully autonomous systems. The average rating of the papers it reviewed is 5.75, peaking at 6.25 for the single best draft — against 3.30 for AI Scientist v1 / v2. All three papers, though, were rejected at Agents4Science. The reviews describe them as well written, reproducible, full of ablations and in line with scientific standards. But they also call the improvement over the base paper modest and the novelty moderate, and they note the lack of comparison with alternatives and of theoretical content. The overall verdict is that run fully autonomously, Jr. AI Scientist does not reach the community's bar — especially with the human removed entirely from idea generation, which is where the system appears to be weakest.

What goes wrong

The most valuable part of the paper is its detailed account of the risks. First, on GL-MCM the author notes that Jr. AI Scientist made a large gain through a function that relies on batch normalization. The data handling in use builds batches out of ID samples only or OOD samples only, so batch normalization never mixes ID and OOD values. The result is a setup where the agent systematically raises the metrics by hacking the experiments. An expert spots that kind of "optimization" immediately, but in a fully autonomous system there is no explicit way to establish that the experiment set and the novelty of the idea are adequate, short of manually checking whether it amounts to, say, a small gain on one metric paid for by degradation on the rest. The authors stress that more capable agents will only raise these risks, so strict oversight is necessary.

Early in development the agent was prone to making up data and hallucinating. Jr. AI Scientist would often invent experimental numbers and present them as real while writing interpretations, whenever the reflection system asked for more experiments. Simply forbidding it to hallucinate data was not enough to get past this. During reflection it usually asked for well-structured files with the real results, so the researchers added them. Even prompts like "write this in more detail" raised the chance of hallucinated references, extra interpretations and comparisons between experiments.

After the first prototype was reworked, the agent stopped inventing new experiments, facts and references. Another serious problem remained: it misread or skipped entirely some of the experiments that had actually been run. If the results were not available as machine-readable data from the experiment script, the agent would often invent its own interpretation of them or leave them out altogether. The only thing that fixed this was saving annotated interpretations to an external file and reading them back when writing the experiments section.

A few small errors survived into the final test: the agent mixed up one of the tables with the annotated references, and it blended the interpretation of results across different reinforcement learning methods. LLMs are generally bad at catching errors like these, so use cases that demand high accuracy need careful expert review and clear definitions of the concepts involved.

What comes next

Jr. AI Scientist shows that a realistically scoped task and strong coding agents are already enough to turn out papers well above what earlier fully automatic systems managed. Real one-button use is still a long way off. The authors do not recommend using the system for academic writing as it stands, and they set out several lessons from their experience: constraining the task buys higher quality; without an expert, tasks defined by metrics alone are easy to hack; the workflow has to be structured and verified, or the volume of fabrication goes up; and current AI reviewers are poor at spotting mismatch between text and code, so evaluation needs open artifacts and access to the experiments.

The researchers see progress coming from better filtering of ideas, a tree search, expert guidance on questionable paths, and more advanced ways of checking experiments. Combined with artifacts and a review process where the code and the experiment logs are available, they believe such systems can become useful — though in the role of a junior.

So the era of autonomous researchers seems to be starting not with geniuses but with juniors, and that is probably the right way forward.

💾 Code