Let’s see whether AI agents can make discoveries in physics.

Physics Is More Than Equations

Solving a physics problem is not the same as doing research. Real research means finding relevant papers, understanding which questions remain open, choosing a method, checking the calculations, and explaining what the results show. AI agents are now being trained to handle all of this. But how can we tell whether they can do research—not just answer questions and produce polished demos?

The authors of FrontierPhysics assembled a benchmark of 56 research tasks created by physicists based on their own work. Agents must go from reviewing the literature to producing a scientific paper and results that can be checked programmatically. Twenty-two models took part in the evaluation. The best result was 27.4% successful attempts.

Models often know the relevant physics and can run calculations. What they lack is rigorous verification of their own conclusions—and the experience that helps a researcher understand what matters in a particular situation.

Tasks From Real Research

Most physics benchmarks ask a clearly defined question with a known answer. That’s useful for testing knowledge and calculations, but it says little about whether a model can handle research where it first has to figure out what to investigate.

In FrontierPhysics, each task starts with a broad project description, much like the brief a lab supervisor might give a new team member. The agent isn’t handed a ready-made method. It has to study the topic, devise a plan, carry out calculations or experimental work, and then present its results.

Each task requires two kinds of output. The first is verifiable data—for example, calculated quantities, simulation results, or processed measurements. The second is a scientific manuscript describing the context, method, results, and conclusions. Getting the numbers right isn’t enough: the paper must show where they came from and what conclusions they support.

The tasks cover nine areas of physics, from condensed matter physics to astrophysics. They were proposed by 42 researchers, and 56 were selected for the benchmark from 182 submissions. Before acceptance, each task was reviewed for clarity, physical soundness, and its fit with real research practice. The authors also asked submitters to show that the original project had required at least two weeks of work. By their estimate, developing the projects behind the benchmark took more than 8,000 hours in total.

How FrontierPhysics works: physicists create and review tasks, agents conduct the research, and automated tests and expert criteria evaluate the results.

How Research Is Evaluated

The authors use two independent forms of evaluation. The first is automated testing, which compares an agent’s numerical results with reference values to determine whether its calculations are correct.

The second evaluates the work process and the quality of the paper. It uses criteria written by the task authors to check, for example, whether the agent found the relevant research, accounted for real equipment constraints, justified its choice of method, and supported its conclusions with data.

The criteria are divided into mandatory and additional requirements. Mandatory criteria act as gates: the agent must meet all of them, along with passing the automated tests. If it fails any of these requirements, the attempt does not count as successful—even if parts of the paper look convincing. Additional criteria assess the quality of the research, such as how well the agent explains its method or connects its results to previous work.

This distinction helps separate two different problems. An agent might get the right numerical answer but write a weak paper that doesn’t explain why the result matters. Or it might produce a coherent paper whose calculations don’t support its conclusions. In the first case, the work isn’t persuasive enough; in the second, it’s scientifically wrong.

Results: Getting the Numbers Right Isn’t Enough

The evaluation compared 22 models across 26 configurations. Each configuration attempted all 56 tasks three times. The leading models ran in different software harnesses that give agents access to tools and let them work through tasks step by step.

Claude Opus 5.5 came out on top, succeeding in 27.4% of attempts—46 out of 168. It was followed by GPT-6 Astra, with 23.2% in its standard configuration, and GPT-6.1 Sol, with 19.6%. No configuration succeeded on even a third of all attempts. The results show just how far solving individual problems is from carrying out a full research project.

The overall success rate doesn’t show where agents go wrong. So the authors separately analyzed 168 GPT-6 Astra attempts. Three problems came up most often:

🔎 Insufficiently rigorous verification. An agent may check a result using a method that relies on the same mistaken assumption that led to the result in the first place. That kind of check won’t catch the error. Another common failure is overlooking rare cases: a method may work on some data but fail the hidden tests.

🧭 Lack of expert intuition. An agent may understand the laws of physics but not know which mechanism to investigate first. In one task, for example, it might fail to consider the decay of excited states as a source of experimental error.

📚 Difficulty with the scientific literature. An agent may miss a key paper, struggle to connect sources, or fail to use them to support its conclusions.

The authors flagged insufficiently rigorous verification in 48% of attempts, lack of expert intuition in 34%, and problems with the literature in 27%. The categories could overlap: a single attempt could fail for several reasons. Following instructions was also a challenge, cited in 26% of failed attempts.

The gap between doing the physics and drawing a research conclusion is especially telling. An agent may calculate a quantity correctly but fail to explain why it matters, overlook a limitation of the method, or miss that the calculation alone doesn’t test the assumption at issue. Knowing how to calculate and knowing how to do research are different abilities.

Checking the Checkers

When results depend on automated tests and evaluation criteria, it’s not just the agent that can make a mistake. An evaluator may reject a correct solution because of an overly strict tolerance, an ambiguous instruction, or a flawed assessment of the paper.

The authors specifically looked for such cases. Physicists reviewed the agents’ work and compared their solutions with the results of the automated and expert evaluations. They found incorrect assessments in nine of GPT-6 Astra’s 168 attempts: in eight cases, the agent had produced a correct result but was rejected because of a flaw in the task or evaluation; in one case, the evaluating agent was at fault.

After correcting these assessments, GPT-6 Astra’s success rate rose from 14.9% to 20.2%. That’s a substantial difference—and a reminder that open-ended scientific tasks require not only accurate tests but also scrutiny of the evaluation methods themselves.

The team also tested the criteria on manuscripts by the task authors. For 33 tasks, they used the original scientific materials where available. Those materials scored an average of 96.2% on the applicable criteria. This check doesn’t guarantee that every evaluation is correct, but it helps confirm that the criteria recognize high-quality research.

Conclusion

FrontierPhysics tests whether AI agents can go from reviewing the literature to producing verifiable results and a scientific paper. So far, they rarely manage it: the best success rate was 27.4%.

According to the authors’ analysis, agents are more likely to fail because they don’t rigorously verify their conclusions, lack expert intuition, or review the literature poorly than because they don’t know the physics or make calculation errors. Giving a model more tools or a larger compute budget isn’t enough for scientific work. An agent must be able to test its own assumptions, connect its results to earlier research, and recognize which limitations matter.

The benchmark doesn’t yet measure whether an agent can independently choose an entirely new research question and make unexpected discoveries. But it can test whether an agent can do more than produce an answer—whether it can show that the answer deserves to be trusted.