GPT-6-Astra solved 49 of 50 theoretical physics problems for $116

The test was built around the work between big questions

TheoPhysEval was developed by 35 researchers from the International Centre for Theoretical Sciences, Harvard University, KU Leuven, Google DeepMind and other organizations. Its 50 problems cover 11 areas of theoretical physics, from fluid flows and biological processes to particle physics and cosmology.

The authors deliberately left out famous unsolved problems. Instead, they tested work that often applies established methods to a slightly altered problem — the kind of exercise that can serve as a graduate student’s first step into research. Researchers sometimes changed values from calculations in their own recent papers so an answer could not simply be retrieved from the original work.

One black-hole problem gives a sense of what “routine” can still involve: find an older paper, translate equations from five-dimensional to four-dimensional spacetime, then use specialized software to calculate the result. Project coordinator Suvrat Raju, a co-author of the paper, spent about a day solving it. The authors say that, before modern agents, only a few specialists would have combined the necessary physics knowledge and computational experience, and even they might have needed several days.

The strongest systems were:

Each agent took the test twice: once with internet search and once without. Search made little difference. Astra and Opus each scored 92% without it, and the authors suggest that leading models may have memorized a substantial amount of theoretical physics literature.

The benchmark’s own answer key needed checking

Some of the agents’ apparent mistakes turned out to be mistakes in the benchmark’s reference solutions. After specialists reviewed several problems, the researchers corrected subtle errors; Astra’s answers in those cases were then found to be right.

That matters for interpreting the headline score. The authors also say TheoPhysEval produced better results than some other physics benchmarks, including CritPt, despite their view that its problems are harder on average. A recent study found that some theoretical physics benchmarks misgrade answers and can substantially underestimate models’ capabilities.

The remaining problem resisted every model. It asked agents to recover an equation that could generate a given curve. The specialists’ solution relied on mathematical forms that only broadly qualify as the requested differential equation. The authors acknowledge that the problem may seem “unfair,” but argue that it resembles real physics, where researchers often have to infer what produced imprecise data.

The dataset and testing setup are available on Hugging Face.

Calculation is not the same as research

The authors argue that some current models may already outperform leading human researchers on particular calculations, including in specialized areas. They call that a major milestone. But in mathematics, AI agents have also solved important open problems; OpenAI says one example is the Navier–Stokes problem on fluid motion, one of the Millennium Prize Problems. The authors say theoretical physics has not yet seen a comparable breakthrough.

Their explanation is that a verifiable calculation tends to come at the end of a research process. Before that, someone has to build a theory and ask the right questions. When a problem proves too difficult, a good researcher may shift to a related, more tractable question that remains interesting. Current agents do not yet make that kind of move.

I think that distinction is the most useful result here. A score of 49 out of 50 measures the ability to execute a carefully framed task; it does not measure the judgment that decides which task deserves attention. The paper’s authors say that physicists have a shared, though far from unanimous, sense of what makes a problem interesting, while agents can get stuck generating details a person would find dull. They stress that these observations are not quantitative predictions about where AI is heading.

The result is already changing how some of these problems can be used in training: tasks that might once have served as a beginner’s first independent project may no longer be appropriate for that purpose, unless the goal is learning the material.

The cost also points to a tension. Astra’s full run cost just under $116, but the authors expect spending to rise sharply for harder problems. Theoretical physics has historically required far less money than experimental physics, helping keep opportunities comparatively equal. If research with capable agents becomes more expensive as the problems get harder, a group’s budget could matter more even in a field once known for its lower costs.

The authors argue that theoretical physics should begin debating access and fairness before agents reshape the field. If calculation becomes cheap but research judgment remains scarce, the advantage may shift from knowing how to solve a problem to having the resources — and the insight — to choose one worth solving.