Testing beyond the training data
Thompson presented at MIT last week on how computing and AI are developing, arguing that AI could reshape several areas of science. In the new study, he worked with graduate student Gabriel Manso and research associate Claudio Meirink Verdu to test whether OpenAI models could solve theoretical problems from scientific papers.
The researchers tested o1, o3, GPT-5, GPT-5.1, GPT-5.4 and GPT-5.5 on problems published online between December 2025 and May 2026. The papers containing the solutions appeared after GPT-5.5 and the earlier models had finished training. Researchers used the published work as a benchmark and blocked internet access, so the models could not look up solutions.
The scope was deliberately narrow: the researchers selected fields where models could reason without simulations or experiments. That makes the study a test of theoretical problem-solving, not a demonstration that AI can carry out science in general. But it does extend the debate sparked by recent mathematical results into fields that also rely on mathematical and theoretical reasoning.
Mathematics has already raised the stakes
Last month, OpenAI said an unreleased model had solved a longstanding, difficult problem involving the Navier–Stokes equations, which describe fluid motion. The announcement drew controversy after a New York University mathematician, working with another researcher connected to Anthropic, accused OpenAI of getting ahead of them and announcing a solution after learning of their progress.
OpenAI and other organizations later reported hundreds of new solutions to longstanding mathematical problems. The claims unsettled parts of the research community. Some mathematicians called for limits on AI use, arguing that working through a problem can be as valuable as arriving at an answer.
The new study points toward a wider question: if models can make progress on theoretical problems in multiple fields, how reliably can researchers tell a genuine result from a plausible-looking mistake?
More than an answer
Thompson sees both benefits and problems in AI’s spread to other scientific fields. Theoretical discoveries can underpin practical advances, and wider use of AI could speed that transition. AI labs are also studying how models might conduct real experiments alongside theoretical analysis.
But the research does not remove the need for human judgment. Manso says people will still need to propose good ideas and connect facts. He also warns that models can fail on very difficult problems. OpenAI withdrew several solutions from its latest dataset after publication, underscoring why the full reasoning needs scrutiny: a model may label an answer correct even when its solution does not hold up.
I think the key distinction is between producing an answer and establishing that it is sound. The study tests the first under controlled conditions; the second still depends on researchers checking the work. As AI moves from mathematics into other sciences, that verification burden may become as important as the model’s ability to solve problems.
Daily AI news
Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.
Only what matters — every day
Follow on X