Top LLMs score 30 out of 100 on a benchmark of the full research cycle

Today's LLMs can do a lot: explain hard topics, write code, hold a long thread of reasoning. But science is not just knowing the answers. It is a research cycle: work through the literature, come up with a hypothesis, test it with an experiment, then interpret the results honestly and adjust the plan. And this is where the field has long lacked a shared language and a shared yardstick: what exactly counts as an AI's scientific intelligence, and how do you test it on a workflow that looks like a researcher's actual job rather than on isolated problems.

Researchers at Shanghai AI University have released a study, Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows, that offers an answer: they introduce a practical definition of Scientific General Intelligence (SGI) and build a benchmark that tests not erudition but a model's ability to move through the stages of scientific research.

The scientific cycle as a measurable frame

Instead of abstract talk about being almost at AGI, the authors build on the Practical Inquiry Model — a model of practical inquiry with four connected stages. In human science they repeat over and over: first we gather and critically assess knowledge, then we form an idea, then we run an experiment, and afterwards we try to work out what actually happened.

The researchers turn this framework into four task types, made as close as possible to what a working scientist does:

1. deep scientific research over sources, where what counts is not general statements but precise, often numerical conclusions;
2. research idea generation, but with a requirement that the implementation be thought through;
3. experiments — from completing scientific code to planning a lab protocol;
4. reasoning over data and images, including comparing conditions and drawing causal conclusions.

How SGI-Bench was built, and why that matters

The benchmark itself, SGI-Bench, is more than a thousand interdisciplinary examples. The topics are drawn from Science’s 125 Big Questions, and the data and questions go through multi-stage vetting. Researchers at master's and PhD level took part in assembling them, after which the questions were further cleaned up by rules, checked by models and reviewed by domain experts.

One move worth singling out is the difficulty filtering: the authors ran the tasks past several strong models and threw out whatever turned out to be too easy. That noticeably changes the character of the benchmark — less a memory exam, more a stress test of robustness.

How the models are scored: asking who is right is not enough

The problem with scientific tasks is that an LLM judge tends to produce noisy scores: different domains need different metrics, and sometimes you have to actually execute the code or check the format of a protocol. So the authors build an agentic evaluation: not a single text-generating judge, but a system that can call tools such as a Python interpreter, a PDF parser and specialized metrics.

The result: strong fragments instead of coherent scientific behavior

The picture that comes out is fairly sober. Even the best models score around 30 out of 100 on the composite SGI-Score. That is an important signal: today's LLMs can look confident on individual skills, but gluing those skills into a dependable scientific workflow does not work yet.

In deep research, the step-by-step reasoning sometimes looks sound while the final exact answer is often wrong — exact match stays in the range of 10–20%. A model can search for information correctly and still fall down on numbers, units, integrating sources, or on the last step.

Idea generation looks good: the models really do propose unusual combinations of approaches. But the ideas have a chronic weak spot — feasibility. What is usually missing are the down-to-earth details: where the data comes from, how the pipeline is put together, in what order to run the steps, what resources are needed, how to test the hypothesis so that the experiment can be repeated.

"Dry" experiments expose the familiar problem: the code gets written, but the science does not add up. Executability can be high while scientific correctness is low. The models go wrong most often in numerical methods and simulations — where what matters is not syntax but the choice of a stable algorithm.

"Wet" experiments are more interesting still: protocols demand an order of operations, timings, branches, tracking of samples. Models frequently collapse a complex process into a linear list of steps and lose the logic of the experiment.

Finally, in multimodal experimental reasoning the models do better at recognizing signals and causal cues, but they consistently sag on comparative analysis: when several conditions have to be lined up carefully without losing track of what exactly differs.

An attempt to boost scientific creativity at inference time

One separately interesting piece is Test-Time Reinforcement Learning. The authors try to improve idea generation without ground-truth answers: the model is rewarded for novelty relative to publications found through retrieval, and gradually learns to propose less trivial hypotheses. This does not solve feasibility by itself, but it points in a direction: scientific skills can be trained dynamically in cases where labeling is impossible.

What this changes in AI for science

The main value of the work is in the honest, grounded way it poses the question. SGI here is not a mystical property or a piece of philosophy but a testable ability to go through the scientific cycle: from the literature to the interpretation of results. And the results say it clearly enough: today's LLMs are strong in individual components, but not yet reliable where science demands precision, procedural discipline and stable numerical methods.

For developers this means something simple: the next progress will probably come not only from model size, but from the combination of tool use, checks, self-correction, domain simulators and training.

💾 Code