44% of findings about language models are already outdated

Stanford researchers Owen Quinn and James Zou tested 50 frequently cited findings about large language models against 20 models and found that 44% of the findings were already outdated. Among the 22 findings whose lifespan could be estimated, it averaged just 15.7 months. The models were released by eight developers between July 2024 and September 2026. Another 34% of the findings had weakened substantially, 16% held up, and 6% had grown stronger. After one year, 16% of the findings were outdated; after two years, 39%.

Study: 44% of findings about language models are already outdated

Which findings lose ground fastest

Claims about what models cannot do become outdated fastest. Of nine findings about reasoning, five no longer held up, and none remained valid or grew stronger. Examples include difficulty counting the letters in “strawberry,” doing arithmetic in unfamiliar numeral systems, and retaining information in the middle of a long context. The same group includes models’ tendency to abandon a correct answer when the user pushes back. The researchers suggest these errors may be easier to fix because they have an unambiguous correct answer that developers can target in training.

Observations about models’ preferences and habits last longer. Of the 11 findings that held up or grew stronger, three concern language models acting as evaluators: they prefer their own answers, rate accepted and rejected scientific papers in similar ways, and are swayed by language in medical abstracts.

Three more findings relate to sycophancy and agreement with users. This group also includes findings about models’ tendency toward left-leaning political views, their nearly identical answers to open-ended questions, and uneven distributions when generating random numbers.

Fabricated citations: from 55% to zero — but not for every model

Quinn and Zou separately tested how models fabricate scientific citations. In a 2023 study by Walters and Wilder, GPT-3.5 fabricated 55% of citations. The Stanford researchers gave the same prompts to current models: GPT-6.1 Sol fabricated no citations, while Claude Sonnet 5.5 had a rate of 1%. But Qwen3.7-Flash, also released in 2026, fabricated 69% of citations in the same test.

The Stanford study found that 44 percent of reviewed LLM findings have expired, with another 34 percent weakened. All nine reasoning findings have faded, while sycophancy, deception and strategic behavior has the most findings that persist or have strengthened. | Image: Queen and Zou

Source: the-decoder.com

The researchers caution against concluding that the hallucination problem has been solved. In questions requiring two stages of reasoning, where two known facts must be combined, Claude Sonnet 5.5 gets about 10% wrong, even when it answers each question correctly on its own. The original study reported 43% for ChatGPT. The “reversal curse” — contradictory answers to related questions about time — has also weakened, but both effects can still be observed.

Results on social biases are mixed. The tendency toward left-leaning political views persists, while gender stereotypes and biases detected by implicit association tests have weakened. For bias against speakers of African American English, the trend has reversed.

Differences between models remain large

Even models released around the same time produce different results. Among models released between July and September 2026, the number of outdated findings ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Release date, developer, and model size do not reliably predict which findings will hold up.

Quinn and Zou propose describing empirical findings about language models as observations of a specific model at a specific point in time, rather than as general truths. In their view, publications should specify the model, its version, and the date of the study, and make prompts and analysis code available so results can be checked against future models.

The researchers also cite a post by Andrej Karpathy about how the gap between people’s expectations and the actual behavior of language models affects trust in AI and how often people use it. Karpathy, an OpenAI co-founder who later joined Anthropic, recently noted that the models currently available offer only a preliminary glimpse of future capabilities.

Dataist’s take: The findings show how quickly generalizations about language models can become outdated: even research-based conclusions need to be tied to a model version and a test date. At the same time, differences between models released in the same period make it hard to treat improvements as universal. It is becoming increasingly risky to draw conclusions about the capabilities of all models from a single test.