FRI finds AI benchmarks outran expert forecasts by years

The misses are years, not months

FRI has collected AI forecasts since mid-2022, before ChatGPT launched. Its first LEAP panel included 339 participants:

The computer science group included 30 professors from 20 leading institutions and 10 authors among the 200 most-cited AI researchers. The panels also included superforecasters—generalist forecasters with demonstrated accuracy.

The clearest miss concerns mathematics. AI reached the level of a gold medal at the International Mathematical Olympiad in July 2025, five years earlier than the experts’ median forecast and 10 years earlier than the superforecasters’ median forecast. Those estimates were made in 2022, before ChatGPT, but FRI says the same pattern continued in later forecasts.

Based on their forecasts, experts assigned an average probability of 24.6 percent to the benchmark results that actually happened, while superforecasters assigned just 9.7 percent. For gold-medal performance at the Math Olympiad, the figures dropped to 8.6 and 2.3 percent. | Image: Forecasting Research Institute

Source: the-decoder.com

AI may also have solved one of the Millennium Prize Problems, although it remains unclear whether the solution meets the evaluation criteria. In an August–September 2025 survey, experts gave the median probability of such a solution by the end of 2027 just 10%. Superforecasters put it at 5.4%.

A separate study of AI capabilities in virology produced another large gap. Experts expected models to match the leading team of virologists on a benchmark for finding and eliminating errors by 2030. Superforecasters chose 2034. FRI says that milestone was probably reached in April 2025. A cybersecurity benchmark showed a similar pattern of underestimated progress.

At the median, experts expected AI to match a top team on the Virology Capabilities Test by 2030, and superforecasters by 2034. FRI says it likely happened in April 2025. Respondents tied this milestone to higher expected biorisk, not to any documented rise in actual harm. | Image: Forecasting Research Institute

Source: the-decoder.com

The financial forecasts were cautious too. Participants estimated that the maximum annual recurring revenue of any AI company would reach a median of $20 billion by the end of 2026. Economists predicted $16 billion, while superforecasters predicted $25 billion. FRI puts Anthropic at approximately $100 billion in September 2026 and says that figure has probably already been reached.

Annualized revenue at Anthropic and OpenAI (left) far exceeded the median forecasts of every surveyed group (right). Even the highest group estimate of $25 billion fell well short of the $65 billion reported in July 2026. | Image: Forecasting Research Institute

Source: the-decoder.com

Where the optimism breaks down

The report does not show that every AI forecast was too conservative.

Biosecurity specialists estimated that 22.5% of participants using a language model would be able to complete tasks in a biology lab. Virologists expected 40%, and superforecasters 16.2%. In a controlled trial with a language model and internet access, only 5.2% of participants succeeded. The internet-only group reached 6.6%. The language model produced no measurable advantage, although the trial was small.

Forecasts for autonomous cars may have gone in the opposite direction. Experts put the median share of autonomous rides in US ride-hailing services at 7.3% in 2027. A forecast based on a large language model was lower, at 2.5%.

FRI says forecasts about economic growth, employment and large-scale harm from AI cannot yet be checked reliably. That is an important boundary around the report: benchmark progress is easier to date than broad social outcomes.

At the same time, participants are revising their expectations upward. Among people who completed both surveys, the average probability that AI will become the “technology of the century” rose over nine months from 31% to 36% among experts, and from 28% to 35% among superforecasters.

Experts and superforecasters now expect bigger societal effects from AI than they did nine months earlier. Both groups assign their highest average probability to "technology of the century," on par with electricity. | Image: Forecasting Research Institute

Source: the-decoder.com

The forecasting problem is partly about the clock

My read is that the report is more useful as a study of forecast timing than as a simple argument that AI is advancing faster than expected. A benchmark can be overtaken in a single evaluation cycle; real-world effects often require deployment, reliability and sustained use. The report contains both outcomes: rapid technical gains in mathematics and virology, alongside no measurable benefit in a small biology trial.

The more interesting question is how much of the apparent forecasting failure comes from the forecasts themselves and how much comes from the way they are measured. FRI acknowledges that its interim report is structurally more likely to expose underestimates than overestimates. An underestimate becomes visible as soon as reality passes the forecast. An overestimate can remain unresolved until its deadline arrives.

That asymmetry matters when the report is read as a scorecard. It does not make the early milestones unimportant, but it does mean the current balance of evidence is tilted toward finding excessive caution.

FRI plans to make that process more continuous. It will identify a subgroup of respondents expecting very rapid AI progress through 2040 and publish continuously updated forecasts from large language models alongside human estimates. According to ForecastBench, some models already match superforecasters on certain types of questions. FRI also plans to identify the most accurate LEAP participants and publish their forecasts once enough data has accumulated.

FRI also notes that some of its own estimates rely on large language model forecasts using information unavailable to the authors of the original predictions. That creates a further tension: the systems being evaluated may now help set the forecast against which their progress is judged. Forecasting AI is becoming a moving target just as AI itself keeps moving the target.