GPT-5 and Gemini-3-Pro fail to recall up to a third of stored facts

A group of researchers has pulled apart two failure modes that accuracy benchmarks have been quietly averaging together: a model that never learned a fact, and a model that learned it and cannot reach it. In a paper titled "Empty Shelves or Lost Keys? Retrieval Is the Bottleneck of Factual Knowledge in Model Parameters," they profile 13 language models against 2,150 Wikipedia facts. Gemini-3-Pro and GPT-5 encode 95-98% of those facts in their parameters. Asked to recall them directly, with reasoning switched off, both miss 26-34%. The shelves are stocked. The keys are lost.

Which of the two you are looking at decides where the next round of money goes. An encoding failure is answered by more parameters and more pretraining data, the strategy the industry has spent years and enormous capital executing. A retrieval failure is answered by post-training and inference-time methods, which are cheaper and far less mapped.

The method the authors call knowledge profiling moves the unit of analysis from the question to the fact. Rather than scoring whether a model answered a particular prompt, they assign each fact a state. There are five: encoding error, retrieval error, direct retrieval, retrieval with reasoning, and an answer produced without the fact being encoded at all. Encoding means the fact is represented in the parameters. Retrieval means reproducing a stored fact with no external hints. Recognition means picking the correct fact out of a set of offered options.

Three behavioural checks carry the whole scheme. Facts are pulled from Wikipedia, one of the main pretraining sources. The model is then asked to reproduce a fact inside its original context, which tests encoding. Finally it is asked about the fact in varied phrasings and in both directions, with and without reasoning, which measures how accessible the stored knowledge is.

The benchmark, WikiProfile, was assembled by a fully automated LLM-based data pipeline running Gemini-2.5-Pro with reasoning; the prompts driving it were hand-tuned on a small held-out sample. Facts were extracted from Wikipedia pages as statements about an ordered pair of entities, with the subject appearing first in the document and the object second. Each fact carries 10 tasks: two on encoding, four knowledge questions, and four multiple-choice variants for recognition. Forward and backward questions were built in three stages of generation, refinement and filtering, with each question required to be unambiguous, specific, minimal and to have exactly one answer. Every variant was additionally checked against a search engine, and cases with multiple answers or a need for clarification were removed. After automated filtering and a final manual pass, 2,150 facts survived.

The evaluation itself is large. Thirteen models, each tested with and without reasoning, eight responses per model-fact-task combination, graded automatically by LLM-based judges: roughly 4.5 million answers in total. The manual review covered the benchmark's questions. It did not cover the 4.5 million responses.

The headline result is that encoding at the frontier is close to done. For Gemini-2.5-Pro, Gemini-3-Pro, Gemini-3-Flash and GPT-5, encoding of facts is near saturation while retrieval still lags. Even with reasoning turned on, Gemini-3-Pro and GPT-5 fail to retrieve 11-12% of the facts they hold. Factual errors at the frontier are increasingly errors of access, not absence.

The Gemma 3 family shows what scaling actually buys. Larger models store noticeably more facts, and encoding errors drop accordingly. Retrieval errors stay substantial, and take up a growing share of whatever errors remain. Size improves the inventory faster than it improves the index.

The rare-facts result is the one that should change how people read existing literature. Prior work has tied a model's trouble with obscure facts to limited capacity. Profiling points elsewhere. For facts in the bottom 20% by popularity, the encoding rate sits close to that of facts in the top 20%. The gap in encoding is comparatively small; the gap in direct retrieval is much wider.

The researchers compared the two popularity tiers, bottom and top 20%, on the share of facts encoded and the share the model can retrieve directly. Δ marks the difference between the groups: small for encoding, considerably larger for retrieval. Many rare facts are probably sitting in the parameters already and are simply hard to summon with a prompt.

The same reframing applies to the reversal curse, where a model knows that A is B but cannot answer what B is. That has usually been read as an absence of bidirectional knowledge. In open generation without external material, backward questions are indeed consistently harder than forward ones. In multiple-choice tasks they are not worse, and are often easier. A model that recognises the right answer among distractors but cannot produce it unprompted in the reverse direction has not failed to store the relationship. It has failed to retrieve it when the direction of the question differs from the direction it met during training.

Reasoning recovers a meaningful part of the loss, and it recovers it exactly where direct retrieval is weakest. The largest gains show up on rare facts and backward questions, narrowing both the popularity gap and the direction gap.

In reasoning-optimised models, the reasoning mode recovered roughly 40-65% of facts that were encoded but not directly retrievable. For facts absent from the parameters, the figure was only 5-15%. That asymmetry is the cleanest evidence in the paper that reasoning tokens are, in large part, buying retrieval rather than inference. The chain of thought is not deriving a new answer through multi-step logic; it is walking around the building until it finds a door that opens. The authors note the obvious cost problem: reasoning burns additional compute, and there is no established method for deciding when a model should switch it on.

Two things about the setup deserve more attention than they will get. First, encoding here is operationalised as the ability to reproduce a fact when its pretraining context is supplied. That is a generous definition. A model completing a Wikipedia sentence with the surrounding Wikipedia paragraph in front of it has demonstrated something closer to contextual continuation than to durable storage, and every "the fact is in there" claim in the paper rests on it. Second, Gemini-2.5-Pro built the benchmark, and Gemini-2.5-Pro is one of the frontier models the benchmark then evaluates. A pipeline that generates, refines and filters questions is making choices about what counts as unambiguous, specific and minimal, and those choices are not neutral with respect to the model that made them. Nothing in the writeup addresses how much of the Gemini family's showing survives that circularity.

Those caveats do not dissolve the finding, and the finding is uncomfortable for the scaling argument. If the shelves are nearly full at the frontier, the remaining accuracy gains do not come from another order of magnitude of pretraining. They come from cheaper, less glamorous work on access: prompting, post-training, inference-time policy. The industry has an enormous amount of capital and organisational muscle pointed at the first kind of problem and comparatively little pointed at the second. That mismatch is where the next few points of factual accuracy are going to be won or lost.