Why language models may guess instead of admitting uncertainty

The cost of always having an answer

A hallucination is a plausible-sounding answer that turns out to be false. It can be hard to spot because it may sit inside an otherwise accurate response. A model could explain Einstein’s theory of relativity correctly, then add that unicorns spoke with Einstein and helped him develop it. The surrounding facts can make the invention easier to believe.

There is no reliable way to predict exactly when a hallucination will appear, and it may have nothing to do with a particular prompt. That makes checking answers essential, but tiring: a user who trusts a model because it is usually right may miss the one false claim.

Researchers have tried information retrieval, tools, self-checks based on consistency between answers, and reinforcement learning from human feedback. Yet hallucinations remain in the best language models available today. Asking a model not to hallucinate is not a dependable solution either. The request may modestly reduce errors, but mentioning hallucinations—even to forbid them—can also draw the model toward them.

What the paper argues

In “Evaluating Large Language Models For Accuracy Incentivizes Hallucinations,” Adam Tauman Kalai, Ofir Nachum, Santosh Vempala, and Edwin Zhang propose a simple explanation: under ordinary evaluation, guessing can be more rewarding than admitting uncertainty. Next-word prediction and accuracy scoring may unintentionally favor unsupported answers.

That shifts the problem from model capability alone to incentives. If a model is rewarded for producing an answer, rather than for knowing when it cannot establish one, then a confident guess is not just a failure of knowledge. It is also a predictable response to the system around it.

The researchers identify several possible points of intervention:

Each comes with trade-offs. A design change could solve one problem while introducing another. Training changes may preserve the existing architecture and cost less than rebuilding it, but could also have side effects. Combining both approaches risks problems in both places.

Giving the model room to abstain

Users can try a lighter-weight intervention: tell the model to say when it does not have enough information. But an instruction that discourages guesses can also make a model refuse questions it could answer. Asking for a confidence estimate is another option. It gives users a signal without prohibiting an answer, though repeated confidence labels may eventually become background noise. The source author prefers using them for unusual topics or rare cases, where hallucinations are more likely.

My read is that this is a more useful framing than treating “don’t hallucinate” as a magic prompt. The paper points toward a mismatch between what users need—reliable answers, including a clear admission of uncertainty—and what ordinary evaluation can reward. But the announcement, as described here, does not establish how much any proposed intervention reduces hallucinations, or how often it would make models unnecessarily cautious. That is the practical question developers will have to answer.

There is no known way to eliminate hallucinations entirely. For now, users can compare answers with other sources or ask multiple models the same question; the source argues that the chance of two different AIs producing the same hallucination is extremely low. Verification remains work, not a feature a prompt can guarantee.

If models are trained to treat abstaining as an acceptable outcome, they may become more dependable. The tension is that a system cautious enough not to invent an answer may also decline to give one when it has useful information.