A $535 pipeline labels LLM hallucinations in 14 languages

Even the strongest LLMs will sometimes state, with full confidence, facts that appear nowhere in their sources. In question answering, one wrong word is enough to break the meaning. Most checks available today return a single verdict for the whole answer, and almost all of them work only in English. The authors of PsiloQA set out to do the opposite: cover 14 languages, and learn to find not just a "wrong answer" but the exact spans of text where the model went wrong.

How PsiloQA is built

The team built a fully automated pipeline. First, GPT‑4o takes Wikipedia passages and writes questions with short reference answers at varying levels of difficulty. Then dozens of different LLMs answer those same questions without the context — which makes it far easier to provoke confident but unsupported claims. GPT‑4o then compares each model's answer against the reference and the source passage, and marks the exact stretches of the answer where they disagree. Filtering comes last: malformed questions, refusals to answer and technical annotation glitches are dropped. What remains is a large, clean corpus of real errors rather than artificially injected ones.

What is in the dataset

PsiloQA covers 14 languages, with 63,800 training examples and almost 3,000 in the test set. English does not dominate: there is Hindi, Finnish, Catalan, Chinese, Swedish, Czech, Farsi, Spanish, Basque, French, Italian, Arabic and German. Short hallucinations, a few words long, are common — exactly the granularity that matters in production. Geography and sports lead by topic, but a long tail covers many other areas.

Two illustrative errors: the model confidently says Lillian Richter was born in 1932, where the text says 1915; "Blackpool Stadium" instead of the correct Bloomfield Road; "Amanita rubescens-Team" instead of Russula font-queri. Cases like these are the argument for span-level annotation: one token goes wrong and the fact is already false.

How the methods were compared

The authors tested three classes of approach:

• Token-level uncertainty estimation. Simple, fast methods that try to pick up on the model's own uncertainty.
• Encoders trained to tag incorrect spans from a "context — question — answer" input. The paper trains ModernBERT and the multilingual mmBERT.
• Approaches using external knowledge and LLM verifiers. Extracting the claims and checking them with GPT‑4o, for instance.

Scoring used two metrics: area under the precision–recall curve (AP), and character-level Intersection over Union (IoU) of the spans. The first shows how well a method separates correct fragments from incorrect ones; the second, how precisely it lands on the boundaries of the error.

How the methods actually did

• The fast uncertainty measures come out middling: they catch part of the problem but hold the exact boundaries poorly.
• The most accurate were the encoders trained specifically on PsiloQA. The multilingual mmBERT was especially strong: in 12 of the 14 languages it is the best on both metrics. That supports the case that multilingual pretraining helps a model see errors even where data is fairly scarce.
• LLM fact-checking with external knowledge sometimes reaches a high AP but suffers on IoU — drawing a clean outline around the error is hard.

Multilingual training also turned out to beat training a separate model per language: a single mmBERT transfers more reliably both to closely related languages and to ones written in a different script. More telling still is the transfer beyond PsiloQA itself. A model trained only on the English portion of PsiloQA comfortably outperformed its counterparts on English benchmarks such as Mu‑SHROOM and HalluEntity. And PsiloQA's automatic annotation cost roughly $535, against about $3,000 for a comparable volume of manual English annotation in RAGTruth.

Why this matters

PsiloQA shows that annotation of real LLM errors, down to individual fragments and across many languages at once, can be produced at scale and cheaply. That lowers the barrier for researchers and teams who need a fine-grained detector of factual misses that carries over between languages and tasks. Yes, there is a single annotator and it is automatic, and the focus is Wikipedia QA and nothing else. But the quality of the results and the solid adaptation to different domains suggest the approach is practical as it stands.

💾 Code