Ask a language model for the DOI of a paper it half-remembers and it will give you one. The format will be right. The journal will be plausible. The prefix will look like a real registrant. The DOI will not exist.
This gets called hallucination, which is a poor name — it suggests a malfunction, a glitch that better engineering will remove. It is closer to the opposite. The model did exactly what it was built to do. It produced the text most likely to follow “the DOI is”, and that text is a well-formed DOI. Nothing in the training objective ever asked whether the string points at anything real.
Getting this right matters, because the wrong mental model leads to wrong fixes. If hallucination is a bug, one looks for a patch. If it is a consequence of the objective, one looks for architecture around the model instead.
Fluency is the target; truth is a side effect
Training rewards one thing: assigning high probability to the text that actually came next in the corpus. That objective has no term for accuracy. There is no gradient pushing “correct” up and “incorrect” down. There is only one pushing “likely continuation” up.
Truth sneaks in through the back door. The training text is mostly written by people trying to be right, so true statements are usually the likely ones. The capital of France really is Paris, in the data and in the world, so the model learns it. But the correlation between “likely” and “true” is strong for well-covered facts and weak for everything else. The model has no way to feel the difference. Both cases are one probability distribution over tokens.
This is why the failure is specifically confident wrongness. There is no “unsure” token to emit. Uncertainty would have to show up as a flatter probability distribution — and even then, the sampler picks something, and whatever it picks gets written in the same steady voice as everything else. A model that hedged in proportion to its internal uncertainty would be far more useful, but it was never trained to do that, and the hedging language it does produce is a learned style, not a readout of confidence.
There is no lookup table inside
The strongest intuition to install: the model does not store facts, it
stores patterns. There is no row anywhere saying “Marie Curie, born 1867”.
There are weights that make 1867 the likely continuation of “Marie Curie was
born in”. Those are different things, and the difference shows up exactly at
the edges.
Well-covered facts appear thousands of times in training, so the pattern is sharp and the answer is right. Obscure facts appear once or twice, so the pattern is fuzzy and the model falls back on the shape of the answer: a plausible year, a plausible surname, a plausible page number. The output looks identical either way.
The same mechanism explains the details of the failure that people find uncanny:
- Invented citations are formatted perfectly. Format is a strong, well-learned pattern; the specific paper is not.
- Made-up API methods look like real ones.
df.drop_duplicates_by()fits everything the model learned about how pandas names things. - Wrong answers get more detailed under pressure. Asking “are you sure?” conditions on text where someone is defending a claim, and defences come with supporting detail. The detail is generated the same way the original claim was.
- The model contradicts itself across a long answer. Nothing holds a consistent world state; each token is predicted from the text so far.
What actually reduces it, and what does not
Start with what does not work, because it is what people try first.
Asking for accuracy does not create accuracy. “Be factual”, “only state things you are certain of”, “do not hallucinate” — these change the style of the output, not its grounding. They condition the model toward text that sounds careful. A careful-sounding wrong answer is worse than a casual one, because it defeats the reader’s own alarm.
Asking the model to rate its own confidence is weak evidence. The number is generated the same way everything else is: as a plausible continuation. Some correlation with correctness exists, but it is far too loose to gate a decision.
A bigger model reduces the rate without changing the kind. More parameters and more data mean sharper patterns for more facts. The tail is still a tail.
What works shares one property: it puts something outside the model in the loop.
| Approach | Why it helps |
|---|---|
| Retrieval (RAG) | The facts arrive in the prompt, so the model is summarising given text rather than recalling. Sharply narrows the space it can invent in. |
| Constrained output | Restrict answers to an enum, a schema, or rows from a table. A model that can only pick from real IDs cannot invent one. |
| Verification pass | Check claims against a source with code — does the DOI resolve, does the function exist, does the SQL run. Cheap, and catches the confident cases. |
| ”Not in the context” as a valid answer | Give the model a legal way out and show examples of taking it. Without one, the likeliest continuation is always an answer. |
Retrieval is the workhorse, and it moves the problem rather than removing it. The model will still misread a retrieved passage, or blend two documents, or answer from memory when retrieval returns nothing useful. Retrieval quality becomes the thing to measure — the same discipline as any other component, with a labelled set of queries and a recall number, and the same warning that a single headline metric hides the failures that matter.
IBM Technology gives a short version of the same mechanism in Why Large Language Models Hallucinate.
For teams building on these models, the design rule falls out of the mechanism. Every factual claim the system emits needs something outside the model to check it against: a DOI that resolves, a function that exists in the imported version, a query that runs, or a person who knows the subject. Where no such check exists for a class of claim, that class does not leave the building. Working out which check applies to which claim is design work, and it has to happen while the feature is being specified.