There is a particular kind of error a language model makes that feels different from every other kind of software failure. A database returns the wrong row and you can trace the query. A null pointer crashes and the stack tells you where. But when a model confidently cites a paper that does not exist, the failure has no fingerprint. The output is well-formed. The grammar is correct. The tone is authoritative. Nothing about the artifact itself betrays that it is fiction.
This essay is about that gap — between the appearance of an answer and the existence of one — and why I think we have been framing it wrong. Hallucination is not a defect that a sufficiently large model or a sufficiently clever fine-tune will eventually remove. It is a structural consequence of what these systems are. The sooner we design around that, the sooner we stop being surprised by it.
The mirror, not the lamp
The metaphor I keep returning to is a mirror. A lamp produces light from a source; a mirror reflects what is in front of it. A language model is a mirror polished against a corpus of text. When you stand in front of it with a prompt, it returns the shape of what such a prompt has historically been answered with — the statistical silhouette of an answer, rendered in fluent prose.
Most of the time, the silhouette and the truth coincide closely enough that the distinction does not matter. The mirror reflects the world because the world is what trained it. But at the edges of the training distribution — obscure facts, recent events, specific citations, narrow technical claims — the mirror does not stop reflecting. It continues to produce the silhouette of an answer, drawn from adjacent shapes in its training. The output looks like a reflection. It is, in fact, an extrapolation.
Anthropic's model card and OpenAI's system cards both treat hallucination as a measurable property of the system rather than an aberration to be eliminated. That framing is the honest one. Both labs publish hallucination rates the way a manufacturer publishes tolerances. The mirror has a flatness rating. It is never going to be perfectly flat.
Why fluency makes it worse
The taxonomy researchers use — intrinsic hallucinations (contradicting the source you gave the model), extrinsic hallucinations (adding facts that were not in the source), open-domain hallucinations (fabricating from training memory) — is useful for evaluation, but it obscures what makes the phenomenon dangerous in practice. The danger is not the error rate. It is the decoupling of fluency from accuracy.
In most communication, fluent and confident delivery is correlated with knowing what you are talking about. A human who speaks crisply about a topic has usually earned that crispness through some form of contact with the subject. We read confidence as a costly signal. Language models break this signal entirely. The fluency of the output carries no information about the factual grounding of the content. A fabricated citation reads exactly like a real one because the model is good at producing citations, not at having read papers.
Researchers at Stanford HAI and MIT CSAIL have written accessibly about why this decoupling is structural rather than incidental. The same training objective that makes the model fluent — minimizing next-token loss across a vast corpus — is indifferent to whether the tokens it predicts describe something true. Fluency is what the loss function rewards. Truth is a property of the world the model cannot directly check.
This is what makes the mirror metaphor sharpen rather than soften the problem. A cracked mirror still reflects. The crack does not announce itself in the reflection — it distorts it. And if you have only ever seen the reflection, you have no independent access to the room behind you.
The mitigation that admits the diagnosis
If you watch what serious practitioners actually build, you can read the diagnosis through the response. Retrieval-augmented generation — the dominant production pattern, documented exhaustively in the LangChain and LlamaIndex guides — works by refusing to trust the mirror's memory. Instead of asking the model what it knows, you retrieve the relevant documents, hand them to the model, and ask it to answer from those documents alone. The model becomes a reading-comprehension engine rather than an oracle.
RAG is interesting not because it eliminates hallucination — it does not, and intrinsic hallucinations against the retrieved context remain a real failure mode — but because of what its adoption admits. The entire pattern is a concession that the model's internal knowledge cannot be trusted as a source of truth. The fix is not to make the model know more. The fix is to externalize the knowledge entirely and constrain the model's role to a narrower one: stitching retrieved material into a coherent answer.
Chain-of-thought prompting and RLHF compress hallucination rates further, but they operate on the same mirror. They change which silhouettes the mirror prefers to return. They do not give it access to the room. This is why benchmark improvements on factuality have been steady and real, and yet the practitioner consensus has converged on retrieval as the load-bearing technique. The benchmarks reward better mirrors. Production rewards admitting the mirror has limits.
The epistemic problem is the harder one
Here is the part I find genuinely uncomfortable. The technical mitigations are improving. Hallucination rates on standard benchmarks have dropped meaningfully across model generations. And yet the practical risk in deployment has not dropped at the same rate, because the errors that survive the mitigations are increasingly the ones that look most plausible.
A model that hallucinates obviously is annoying but tractable. You learn to spot the tells — the suspiciously round numbers, the citation in a format that does not match the journal, the confident claim about an event the model could not have seen. A model that hallucinates with the same prose rhythm as its accurate output is a different problem entirely. The signal-to-noise ratio improves on the surface while the reader's ability to discriminate degrades.
The high-stakes domains feel this most sharply. In medicine, law, and finance, the cost of a confident falsehood is asymmetric — a hallucinated dosage, a fabricated precedent, a misstated cap-table fact does not get corrected by the next paragraph being fluent. The cost is paid by the person who acted on the output. And the verification burden does not scale gracefully: if every model output requires independent verification, the model has not saved time; it has shifted the labor from drafting to fact-checking, and fact-checking is the harder of the two tasks.
Designing around the mirror
If hallucination is structural, the engineering question stops being "how do we eliminate it" and becomes "what does a system that assumes it look like." A few moves follow from taking this seriously.
Ground at retrieval, not at generation. Treat the model's parametric memory as a writing assistant, not as a knowledge base. If the answer requires a fact, the fact must enter the prompt from a verifiable source, with the citation preserved through to the output. The model is allowed to phrase. It is not allowed to know.
Make provenance a first-class output. Every claim a model emits in production should be traceable to the document, row, or tool call that produced it. If it cannot be traced, it should not be displayed as a claim — it should be displayed as a draft, with the user explicitly informed that the model is operating without grounding.
Resist the temptation to smooth confidence. The current generation of products has learned that hedged language tests poorly with users, and the result is interfaces that present model output with the same visual weight as verified information. This is the cracked mirror problem rendered as UX. The fluency of the medium is collapsing the distinction the system itself cannot make.
Reserve the high-stakes call for the human. Not as a moral position but as an engineering one: a system whose error mode is fluent fabrication is not a system you put in the final-decision seat in domains where falsehood is expensive. You put it in the drafting seat, the summarizing seat, the first-pass seat — places where a human reads behind it.
The wider frame is this. We spent the first wave of LLM deployment hoping the next model would fix the hallucination problem. The next model will not, and the model after that will not either, because the property we are asking them to lose is the property that makes them work. The mirror reflects whether the room is there or not. The discipline is to stop confusing the reflection for the room, and to build the room ourselves — out of retrieved documents, preserved citations, and the older, slower habits of verification that the fluency of these systems has tempted us to set aside.
Top comments (0)