A user reports a bad answer. Somebody on the team says the model hallucinated. Somebody else suggests trying a newer model. Two weeks and a migration later, the same question produces a different wrong answer.
This cycle repeats in a lot of organizations, and it's built on a misdiagnosis. In a retrieval-based system, the overwhelming majority of bad answers are not generation failures. The model faithfully summarized whatever it was handed. The problem is what it was handed.
Learning to tell these apart is the single most useful diagnostic skill for anyone maintaining one of these systems, and it doesn't require touching the model at all.
Four places an answer can go wrong
Before triage, it helps to name the layers. A bad answer originates in exactly one of these, and they need completely different fixes.
The source layer. The correct information doesn't exist in your indexed content, or it exists alongside three contradictory versions.
The indexing layer. The information exists but was mangled on the way in. A table got flattened into meaningless text. A document was split so that a rule and its exception ended up in separate pieces.
The retrieval layer. The right content exists and is well-formed, but the search didn't surface it for this particular question.
The generation layer. The right content was retrieved and provided, and the model still produced something wrong.
Only the last one is a hallucination in any meaningful sense, and in practice it's the rarest. Modern models are quite good at summarizing what's in front of them. When they don't, it's usually because what was in front of them was contradictory, incomplete, or buried in noise.
A triage sequence that takes ten minutes
When a bad answer comes in, work backward through the layers rather than starting with theories.
Step one: does the correct answer exist in your content at all? Search your source repository directly, the old-fashioned way. If nothing authoritative exists, you have a documentation problem wearing an AI costume. Stop here.
Surprisingly often, this is where it ends. The system was asked something the organization never wrote down.
Step two: is there exactly one version of the truth? If your search turns up the current policy and two superseded ones, the retrieval layer had no way to know which mattered. It probably returned a mix, and the model did its best to reconcile documents that disagree. That output looks like an invention and is actually faithful synthesis of contradictory input.
Step three: was the right passage retrieved? Look at what the system actually pulled for that query. Most platforms will show you. If the relevant passage isn't in the retrieved set, the model never had a chance and the model is not your problem.
Step four: was it retrieved but drowned? Sometimes the right passage is in position eight out of ten, surrounded by nine loosely related ones. The signal exists, the ratio is bad. This is a different failure than not retrieving it, and it has a different fix.
Only then consider generation. If the correct passage was retrieved, clearly stated, and prominently placed, and the answer still contradicts it, you have a genuine generation issue. Now the prompt design or the model is worth examining.
Patterns that account for most failures
Having run this triage enough times, the same handful of causes dominate.
Vocabulary mismatch. Users ask using internal shorthand. Documents use formal names. Nothing matches. This shows up as the system finding "nothing relevant" for questions that clearly have answers, and it's especially common with acronyms, product codenames, and terms that mean something different outside your industry.
Tables destroyed during ingestion. Pricing grids, comparison matrices, and specification tables carry meaning in their structure. Flatten them into a text stream and the relationship between a row label and its value disappears. The retrieved chunk technically contains the number and no longer indicates what the number refers to.
Rules separated from their exceptions. Documents get split into pieces. When a policy statement lands in one chunk and its "except when" clause lands in the next, retrieval can surface the rule without the qualifier. The answer is confidently and dangerously incomplete.
No sense of time. Two documents describe a process. One is from 2022, one from this year, and nothing in the content signals which supersedes which. Retrieval has no basis for preferring the recent one unless recency was made part of the system deliberately.
Questions that span documents. "How does our refund policy apply to enterprise contracts signed before the pricing change?" needs three sources combined. Single-pass retrieval tends to return material on one of the three and produce an answer that addresses a third of the question.
Over-retrieval. Someone increases the number of results returned, reasoning that more context is safer. More context frequently means more noise, and the relevant passage becomes harder for the model to weigh properly. Precision beats volume.
Why "use a better model" keeps failing
It's an appealing move because it's a single decision and someone else does the work.
But if the correct passage never reached the model, a smarter model produces a more articulate wrong answer. If the retrieved sources contradict each other, a smarter model synthesizes the contradiction more smoothly. If your documentation is genuinely wrong, a smarter model reproduces the error with better prose.
Model upgrades help with reasoning quality, instruction adherence, and handling long context. They don't help with anything upstream of what the model sees, which is where these problems live. Teams doing serious RAG development spend the bulk of their debugging effort in the retrieval and ingestion layers for exactly this reason.
What to instrument so triage isn't guesswork
You can't run the sequence above if the system doesn't expose its own behavior. A few things are worth building in from the start.
Log what was retrieved for every query, with scores, and keep it long enough to investigate complaints that arrive a week late. Without this, every diagnosis is speculation.
Track how often the system returns nothing relevant. A rising rate usually means a vocabulary or coverage gap rather than a model issue.
Maintain a fixed set of questions with known correct answers and run it regularly. Retrieval quality drifts as documents accumulate, and drift is silent. A hundred questions run weekly will catch a regression that user complaints would surface a month later.
Measure appropriate refusals as a success rather than a failure. If you only reward answers, you'll train the system and yourselves toward confident guessing.
Give users a one-click way to flag a bad answer that captures the query, the retrieved sources, and the response together. A complaint without that context costs an hour to reconstruct.
The reframe
When an answer comes back wrong, the productive question is not "why did the model make that up." It's "what did the model see, and would a careful human reading only that have said something different?"
Usually the answer is no. The model saw a mess and reported it accurately. Fixing the mess is unglamorous work involving documents, chunking strategy, and terminology mapping rather than anything that sounds like AI engineering.
That's also why it works.
Top comments (0)