RAG is not magic: the retrieval bugs that make your bot confidently wrong

The bug report says "the LLM hallucinated." It usually lies.

A support bot tells a customer that refunds take 90 days. Your policy says 14. Everyone in the channel writes the same thing: the model made it up. Ship a stronger prompt, maybe swap to a bigger model, move on.

I have chased a lot of these, and most of the time the model did exactly what it was told. It got handed a chunk of text that said "90 days" and it summarized that chunk faithfully. The number was wrong because the retrieval was wrong. The model was the last honest step in a broken pipeline.

Here is the mental shift that saves hours. Before you touch the prompt, print what was actually retrieved for that query. The list of chunks, their scores, and the source document ID. Nine times out of ten the answer to "why did it say that" is sitting right there in plain text, and you never had to argue with a black box at all.

The failure modes are boring, which is why they hide

None of these are exotic. That is the point. They are dull, mechanical, and they wear the costume of a hallucination so you stop looking.

Chunking that splits the answer. You chunk by fixed token count. The sentence "Refunds are processed within 14 business days" lands at a chunk boundary, so "within 14 business days" ends up in chunk 41 and the subject "refunds" sits at the tail of chunk 40. Retrieval pulls chunk 41. The model sees a floating "14 business days" next to some other topic and stitches together something plausible and wrong. Fix: chunk on structure, not on raw token counts. Split on headings and paragraphs, keep a sentence whole, and let chunks overlap a little so a fact never gets guillotined.

Embedding mismatch. You embedded your documents with one model and you embed the incoming query with a different one, maybe because someone bumped a version in a config six weeks ago. Now the query vector and the document vectors live in slightly different spaces. Scores look fine. Results are subtly garbage. Fix: pin the embedding model, log its exact name and version next to the index, and refuse to query an index built by a different model.

Stale index. Someone edited the refund policy in the docs. Nobody re-embedded. The index still holds the old text from three months ago. The model quotes the past, confidently, with a citation, which makes it worse because now it looks trustworthy. Fix: tie reindexing to your content pipeline, stamp every chunk with the source's last-modified time, and treat a stale timestamp as a first-class alert.

Retrieving the wrong doc, then dutiful summary. The query is "cancel subscription." Top hit is the "cancel a scheduled report" doc because it is dense with the word cancel. The model gets clean, well-written, entirely irrelevant text and summarizes it beautifully. This is the sneakiest one, because the output reads great. It is fluent. It is just about the wrong thing. Fix later, but first: look at the retrieved doc ID and you will spot it instantly.

No "I found nothing" path. This is the quiet killer. If nothing relevant exists, most pipelines still return the top three chunks, because top-k always returns k things. There is no floor. A cosine score of 0.31 gets passed along the same as a 0.89, and the model, handed weak scraps, fills the gap with invention. That last part is the only real hallucination in the bunch, and you invited it by never letting retrieval say no.

Debug by looking, not by guessing

The whole method is one habit: inspect the retrieval before you blame the generation. Concretely, log four things for every answer.

1. The exact query string sent to the retriever (after any rewriting, because query rewriting is its own bug farm).
2. The top-k chunks with their similarity scores.
3. The source document ID and last-modified timestamp for each chunk.
4. The final prompt the model actually received.

Now the questions answer themselves. Was the right document even in the top-k? If no, it is a retrieval problem, full stop, and the model was never given a chance. Were the scores all low? Then your pipeline should have bailed and said "I do not have that," and the fix is a threshold, not a prompt. Did the right chunk get retrieved but arrive cut in half? Chunking. Did a whole unrelated doc win on keyword overlap? Ranking.

Two cheap upgrades pay for themselves fast. First, add a relevance floor. If the best score is below a threshold you tuned on real queries, return "I could not find this in the docs" instead of forcing an answer. A bot that admits a gap beats a bot that invents one, every single time. Second, put the retrieved sources in the response, even in a debug field. The day you can see what the model was reading is the day these bugs stop feeling like magic and start feeling like the plain data problems they always were.

The model is not your least reliable component. Your retrieval is, and it does not talk back, so it takes the blame quietly while the model takes the heat. Turn on the lights. Read what came back. Most "hallucinations" confess the moment you look at the evidence they were built from.

AGINE Academy is an independent product by AGINE AI (not affiliated with Anthropic). We teach building with Claude by doing the work, not watching lectures.