Why Does RAG Hallucinate When the Right Documents Exist?

You check the logs. The right document was retrieved. It's sitting in the prompt, rank 2 of 5. And the model still gives a confidently wrong answer.

This is the most frustrating RAG bug, because every dashboard says "retrieval worked." The usual advice (improve your embeddings, tune your chunking) doesn't help, because retrieval isn't what failed.

Here's the uncomfortable truth: retrieval puts evidence in the context. It doesn't force the model to use it faithfully. Between "document retrieved" and "answer correct" there are at least five places things go wrong.

1. "Retrieved" is not "retrieved cleanly"

The right document is in the top-k, but so is a lot of noise: near-duplicates, older versions, and passages that are on topic but don't answer the question.

Models are bad at ignoring distractors. If the correct chunk says the refund window is 30 days and a stale chunk says 14, the model may blend them, or pick the one that sounds more natural.

Check: For failing queries, print the full prompt. Count how many chunks actually help answer the question. If it's 1 out of 8, you have a precision problem, not a hallucination problem.

Fix: Retrieve wider, then rerank and cut down to fewer chunks. Fewer, better chunks beat more, noisier ones.

2. Position matters (lost in the middle)

LLMs don't attend to all positions in a long prompt equally. Evidence buried in the middle of a stuffed context is easier to miss than evidence at the start or end.

Check: Re-run the failing query with the correct chunk moved to the first position. If the answer flips to correct, you've found your bug.

Fix: Put your best-ranked chunks at the edges of the context, and keep the total context lean.

3. The chunk is right but incomplete

You retrieved the paragraph with the answer, but chunking split off the exception, the table header, the footnote, or the "unless" clause in the next chunk.

The model sees half a rule and happily completes the rest from its own priors.

Chunk A: "Customers can return items within 30 days..."
Chunk B: "...except for final-sale items and gift cards."   <- never retrieved

Fix: Use overlap, structure-aware chunking (split on headings, not on token counts), and parent-document retrieval, where you match on a small chunk but pass in its surrounding section.

4. Context vs. memory: the model trusts itself

When retrieved text disagrees with what the model "already knows," the model has to pick a side. Sometimes it ignores your context and answers from its training data. This is most common on well-known topics where your internal docs differ from public knowledge (your product's API vs. the "standard" one).

It also happens in reverse: a model can be talked out of a correct belief by a wrong retrieved passage.

Fix: Say it explicitly in the system prompt. Instruct the model to answer only from the provided context and to treat the context as authoritative even when it contradicts general knowledge.

5. Synthesis errors: right facts, wrong conclusion

Sometimes every fact in the context is correct and the answer is still wrong. The model did arithmetic badly, combined two conditions incorrectly, misread a date range, or overgeneralized ("some plans include X" becomes "all plans include X").

No retrieval improvement fixes this. It's a reasoning and faithfulness failure.

Fix: Use a stronger model for synthesis-heavy queries. Ask for step-by-step extraction before the final answer. Split the work into "extract the relevant facts" first and "answer from those facts" second.

A minimal grounded prompt

SYSTEM = """You answer using ONLY the numbered sources below.
Rules:
1. Every claim must cite a source like [2].
2. If the sources don't contain the answer, say "I don't have enough information."
3. If sources conflict, say so and prefer the most recent one.
4. Never use outside knowledge."""

def build_prompt(question, chunks):
    sources = "\n\n".join(
        f"[{i+1}] ({c.title}, updated {c.updated_at})\n{c.text}"
        for i, c in enumerate(chunks)
    )
    return f"{SYSTEM}\n\nSOURCES:\n{sources}\n\nQUESTION: {question}"

Two things make this work. Citation forcing makes it harder for the model to invent unsupported claims. And an explicit refusal path gives the model a safe way out. Without one, the model is pushed to answer even when the context is thin.

Debug in this order

When a RAG answer is wrong but the right doc was retrieved, don't guess. Work down this list:

1. Is the correct chunk actually complete? (chunking)
2. Is it buried among irrelevant or conflicting chunks? (precision, versions)
3. Does moving it to the top fix the answer? (position)
4. Does the answer contradict model prior knowledge? (memory conflict)
5. Are the facts right but the conclusion wrong? (reasoning)

Measure faithfulness, not just retrieval

Most teams track retrieval metrics (precision@k, recall, MRR) and stop there. That tells you whether the evidence arrived, not whether it was used correctly. Add:

• Faithfulness: is every claim in the answer supported by the retrieved context?
• Answer correctness against a labeled set of real questions
• Context precision: how much of what you retrieved was actually useful

Tools like RAGAS and DeepEval can score these. Run them on every change to chunking, prompts, or models, the same way you'd run unit tests.

Takeaway

RAG doesn't eliminate hallucination. It changes where hallucination happens. The model can fail on noisy context, buried evidence, split chunks, conflicts with its memory, or plain bad reasoning.

Whichever RAG solutions you build or adopt, judge them on faithfulness and answer correctness, not just on whether the right document was retrieved. Fix the failure modes in the order above, and most "the doc was right there!" bugs disappear.

Comments

[AgentSearch]

The "move the correct chunk to position one and re-run" check is such a cheap way to separate a position problem from a retrieval one, I'm stealing it. One more case I'd add to the incomplete-chunk section: when the source is a web page, boilerplate often lands in the same chunk as the answer (cookie text, nav, "related articles"), which pushes the useful sentence into the middle of the chunk and makes the lost-in-the-middle effect worse. Cleaning pages to plain markdown before chunking, and splitting on headings, fixed a lot of these for me without touching the prompt.