Why RAG Systems Hallucinate Even When Retrieval Works

Imagine you are building an internal AI assistant for a company's HR team.
An employee asks: “Can I get a refund for my annual software subscription if I cancel within 30 days?”
The company's knowledge base contains the answer. Somewhere inside a 40-page billing policy, it says: Refunds are available within 30 days.
A few paragraphs later, however, the policy adds an important condition: This applies only to enterprise customers. Annual plans are excluded.
Your RAG pipeline receives the question, generates an embedding, searches the vector database, and retrieves the first sentence.
The LLM sees: “Refunds are available within 30 days.”
It responds: “Yes, you can receive a refund if you cancel within 30 days.”
Technically, the model didn't invent the first sentence. The retriever found a real piece of information. The problem is that the information was incomplete. And this is where many RAG systems start breaking down.
We often describe hallucination as an LLM problem: the model doesn't know something, so it makes something up. In production RAG systems, the failure can happen much earlier. The model may simply be generating an answer from bad evidence. That leads to an important distinction: Retrieval is not grounding.

Vector Similarity Doesn't Mean Answer Relevance

The first stage of a typical RAG pipeline looks simple. A user submits a query. The application converts it into an embedding and searches a vector database for similar chunks. Something like:

The assumption is that the ten returned chunks are the ten most relevant pieces of information. They aren't. They are the ten closest vectors according to the database's similarity function. Those concepts overlap, but they are not identical.
Consider a question: “Does the enterprise plan support refunds after cancellation?”
A vector database might return chunks about:

• enterprise billing
• cancellation
• refund periods
• invoices
• subscription renewals
• annual plans All of these can be semantically close to the query. But semantic similarity doesn't guarantee that a chunk contains the evidence required to answer it. A chunk can be about refunds without containing the refund eligibility criteria. It can mention annual plans without explaining their refund policy. It can discuss cancellation without establishing what happens after cancellation. This is why I think of the first retrieval stage as candidate generation, not answer retrieval. The retriever's job is to avoid missing useful information. It should cast a relatively wide net. The next stages should determine what actually belongs in the answer. ## Retrieve Broadly, Then Rerank This is where reranking becomes important. Instead of asking the vector database to perfectly identify the final five documents, retrieve a larger candidate set first. For example, retrieve 20 or 50 candidates. Then use a reranker to evaluate each candidate against the original query. The architecture becomes: Query → Vector Retrieval → Candidate Set → Reranking → Relevant Context This separation is useful because the two stages have different jobs. Vector retrieval is primarily concerned with recall. You want to avoid throwing away potentially useful information too early. Reranking is concerned with precision. Now that you have a candidate pool, you can be much more selective about what actually deserves to reach the LLM. This distinction becomes especially valuable when the knowledge base is large and contains many documents discussing the same concepts. A query about "refunds" might retrieve hundreds of things related to billing. Only a handful may actually contain the answer.

Then There Is the Chunking Problem

Even perfect retrieval cannot fix a bad chunk. This is one of the most overlooked problems in RAG. Take the same refund policy. The original document might contain: Refunds are available within 30 days.
Then immediately underneath:

• Eligibility: Enterprise customers only.
• Condition: Contract must be active.
• Exception: Annual plans are excluded. A naive chunking algorithm might separate these into different chunks. The first chunk becomes: Refunds are available within 30 days. That sentence is highly relevant to the user's question. Its embedding might even rank near the top. But it is not sufficient to answer the question. The chunk has lost the conditions that determine whether the statement actually applies. This is a subtle but important failure. The embedding model didn't misunderstand the sentence. The vector database didn't necessarily fail. The chunking strategy destroyed the relationship between the statement and its constraints. That's why chunking shouldn't simply be treated as: “Split every 500 tokens.” The correct chunk boundary depends on the structure of the information. For a technical document, an endpoint might need to stay with its parameters and constraints. For a policy document, a rule might need to stay with its exceptions. For a table, the column headers may be essential to interpreting the values. For documentation, a heading may provide the context required to understand everything underneath it. The objective isn't to create uniformly sized chunks. The objective is to create semantically complete retrieval units.

More Context Can Actually Make Things Worse

Once developers notice retrieval problems, another instinct usually appears: “Let's just send more documents to the LLM.”
If five chunks aren't enough, send ten. If ten aren't enough, send twenty. Modern models have large context windows, so why not use them? Because context capacity and context quality are two different things.
Imagine your final context contains:

• the actual refund policy
• the annual-plan exclusion
• billing portal documentation
• invoice generation instructions
• enterprise onboarding documentation
• subscription renewal information
• unrelated customer support procedures The model now has more information. But it doesn't necessarily have better evidence. The additional documents increase noise and create more opportunities for the model to connect concepts that shouldn't be connected. The goal shouldn't be to maximize the amount of context. It should be to maximize the relevance of the context. This is why a good RAG system should apply relevance thresholds after reranking rather than blindly taking the first N documents. The question should be: “Is this chunk useful enough to support an answer?” Not: “Can I fit another chunk into the context window?” ## What Happens When Retrieval Finds Nothing? This is probably the most important failure case in the entire pipeline. Suppose someone asks: “What is our refund policy for government contracts?” And your knowledge base contains no information about government contracts. The retriever returns nothing useful. What should happen? A dangerous implementation treats this as an invitation to ask the LLM anyway. The LLM has been trained on enormous amounts of information. It may know something about refund policies, government contracts, or similar business practices. So it produces an answer. And the answer sounds convincing. That's exactly the problem. The RAG system has silently changed from: “Answer using our knowledge base.” To: “Answer using whatever you know.” At that point, the retrieval layer is no longer enforcing anything. A better system has an explicit abstention path: No reliable evidence → no answer. The system should be comfortable returning: “Insufficient evidence to answer this question.” That might feel less impressive than a confident response. From a reliability perspective, it's far better. A production RAG system needs the ability to say “I don't know” when the knowledge layer doesn't contain sufficient evidence.

Grounding Is About Controlling What the Model Can Claim

Once relevant context has been selected, the generation stage becomes much easier to reason about. The prompt should establish a clear evidence boundary:
Answer using only the supplied context. If the context does not support the answer, say that there is insufficient evidence. That instruction doesn't make hallucinations impossible.
LLMs can still generate unsupported statements. But it establishes an important architectural principle: The model should not be treated as the source of truth. The retrieved evidence is the source of truth. The LLM's job is to interpret and communicate that evidence. This changes how you design the entire system.
Instead of asking: “How do I make the model know more?”
you start asking: “How do I make sure the model has the right evidence before it answers?”
That is a much more useful engineering question.

Verification Is the Missing Layer in Many RAG Systems

Even after retrieval, reranking, filtering, and generation, one problem remains. The model can still produce a claim that isn't actually supported by the context. For example, the context might say: “Refunds are available within 30 days for enterprise customers.”
The generated answer might say: “All customers are eligible for refunds within 30 days.”
The model has changed the scope of the original statement. The retrieved document was correct. The generated answer was not. This is why production RAG systems increasingly need some form of post-generation verification. The generated claims should be evaluated against the evidence used to produce them.
This can involve citation validation, entailment checks, structured answer generation, or another verification mechanism depending on the application.
The architecture becomes: Query → Retrieval → Reranking → Filtering → Context Construction → Generation → Verification → Answer
Every stage has a different responsibility. And every stage can introduce a different failure mode.

Don't Debug RAG by Only Looking at the Final Answer

When someone reports: “The chatbot hallucinated.” the first instinct is often to change the model or rewrite the prompt. That's usually too early.
First ask: What did the retriever return?
Then: Which documents survived reranking?
Then: What chunks actually reached the LLM?
Then: Was the information complete, or were important conditions missing?
Finally: Which specific claim in the answer isn't supported by the retrieved evidence?
A useful debugging trace looks like:

This turns a vague “hallucination problem” into an observable pipeline failure. Maybe retrieval recall was poor. Maybe the right document was retrieved but ranked too low. Maybe chunking separated a rule from its exception. Maybe irrelevant documents dominated the context. Maybe the model generated an unsupported claim. Maybe the system should have abstained.
Without tracing these stages, you're essentially debugging the entire system through the final sentence.

RAG Is an Information System, Not Just an LLM Feature

The simplest mental model for RAG is: Vector database + LLM. That's enough for a demo. It isn't enough for a reliable production system. A production RAG architecture looks more like:

The LLM is only one part of this chain.
If retrieval fails, generation starts with bad evidence. If chunking fails, important context disappears. If reranking fails, irrelevant information reaches the model. If filtering fails, noisy context gets passed downstream. If the system doesn't handle retrieval failure, the LLM may answer without evidence. If verification is missing, unsupported claims can reach the user.
This is why many hallucination problems in RAG are not fundamentally model problems. They're systems problems. The LLM can only generate from the information and constraints it receives. So before replacing your model, increasing its context window, or adding another prompt instruction, inspect the pipeline. Ask where the evidence was lost.
Because sometimes the model isn't hallucinating because it doesn't know the answer. It is hallucinating because your system gave it an incomplete answer and asked it to fill in the gaps. And that's the core principle behind reliable RAG:
Don't just retrieve information. Build an evidence chain strong enough to constrain what the model can claim.