Our RAG pipeline looked solid in testing.
- Documents were indexed.
- Embeddings were generated.
- Semantic search returned results.
- The LLM received retrieved context.
- Answers included citations.
- Our evaluation set was passing.
Then we put the system in front of real users.
One of them asked a question that should have been easy to answer from our knowledge base.
The system returned a confident, detailed answer.
There was only one problem:
the answer wasn't supported by our source documents.
We had a production RAG hallucination.
Our first instinct was to blame the LLM.
That turned out to be the wrong place to start.
The failure began much earlier in the pipeline.
What We Thought RAG Guaranteed
Our original mental model was roughly:
User Question
↓
Embedding
↓
Vector Search
↓
Relevant Documents
↓
LLM + Retrieved Context
↓
Grounded Answer
The assumption hiding inside this architecture was:
If we give an LLM access to our documents, it will answer from those documents.
That's not actually what RAG guarantees.
RAG gives the model retrieved context.
Whether that context contains the right evidence is an entirely different question.
A more realistic pipeline looks like this:
Question
↓
Query Interpretation
↓
Retrieval
↓
Ranking
↓
Context Construction
↓
Generation
↓
Verification
↓
Answer
Every arrow is a possible failure point.
Our investigation therefore stopped asking:
Why did the model hallucinate?
and started asking:
At what point did the evidence disappear?
That question led us to the actual problem.
Reproducing the Failure
Before changing prompts, embeddings, or models, we needed a reproducible test.
For every failed response, we captured:
{
"query": "...",
"retrieved_chunks": [],
"retrieval_scores": [],
"final_context": "...",
"generated_answer": "...",
"citations": [],
"model": "...",
"prompt_version": "..."
}
That immediately gave us something our original monitoring didn't:
visibility into the entire RAG chain.
Previously, we were mostly monitoring whether requests succeeded.
HTTP 200 ✓
Latency acceptable ✓
LLM call successful ✓
Vector DB available ✓
Operationally, everything looked healthy.
Semantically, it wasn't.
A RAG system can return HTTP 200 responses all day while giving users incorrect information.
Root Cause #1: Retrieval Returned Relevant-Looking, Not Answer-Bearing, Chunks
This was the first major lesson.
Vector similarity is not the same thing as answer relevance.
Suppose the knowledge base contains:
Document A:
Enterprise customers can request refunds
within 30 days after account cancellation.
And another document contains:
Document B:
Account cancellation requests are processed
within five business days.
A user asks:
How long after cancellation can an
enterprise customer request a refund?
Semantic retrieval might return Document B because terms such as:
- cancellation
- request
- customer
- processing
- days
are semantically close to the query.
But Document B doesn't contain the answer.
If that becomes the model's primary context, the generator is now being asked to produce an answer from insufficient evidence.
That is a retrieval failure before it becomes a generation failure.
Similarity Score ≠ Confidence
One mistake was treating retrieval scores as though they represented factual confidence.
They don't.
Conceptually:
results = vector_store.search(query, k=5)
for result in results:
print(result.similarity)
might produce:
Chunk A: 0.86
Chunk B: 0.83
Chunk C: 0.79
Chunk D: 0.77
Chunk E: 0.74
Those numbers can look reassuring.
But they answer something closer to:
How similar is this chunk to the query in embedding space?
They do not necessarily answer:
Does this chunk contain enough evidence to answer the question correctly?
That distinction matters enormously in production RAG.
Root Cause #2: Our Chunking Destroyed Context
Then we inspected the documents themselves.
Some source material looked like this:
Refund Policy
Enterprise Plan
Customers who cancel an enterprise subscription
may request a refund within 30 days.
Exceptions
Accounts terminated for policy violations
are not eligible for refunds.
But after ingestion, the information could be split into chunks resembling:
Chunk 41:
Refund Policy
Enterprise Plan
Customers who cancel an enterprise...
Chunk 42:
...subscription may request a refund within
30 days.
Exceptions
Chunk 43:
Accounts terminated for policy violations
are not eligible for refunds.
Now imagine retrieving only Chunk 41.
The topic is correct.
The actual answer isn't there.
Or retrieve Chunk 42 without enough metadata.
The answer is there, but some of the conditions surrounding it may be missing.
This is why chunk size shouldn't be treated as a magic configuration value.
chunk_size = 500
chunk_overlap = 50
is not an architecture.
It's a configuration.
Good chunking depends on the structure and meaning of the documents.
For structured business documents, useful boundaries may be:
Document
├── Section
│ ├── Policy
│ ├── Conditions
│ └── Exceptions
rather than:
Every N characters
The objective is not to create equal-sized chunks.
The objective is to preserve answerable units of information.
Root Cause #3: We Retrieved Too Much Context
Our next reaction was predictable.
If retrieval was missing information, increase top_k.
top_k = 5
became:
top_k = 15
More context should mean better answers, right?
Not necessarily.
Now the prompt contained:
Highly relevant evidence
+
Partially relevant evidence
+
Old information
+
Related but different policies
+
Repeated passages
The correct information might technically be present.
But it was competing with noise.
This changed our mental model from:
More context = More accuracy
to:
Better evidence = Better grounding
The goal of retrieval isn't to maximize the number of chunks sent to the model.
It's to construct the smallest context that sufficiently supports the answer.
Root Cause #4: Similar Documents Contradicted Each Other
Another class of failures came from document lifecycle issues.
Consider:
policy_v1.pdf
Refund period: 30 days
and:
policy_v2.pdf
Refund period: 14 days
If both remain searchable, the retriever can return both.
The LLM now receives conflicting evidence:
Context A → 30 days
Context B → 14 days
What should it do?
Unless the system knows which document is authoritative, the model has been given an ambiguity that the retrieval layer should have resolved.
This made metadata much more important.
Instead of indexing chunks with only:
{
"text": "..."
}
we needed information closer to:
{
"text": "...",
"document_id": "refund-policy",
"version": "2.1",
"effective_date": "2026-04-01",
"status": "active",
"department": "finance",
"access_level": "internal"
}
Now retrieval can apply business rules before semantic ranking.
Query
↓
Metadata Filter
↓
Semantic Retrieval
↓
Reranking
↓
Context
That is much safer than asking the model to decide which policy version is correct.
Root Cause #5: We Had No Evidence-Sufficiency Gate
This was probably our biggest architectural mistake.
Our pipeline behaved like this:
Question received
↓
Retrieve something
↓
Generate an answer
The word something was the problem.
We had implicitly assumed that every retrieval result deserved an answer.
Sometimes the correct response should simply be:
I don't have enough information in the available sources to answer that reliably.
But the pipeline didn't have a strong mechanism for reaching that state.
If weak context came back, it was still passed to the generator.
And language models are very good at completing patterns.
That is useful when generating text.
It becomes dangerous when the product requires factual grounding.
Adding an Evidence Gate
We changed the architecture conceptually to:
┌─────────────────┐
Query ──────►│ Retrieval │
└────────┬────────┘
↓
┌─────────────────┐
│ Reranking │
└────────┬────────┘
↓
┌─────────────────┐
│ Evidence Enough?│
└───────┬───┬─────┘
│ │
YES NO
│ │
↓ ↓
Generate Refuse
The exact implementation depends on the system, but the principle is simple:
No evidence, no answer.
A production RAG application needs an abstention path.
Root Cause #6: Our Prompt Encouraged Helpfulness More Than Grounding
Our original system prompt contained instructions similar to:
You are a helpful assistant.
Answer the user's question clearly and completely.
Nothing is inherently wrong with that.
But for a knowledge-grounded application, it was incomplete.
The model's priorities should be explicit.
We moved toward instructions conceptually like:
Answer only using information supported by the
provided context.
If the context does not contain sufficient
information, state that you cannot answer from
the available sources.
Do not infer missing facts.
Do not invent policies, numbers, dates, names,
or procedures.
Cite the source supporting each factual claim.
This helped.
But prompt engineering was not the root fix.
If retrieval supplies the wrong evidence, no beautifully written system prompt can manufacture the correct document.
The prompt should be a guardrail, not compensation for a broken retriever.
Root Cause #7: We Confused Citations With Grounding
Our system could generate citations.
That initially made responses look trustworthy.
Something like:
The refund period is 30 days. [Policy 4]
looks much safer than:
The refund period is 30 days.
But a citation doesn't prove that the cited source actually supports the claim.
This gave us another validation problem.
For each claim:
Generated Claim
↓
Referenced Source
↓
Does Source Support Claim?
↓
YES NO
↓ ↓
Keep Reject/Flag
Citation generation and citation verification are different features.
A trustworthy RAG system needs to care about both.
The Architecture We Wanted After the Incident
Our original architecture was essentially:
User
↓
Vector Search
↓
LLM
↓
Answer
After the RCA, the design we wanted looked closer to:
User Query
↓
Query Processing
↓
┌─────────────────────┐
│ Retrieval │
│ │
│ Semantic Search │
│ Keyword Search │
│ Metadata Filtering │
└──────────┬──────────┘
↓
Reranking
↓
Context Selection
↓
Evidence Sufficiency
↙ ↘
insufficient sufficient
↓ ↓
Abstain Grounded LLM
↓
Claim Verification
↓
Citation Checking
↓
Answer
↓
Logging + Evaluation
This is a more useful way to think about Retrieval-Augmented Generation architecture in general: RAG is a pipeline, not simply “vector database + LLM.”
And pipelines fail at interfaces.
Why Hybrid Retrieval Helped
Pure semantic search was useful, but it wasn't ideal for every query.
Some questions contained exact identifiers:
POL-1042
or product names, error codes, version numbers, and other tokens where lexical matching mattered.
That led us toward combining retrieval strategies.
Conceptually:
semantic_results = vector_search(query)
keyword_results = bm25_search(query)
candidates = merge(
semantic_results,
keyword_results
)
final_results = rerank(
query,
candidates
)
The idea isn't that hybrid retrieval magically fixes hallucinations.
It gives the system multiple ways of finding the evidence.
Semantic retrieval is strong at meaning.
Lexical retrieval is strong at exact matches.
Reranking then gives us another opportunity to ask:
Which candidate actually answers this query?
Reranking Was More Important Than Increasing top_k
Instead of passing every retrieved result to the model, we separated candidate retrieval from final context selection.
Retrieve 20 candidates
↓
Rerank candidates
↓
Select best 3–5
↓
Evidence check
↓
LLM
That distinction matters.
Retrieval optimizes for recall:
Did we find the potentially useful document?
Reranking helps improve precision:
Which of those documents is actually most useful for this question?
Those are different jobs.
We Changed What We Evaluated
Before production, our evaluation focused heavily on final answers.
Question → Answer → Correct?
After the incident, that wasn't enough.
A RAG system is composed of multiple systems, so we needed multiple evaluation points.
Retrieval evaluation
For known questions:
Did we retrieve the document containing
the expected evidence?
Useful metrics can include:
Recall@K
Precision@K
MRR
nDCG
depending on the application and evaluation design.
Context evaluation
Then:
Does the final context actually contain enough
information to answer the question?
This catches failures between retrieval and generation.
Groundedness evaluation
Next:
Are claims in the generated answer supported
by the supplied context?
Answer evaluation
And finally:
Is the answer correct and relevant to the
user's question?
These dimensions shouldn't be collapsed into one score.
A response can be:
Relevant but unsupported.
Or:
Supported but incomplete.
Or:
Correct even though retrieval failed.
That last case is particularly dangerous because the model may have answered from its pretrained knowledge rather than your approved knowledge base.
The Production Metrics We Were Missing
CPU usage wasn't going to tell us this system was hallucinating.
Neither was HTTP error rate.
RAG requires semantic observability.
The metrics we wanted to track included:
Retrieval hit rate
Retrieval relevance
No-answer rate
Groundedness
Citation support
Answer correctness
User corrections
Retrieval latency
Generation latency
Token usage
And every problematic response should be traceable.
trace_id
│
├── query
├── rewritten_query
├── retrieved_document_ids
├── retrieval_scores
├── reranker_scores
├── final_context
├── prompt_version
├── model_version
├── answer
└── citations
Without this, debugging a hallucination becomes guesswork.
With it, you can trace the failure backward.
Our Debugging Order Changed
Before this incident, the instinctive debugging sequence was:
Wrong answer?
↓
Change prompt
↓
Try another model
↓
Increase context
Now we'd investigate in this order:
1. Does the correct information exist?
If not:
Knowledge-base problem.
2. Was the correct information indexed?
If not:
Ingestion/indexing problem.
3. Was the answer-bearing chunk retrieved?
If not:
Retrieval problem.
4. Did reranking preserve it?
If not:
Ranking problem.
5. Did it reach the final context?
If not:
Context-construction problem.
6. Was the context sufficient?
If not:
Evidence-sufficiency problem.
7. Was the context correct but the output unsupported?
Now we're finally looking at a:
Generation/grounding problem.
This debugging order prevents teams from immediately changing models when the model never received the right information in the first place.
A Minimal Production RAG Checklist
Before shipping another RAG application, these are questions I'd want answered.
Data
- Are source documents authoritative?
- Are obsolete documents removed or versioned?
- Is document metadata preserved?
- Are permissions enforced during retrieval?
- Is re-indexing reliable?
Chunking
- Do chunks preserve semantic meaning?
- Are headings and document relationships retained?
- Are important conditions separated from their exceptions?
- Can a retrieved chunk stand on its own?
Retrieval
- Have we tested retrieval on real user queries?
- Are exact identifiers handled?
- Should we use hybrid search?
- Is
top_kevaluated rather than guessed?
Ranking
- Do we rerank retrieved candidates?
- Does ranking optimize for answer relevance?
Generation
- Is the model explicitly required to use supplied evidence?
- Can it abstain?
- Are unsupported assumptions prohibited?
Verification
- Do citations actually support claims?
- Can unsupported statements be detected or flagged?
Evaluation
- Do we have a representative evaluation dataset?
- Do we evaluate retrieval separately from generation?
- Are failures automatically added to regression tests?
Observability
- Can we reconstruct the exact context used for a bad answer?
- Are prompt, model, retriever, and index versions logged?
These are also the kinds of considerations that matter when designing broader enterprise RAG architecture rather than treating retrieval as a small feature attached to an LLM.
The Most Important Fix Wasn't a Better Model
One of the easiest responses to a production hallucination is:
Let's use a smarter LLM.
Sometimes that helps.
But consider this pipeline:
Wrong Document
↓
Excellent LLM
↓
Wrong Answer
A more capable model cannot reliably recover information that was never provided.
And if the correct document exists but the system retrieves the wrong one, the real engineering problem isn't model intelligence.
It's information delivery.
That's why we now think of production RAG reliability as:
Data Quality
×
Retrieval Quality
×
Context Quality
×
Generation Discipline
×
Verification
×
Observability
Weakness in any one layer can affect the final answer.
The Real Lesson From Our RAG Hallucination
RAG doesn't eliminate hallucinations.
It gives us more control over the evidence available to the model.
That's valuable, but it also means we're responsible for the entire evidence pipeline.
Our biggest mistake was thinking about RAG as:
LLM + Search
We should have thought about it as:
Knowledge System
↓
Retrieval System
↓
Evidence System
↓
Generation System
↓
Verification System
The hallucinated answer was simply the final visible symptom.
The real failure happened upstream.
So when a RAG application produces a confident wrong answer in production, don't begin by rewriting the prompt.
Start with one question:
Did the model actually receive enough trustworthy evidence to answer correctly?
If the answer is no, you don't have a prompt problem yet.
You have a retrieval problem.
Final Takeaway
The biggest lesson from investigating a RAG hallucination is that the generated response is only the end of a much longer chain.
A production-grade RAG system needs more than embeddings, a vector database, and an LLM.
It needs:
clean knowledge
+ thoughtful chunking
+ reliable retrieval
+ reranking
+ evidence sufficiency
+ grounded generation
+ citation verification
+ evaluation
+ observability
And, perhaps most importantly, it needs permission to say:
“I don't have enough evidence to answer that.”
In a production knowledge system, a careful refusal is often much more valuable than a confident guess.
Top comments (0)