Part 7 of 8 — RAG Article Series
Previous: RAG, Fine-Tuning, or Long Context? (Part 6)
The Team That Blamed the Model
TechNova's RAG system worked well at launch. Return policy questions got correct answers. Troubleshooting queries surfaced the right procedures. The team shipped, moved on to other work, and checked the dashboard occasionally.
Three months later, support tickets started referencing bad AI answers. A customer was told the return window was thirty days. Another got a troubleshooting procedure that did not match their firmware version. The team's first instinct: the model must be degrading. They started evaluating newer, more expensive models.
The root cause was not the model. TechNova's return policy had changed from thirty days to fifteen days after launch, but the ingestion pipeline had not been re-run. The old chunks were still in the index. The retriever was faithfully returning outdated content. The model was faithfully generating from it. Both were doing their jobs. The data between them was stale.
This is the failure that evaluation exists to catch. Not "is the model good enough?" but "is the system returning the right answers, and if not, which part is wrong?"
A wrong answer can come from several places. The retriever can return the wrong chunks. The right chunks can be lost or changed during prompt assembly. Or the model can mishandle the context it actually received. To the user, all three can look identical — a confidently incorrect response. They are different problems and they do not have the same fix. The rest of this article separates them, because useful RAG debugging starts with identifying which layer failed.
Retrieval Metrics
Retrieval metrics answer one question: did the retriever return the right content? These metrics evaluate what happened before the model saw anything.
Context Precision
Of the chunks you retrieved, how many were actually relevant to the question? If you retrieve five chunks and three are useful, precision is 60%. The other two are noise — irrelevant content that the model has to read, reason about, and hopefully ignore. High noise means the retriever is casting too wide. The fix is usually in chunking (smaller, more focused chunks) or retrieval approach (adding reranking — a second pass that re-orders the retrieved chunks — or switching to hybrid search).
Context Recall
Of all the relevant content in your knowledge base, how much did you retrieve? If the correct answer requires information from two chunks and the retriever found both, recall is 100%. If it found only one, recall is 50% and the model is generating from incomplete information. Low recall means you are missing signal — the right content exists but the retriever did not find it. The fix is usually increasing the number of chunks retrieved (top_k), improving the embedding model, or adding query expansion — approaches that widen what the retriever finds.
Mean Reciprocal Rank
Was the best chunk ranked near the top? For a single query, the reciprocal rank is one divided by the position of the first relevant chunk: position 1 gives 1.0, position 3 gives 0.33. Mean Reciprocal Rank is the average of those reciprocal ranks across a set of test queries, so it tells you how high the first relevant result tends to appear. This matters because many systems use only the top 1–3 chunks for prompt assembly. If the best chunk is consistently at position 4 or 5, it never reaches the model. If recall is healthy but MRR is low, reranking may help because the retriever is finding the right content but not ranking it high enough.
One limitation to keep in mind: MRR only looks at the first relevant chunk. For a question that needs several chunks to answer, MRR can look excellent while a second required chunk sits far down the list. When answers depend on combining multiple chunks, lean on recall rather than MRR to see whether all the needed content was retrieved.
Generation Metrics
Generation metrics answer a different question: did the model use the retrieved context correctly? These metrics only make sense after you have confirmed that retrieval is working. If the retriever returned the wrong chunks, generation metrics tell you nothing useful.
A note on what not to use. BLEU and ROUGE — common metrics for comparing generated text to a reference answer — are the wrong tool for RAG. They measure surface overlap with a reference answer, which can be useful in tasks such as translation and summarization when reference wording is meaningful. For RAG, wording overlap is a poor proxy for whether an answer is grounded in the retrieved context. RAG has no single correct answer; it has a correct answer for the retrieved context. A faithful, relevant response can score poorly on BLEU if its wording differs from the reference, and a plausible-sounding hallucination can score well. The three metrics below measure what actually matters: did the model stick to the retrieved context, did it answer the question, and did it cover what the context supports.
Faithfulness
Did the model stick to the context it actually received, or did it add facts that were not in any chunk? A faithful answer draws only from the context the model was given at generation time. An unfaithful answer introduces information the model pulled from its training data — which may be outdated or wrong. This is the RAG-specific version of hallucination: the model had the right context in front of it and still generated beyond it. One caution from the diagnostic spine: before calling an answer unfaithful, confirm the supporting chunk actually reached the model. If it was retrieved but dropped during prompt assembly, that is an assembly problem, not a faithfulness problem.
TechNova example: the retriever returns the correct return policy chunk (15 days), and that chunk reaches the model, but the answer adds "You can also exchange the product within 30 days" — a fact from training data that is no longer true. Retrieval was correct, the context was intact, and the generation was unfaithful.
Answer Relevance
Did the model actually answer the question that was asked? A relevant answer addresses the user's query directly. An irrelevant answer may be factually correct but off-topic. If the user asks about the return policy and the model responds with warranty information — even though the warranty chunk was correctly retrieved alongside the return policy chunk — the answer is irrelevant. The model chose to answer from the wrong chunk.
TechNova example: the customer asks "How do I reset my WH-1000?" The retriever returns both the troubleshooting guide and the return policy. The model answers with the return process. Factually correct, but irrelevant to the question.
Completeness
Did the answer cover what the context supports? A complete answer addresses all the conditions and details present in the retrieved chunks. An incomplete answer cherry-picks. If the return policy chunk says "15 days from date of delivery, original packaging required, 7-day window for refurbished products purchased through the TechNova Outlet," and the model responds only with "15 days," it is faithful and relevant but incomplete. The customer may return a refurbished Outlet item expecting 15 days and get denied.
A note on metric names. Faithfulness and answer relevance are also implemented by evaluation frameworks such as RAGAS, but the definitions here are teaching versions meant to build intuition. A framework may compute them differently, and completeness as used here is a custom lens for this article rather than a standard framework metric, so do not expect the exact numbers or names to line up when you run one of these tools. The point is the diagnostic question each metric asks, not a specific formula.
The Diagnostic Spine
This is the single most important debugging habit in RAG: when the answer is wrong, inspect the retrieved chunks first.
If the chunks are wrong — irrelevant, stale, too broad, from the wrong document — the problem is retrieval. No amount of prompt engineering or model upgrading will fix it. The model is generating from bad input.
If the chunks are right, there is one more thing to check before you blame the model. The chunks your retriever returned and the chunks the model actually saw are not always the same. Look at the final assembled prompt. Context that overflowed the token budget gets truncated, a template bug can drop or mangle a field, conversation history can crowd out the retrieved evidence as a chat grows, and chunk order can change what the model attends to. If the right chunk never made it into the prompt intact, that is an assembly problem, and tightening the prompt wording or swapping models will not fix it either.
If the chunks are right and they reached the model intact, and the answer is still wrong, the model hallucinated beyond the context, misinterpreted a condition, or ignored a relevant chunk — the problem is generation. Tighten the prompt, lower the model's temperature setting (the setting that controls randomness), or try a model that follows instructions more closely.
Four diagnostic signals have appeared across this series. Fluent but wrong answers — well-structured, confident, incorrect — most often mean the retriever returned the wrong chunks, though the same symptom can come from a chunk that was retrieved but never reached the model, or from the model hallucinating past good context. Vague or hedging answers ("the return policy may vary") usually mean the chunks are too broad or generic — a chunking problem. Contradictions across sessions ("thirty days" today, "fifteen days" tomorrow) point to stale data in the index alongside current data — the data freshness problem Part 8 addresses. And correct but irrelevant answers usually mean adjacent content was retrieved instead of the right one, or the model picked the wrong chunk from a right retrieval — check retrieval first, and if the chunks are good, it's a generation-side selection issue.
The same four signals collapse into a quick lookup table when you are debugging in the middle of an incident:
| User-visible symptom | Likely issue area | First thing to inspect |
|---|---|---|
| "AI says it doesn't know, but the answer is in the docs." | Usually retrieval (the right chunk was not returned); sometimes generation (the model abstained despite having enough context) | Context recall. Inspect the retrieved chunks first; if the needed evidence is present, check the assembled prompt and the model's abstention instructions. |
| "Answer is detailed and confident but factually wrong." | Usually retrieval (wrong chunks); sometimes assembly (right chunks did not reach the model intact) or generation (hallucinated beyond context) | Inspect retrieved chunks first. If chunks are right, check the assembled prompt, then faithfulness. |
| "Answer is correct but off-topic." | Retrieval (adjacent content) or generation (wrong chunk selected) | Context precision. Then answer relevance. |
| "System gives different answers across time for the same question." | Data freshness — stale and current chunks both in the index | Inspect the index for duplicates and version conflicts. (Covered in Part 8.) |
LLM-as-a-Judge
Manually inspecting every answer is not sustainable. LLM-as-a-judge uses a model to evaluate another model's outputs automatically: you give the judge the question, the context the model was given, and the generated answer, and ask it to score faithfulness, relevance, and completeness on a 1–5 scale with a short written reason.
The shape of a faithfulness judge prompt is small enough to sketch:
You are evaluating a RAG answer for faithfulness.
Question: {question}
Retrieved context: {chunks}
Generated answer: {answer}
Score the answer's faithfulness from 1 to 5,
where 5 = every claim is supported by the context,
3 = the answer contains claims not supported by the context,
and 1 = the answer contradicts the context.
Return: score, one-sentence reason.
The middle anchor matters. The most common faithfulness failure is not an answer that contradicts the context. It is one that adds claims the context never supports, without contradicting anything. If the scale only defines the top and bottom, judges scatter that common case unpredictably. Anchoring the middle gives the judge a clearer boundary for this common failure case and usually improves scoring consistency.
The same shape works for answer relevance and completeness — only the criterion in the scoring instruction changes.
A few refinements worth knowing. Judge prompts are usually rubric-based — anchored at each score level rather than left to the model's interpretation, which usually improves evaluator consistency. When comparing two versions of a system, teams often switch to pairwise evaluation ("which answer is better?"), which is more sensitive than absolute scores at small differences. Pairwise judges can be sensitive to answer order, so randomize the order, or run both orders and compare the results, before trusting a small difference. And before you rely on a judge's trends at all, check it against a sample of human-labeled answers. Until you have seen that its scores track human judgment on your own data, treat its trends with caution.
Judges have predictable biases beyond simple inconsistency. They tend to favor longer, more verbose answers, they can be sensitive to answer order in pairwise comparisons, and they can favor outputs from their own model family. You can blunt these: score against a rubric rather than an open scale, randomize pairwise order, and use a different or stronger model as the judge than the one that generated the answer.
The value of running a judge is interpretation. When faithfulness drops week over week, inspect what changed in the context and generation path: prompt assembly, retrieved evidence reaching the model, system instructions, or the model itself. When answer relevance drops while faithfulness holds, the retriever is likely pulling adjacent-but-off-topic content. The trend line is what matters, not the single run. On a small evaluation set, expect noise: with 20–50 queries, one or two flipped judge scores can move the aggregate several points on their own, so react to sustained shifts rather than single-run wobble, and investigate the specific samples a run flags.
The advantage is throughput — a judge can score thousands of answers in the time a human scores ten — at the cost of subtlety and consistency. A judge model can miss subtle hallucinations that sound plausible but are not in the context. It can be inconsistent: the same answer may score 4 on one run and 3 on the next. LLM-as-a-judge is a useful automation layer, not a replacement for human evaluation. Use it for continuous monitoring. Use human review for building and validating your evaluation set, and for investigating failures the judge flags. And don't overlook the cheapest form of human signal — thumbs-up/thumbs-down buttons in the production app give you a continuous stream of real-user feedback, and the negative ones are your next eval-set candidates.
Building an Evaluation Set
Every metric in this article requires test queries with known-good answers. Without them, you are measuring nothing.
Start with 20–50 queries, manually curated. For each query, record: the question, the expected answer, and which chunks should be retrieved. This is tedious but irreplaceable — the quality of your evaluation set determines whether your metrics catch real problems or generate false confidence.
Once you have a curated foundation, synthetic generation is a useful coverage extender — frameworks like RAGAS can generate test queries directly from your documents, including multi-hop questions that require combining chunks. Treat the generated set as a complement to the curated one, not a replacement: the curated set is your human-verified ground truth, the synthetic set is your reach. Whatever the synthetic generator produces, the answers it grades against should still be checked by a human.
A good evaluation set is not a long list of similar questions. It is a small, deliberate mix of query shapes that stress different parts of the pipeline. For TechNova's product support corpus, that mix looks roughly like this: a straightforward factual lookup ("What is the warranty period on the WH-1000?") tests whether the retriever can find a single canonical chunk; a boundary or condition question ("Can I return a refurbished WH-1000 from the TechNova Outlet after 10 days?") tests whether the model honors qualifiers in the retrieved chunk instead of giving the headline answer; a multi-condition or multi-chunk question ("What is covered under warranty if I bought it refurbished?") tests whether the system can combine information from two chunks — warranty terms and refurbished-product policy; and a stale-data or version-sensitive question ("What does firmware v3.2 fix?") tests whether the index reflects the current changelog and not an older version. A handful of queries from each category will surface more failure modes than fifty variations of a single shape.
A "known-good answer" is not an exact reference string the model has to match word for word. It is a set of facts and conditions the answer must include to be considered correct. For the refurbished-return question, that set might be: 15-day standard window, original packaging required, 7-day window for refurbished products purchased through the TechNova Outlet. The phrasing the model uses does not matter; the presence of those facts does. This is also why faithfulness, answer relevance, and completeness are useful metrics here — they evaluate the answer against the retrieved context and the required facts, not against a fixed reference string.
One caution on where the ground truth lives. Store the expected answers and required-fact lists as separate evaluation data, outside the indexed corpus, not inline in the documents the retriever searches. If gold answers sit inside indexed content, they leak into retrieval and the eval starts measuring against text the retriever can see. It also invites a subtler bias: writing eval queries to match the wording of the gold chunk rather than the way real users ask. Keep the eval set beside the corpus, not inside it.
Sources for good evaluation queries: real customer questions from your support logs, edge cases you discovered during the Part 5 build, and questions that exercise the specific retrieval challenges your documents create.
Run your retrieval pipeline against the evaluation set after every change. Compare retrieval metrics before and after. If precision dropped, you introduced noise. If recall dropped, you lost signal. If MRR dropped, ranking degraded. Without this discipline, optimization is guesswork. This is the offline half of evaluation; the other half is monitoring real production queries and responses and feeding the failures you find back into the curated set — the offline set defines what you measure, production tells you what you missed.
The evaluation set is not a one-time artifact. As documents change — the return policy is updated, a new firmware version ships, a product is retired — the expected answers and the chunks the retriever should return must be updated alongside them. An evaluation set that drifts out of sync with the corpus quietly produces false failures and, worse, false confidence.
In practice, most teams do not build every scorer from scratch. Common starting points are RAGAS (open-source, metric implementations, test-set generation), LangSmith (LangChain-ecosystem traces and evaluation workflows), and the evaluation features built into cloud platforms like Amazon Bedrock and Vertex AI. Pick whichever fits your stack — the patterns above apply either way.
Three Takeaways
1. Separate retrieval metrics from generation metrics — they diagnose different problems. Retrieval metrics tell you whether the right content was found. Generation metrics tell you whether the model used it correctly. Fix retrieval first.
2. When a single answer is wrong, inspect the retrieved chunks first, then check what actually reached the model. For a regression across many answers, start from what changed and from your metrics instead. Wrong answer, inspect chunks, then assembly, then generation. This is the single most important debugging habit in RAG.
3. Start with a small evaluation set of 20–50 curated queries. Expand from real user questions. Manually curated test queries with known-good answers. Run them after every change. Without measurement, optimization is guesswork.
You can measure it. Now ship it safely. Metrics tell you what is wrong today. They do not tell you what will quietly go wrong six months from now — when the policy changes, the index drifts, a prompt-injection slips past the judge, and the dashboard still looks green. Part 8 is about that gap: what it takes to keep a RAG system correct in production after the launch adrenaline wears off.
Next: RAG in Production: What Breaks After Launch (Part 8 of 8)
TechNova is a fictional company used as the running example throughout this series.
Sample code: github.com/gursharanmakol/rag-in-practice-samples
Part of the RAG in Practice series, originally published on AI in Practice Hub alongside the full series and companion code.
Top comments (2)
The diagnostic spine is the kind of thing that seems obvious after you read it, but I've watched teams do exactly the opposite in production incidents. Wrong answer? Must be the model. Swap it. Wrong again? Must be the prompt. Rewrite it. Nobody checks the chunks because the chunks are infrastructure and infrastructure is supposed to be boring.
What lands for me is the quiet implication that most RAG failures are actually data engineering problems dressed up as AI problems. The TechNova return policy example is almost too perfect—both the retriever and the model did their jobs correctly. The system was working. The data pipeline wasn't. But the support tickets don't say "data pipeline failure." They say "the AI is giving wrong answers." The nomenclature shapes where the debugging effort goes.
The evaluation set lifecycle is the other part I'm turning over. The idea that an eval set has a shelf life—that it can drift out of sync and start generating false confidence while the dashboard stays green—feels like the thing nobody budgets for. It's not hard to build an eval set once. It's hard to maintain it as the corpus changes, and the maintenance burden grows silently. Six months in, you're measuring against questions that no longer match the documents you're retrieving from, and the metrics are lying to you.
I wonder how much of this gets solved by tying eval set versioning to document versioning directly—like, when the return policy doc gets updated, the eval queries that reference it get flagged for review automatically. Otherwise it's a manual discipline, and manual disciplines decay. Has anyone built that coupling into their pipeline, or is it still in "we'll get to it" territory?
Your "data engineering problems dressed up as AI problems" framing is sharper than anything I wrote. The nomenclature really does shape where engineers look first.
On eval-set versioning tied to documents, I have not seen anyone publish a really clean implementation yet. Most of the patterns I have come across are discipline-based, and disciplines decay.
That is exactly the territory Part 8 is going to push into.