A retrieval-augmented generation system, usually shortened to RAG, can give the right answer for the wrong reason. It can retrieve the wrong passage, ignore the passage it retrieved, and still land on a correct-looking answer because the language model already knew it. It can also retrieve exactly the right passage and still hallucinate a detail that was never in it. If you only measure whether the final answer matches a reference answer, you will miss both failures, and you will fix the wrong component when performance drops.
This article explains how to evaluate RAG as a system with two distinct jobs: finding the right evidence, and using only that evidence to answer. The distinction matters because the fixes are different. Bad retrieval needs better chunking, embeddings, or ranking. Bad generation needs better grounding, prompting, or abstention. An end-to-end score blends them into one number that tells you neither.
The two jobs of a RAG system
A RAG pipeline has a retriever and a generator. The retriever turns a question into a ranked list of chunks from a corpus. The generator conditions on the question plus those chunks and produces an answer. A failure can sit in either stage, or in the interface between them.
Consider four outcomes for a single question:
- Right retrieval, right answer. The system works as intended.
- Wrong retrieval, right answer. The model answered from its own parametric knowledge, not from your corpus. This looks like success, but it will fail silently when the corpus changes, when the fact is updated, or when the question concerns private data the model never saw in training.
- Right retrieval, wrong answer. The evidence was present, but the generator misread it, combined chunks incorrectly, or added an unsupported claim.
- Wrong retrieval, wrong answer. The system had no chance, but you still do not know whether retrieval or generation deserves the blame.
End-to-end accuracy counts only case 1 and case 2 as success. That is why a RAG system can score well on a static benchmark and still be untrustworthy in production, where the corpus is the whole point.
Evaluate retrieval on its own terms
Retrieval evaluation asks a simple question: for this query, did the system return the chunks that support the correct answer, and did it rank them highly?
Recall at k
The most basic retrieval metric is recall at k, the fraction of relevant chunks that appear in the top k retrieved results. If a question needs two specific passages and the retriever returns only one of them in its top 5, recall at 5 is 0.5 for that question.
Recall matters most when the answer requires synthesis across multiple sources. A system with high recall at 20 but low recall at 5 may still work if the generator can handle a long context, but it wastes context and increases the chance that distracting chunks mislead the generator.
Precision and ranking quality
Precision at k measures how much of what was returned was actually relevant:
Ranking quality goes further. Mean reciprocal rank looks at the position of the first relevant chunk:
Here is the set of evaluation queries. MRR rewards systems that put the best evidence first, which matters because generators pay more attention to early context, and because long contexts dilute attention.
In practice, the useful diagnostic is the pair: recall tells you whether the evidence is present at all, and ranking tells you whether the generator is likely to notice it. A retriever that returns the right chunk at position 18 is technically recalling, but in a system that only passes the top 5 chunks to the generator, it is failing.
Context precision and context recall
Two metrics popularised by RAG evaluation frameworks make the retrieval judgement finer. Context precision asks whether the relevant chunks were ranked above irrelevant ones. Context recall asks whether the retrieved set covers all the claims in the reference answer. The distinction is useful: a system can have decent recall because it returned many chunks, yet poor precision because the relevant ones are buried under noise.
The practical rule is to report retrieval metrics before you touch the generator. If retrieval recall is low, no prompt engineering will save the system. Fix chunking first, then embeddings, then the ranking step.
Evaluate generation on its own terms
Once retrieval is measured, generation needs a separate question: given the retrieved context, is the answer faithful to it, relevant to the question, and complete?
Faithfulness
Faithfulness, sometimes called groundedness, measures whether every claim in the answer is supported by the retrieved context. It is the single most important generation metric for RAG, because a RAG system that hallucinates is worse than a plain language model: it carries the appearance of being grounded in your documents while behaving exactly like an ungrounded model.
A practical way to estimate faithfulness is claim decomposition. Break the answer into atomic claims, then check each claim against the retrieved chunks:
This can be done by a human annotator, or by a strong language model acting as a judge with the context provided and strict instructions to mark a claim unsupported unless the context states it or directly entails it. The judge approach scales, but it introduces its own bias, which we return to below.
A common failure mode is subtle: the answer is mostly faithful, but one number, date, or name is changed. Claim-level scoring catches this where a holistic "looks right" judgement does not, because a single unsupported claim in an otherwise good answer still makes the answer unreliable for decisions.
Answer relevancy
Answer relevancy asks whether the answer actually addresses the question asked, rather than answering a nearby question that the retrieved chunks happen to support. A system can be perfectly faithful and still useless if it summarises the retrieved passage instead of answering.
A simple operational check is to generate questions from the answer alone, then measure how similar those questions are to the original question. If the answer would prompt a very different question, it probably drifted. In practice, most teams use a judge model with a rubric: does the answer address the question directly, completely, and without irrelevant padding?
Completeness
Completeness is the generation counterpart of context recall. It asks whether the answer covers the key points in the reference answer that the retrieved context supports. A short, faithful answer that omits half the required content is not a good answer, even though it scores perfectly on faithfulness alone.
The three generation metrics together form a useful triangle: faithfulness without completeness gives terse, evasive answers; completeness without faithfulness gives confident hallucinations; relevancy without either gives fluent off-topic text. Report all three, not a single blended score.
A worked example: separating the blame
Suppose you have a support assistant over product documentation, and a user asks which plan includes single sign-on. The reference answer is that single sign-on is available on the Enterprise plan only.
Case A: the retriever returns pricing pages for the wrong product. The generator, guessing from general knowledge, says Enterprise. End-to-end accuracy says pass. Retrieval recall says fail, and faithfulness says fail, because the retrieved chunks do not support the claim. The correct fix is retrieval, not generation.
Case B: the retriever returns the correct pricing page. The generator says single sign-on is available on Pro and Enterprise. Retrieval recall is perfect. Faithfulness fails on the Pro claim. The correct fix is generation: stricter grounding instructions, or a verification step that checks each claim against the context before returning the answer.
Case C: the retriever returns the correct page at position 12, but only the top 5 chunks reach the generator. The generator says it does not know. Retrieval recall at 20 looks fine, but recall at 5, the number actually passed to the generator, is zero. The fix is to increase k, improve ranking, or compress chunks so more evidence fits.
In each case the end-to-end label is misleading, and the component metrics point to a different intervention. That is the whole argument for component evaluation.
A minimal evaluation harness
You do not need a large framework to start. A small, honest harness beats a large, opaque one. The structure below is enough for a first evaluation set of 50 to 100 questions with known relevant chunks and reference answers.
def evaluate_rag(questions, rag_system):
rows = []
for q in questions:
result = rag_system.answer(q.text)
retrieved_ids = [c.id for c in result.chunks]
relevant = set(q.relevant_chunk_ids)
recall_at_5 = len(relevant & set(retrieved_ids[:5])) / max(len(relevant), 1)
claims = split_into_claims(result.answer)
supported = sum(1 for claim in claims if is_supported(claim, result.chunks))
rows.append({
"question": q.text,
"recall_at_5": recall_at_5,
"faithfulness": supported / max(len(claims), 1),
"answer": result.answer,
})
return rows
The hard parts are not in this code. They are in building the evaluation set: choosing questions that reflect real user queries, labelling which chunks are truly relevant, and deciding what counts as support for a claim. Spend your effort there. A benchmark built from questions the system already handles well will flatter every component.
Two practical tips help. First, include questions whose answer is not in the corpus at all, and check that the system abstains or says it does not know, rather than hallucinating. Abstention rate on unanswerable questions is a first-class metric for a production RAG system. Second, include questions with updated facts, where the corpus contradicts the model's training knowledge. If the system answers from training data instead of the corpus, your faithfulness metric should catch it, and it is a sign that grounding needs work.
Common pitfalls in RAG evaluation
Using an LLM judge without checking the judge
LLM-as-judge evaluation is convenient and often necessary, but the judge has biases: it prefers longer answers, fluent answers, and answers that match its own style. It can also be fooled by the same retrieval failures it is supposed to detect, if it is given the same misleading context without the reference answer.
Mitigate this by calibrating the judge on a human-labelled subset, reporting agreement with human judgements, and using the judge for ranking systems rather than for absolute scores. If two systems differ by a small margin under one judge, that difference may not survive a different judge or a human panel.
Evaluating on synthetic questions only
Synthetic questions generated from chunks tend to be too clean: they use the same wording as the chunk, they ask about a single fact, and they never contain the ambiguity, typos, or multi-part structure of real queries. A retriever that scores highly on synthetic questions can fail badly on real ones. Mix synthetic questions with real user queries, sampled and anonymised, and label a subset by hand.
Ignoring the cost of context
Retrieval metrics that reward returning more chunks ignore a real trade-off: longer context costs more, increases latency, and can degrade generation quality as the model attends to distractors. Report metrics at the k you actually use in production, and track cost and latency alongside quality. A system that needs 20 chunks to reach the recall of a competitor's top 5 is not equally good, even if its recall at 20 matches.
Treating a single number as the verdict
No single metric captures a RAG system. A useful evaluation report shows, at minimum: retrieval recall and precision at production k, faithfulness, answer relevancy, completeness, abstention rate on unanswerable questions, and latency. When a change improves one metric and harms another, that trade-off should be explicit, not hidden inside an average.
Limitations of this approach
Component evaluation is not free. Labelling relevant chunks and reference answers is expensive, and it bakes in judgements about what counts as relevant that may not match real user needs. Claim decomposition can split a nuanced answer into claims that are individually true but jointly misleading. And metrics computed on a fixed evaluation set will drift as the corpus changes, so an evaluation set built this year may be stale next year.
The response is not to abandon evaluation, but to treat it as a living process: refresh the evaluation set as the corpus and the user base change, re-label a sample periodically, and keep a human in the loop for the cases where metrics disagree with each other. When retrieval and generation metrics point in opposite directions, that disagreement is itself information, and it usually marks the most interesting failure to investigate.
Takeaway
End-to-end accuracy is a useful smoke test, but it is a poor diagnostic. Evaluate retrieval and generation separately, report faithfulness alongside recall, include unanswerable questions, and calibrate any LLM judge against human labels. The goal is not a higher benchmark score. It is a system whose failures you can locate, explain, and fix, and whose answers a user can trust because they are tied to evidence you can show.
Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial
Top comments (0)