Evaluating RAG: measure evidence and answers separately

Build a useful evaluation set, calculate retrieval metrics, and inspect unsupported answers without confusing fluent text with correctness.

Originally published on Stack & Craft.

What you will learn: Label expected evidence, calculate Recall@k and reciprocal rank, and distinguish retrieval success from supported generation.

Prerequisites: Start with the Building and Evaluating a RAG System guide. Worked exercises use Python’s standard library and need no paid model.

Define success before measuring

An evaluation set is a collection of test questions with expected evidence and scoring rules. Include the tasks the product is meant to serve and realistic failure cases. A tiny sample is useful for learning but does not establish production reliability.

Answerable questions

Record the source passages needed to answer, including any conditions or exceptions. Some questions need more than one passage.

Which annual plans can be refunded, and what identifier is required?

Unanswerable questions

Label facts absent from the approved collection. Do not assume a plausible answer should be generated.

The policy has no bank settlement deadline.

Adversarial and access cases

Include misleadingly similar text, conflicting versions, disallowed sources and requests outside the product’s scope.

A restricted HR document must not reach an unauthorized user.

Calculate retrieval metrics carefully

Define the unit of relevance before scoring: document, passage or set of evidence. The local starter checks expected documents; finding a document does not prove its useful passage was returned.

A small worked ranking

For three questions with one expected relevant document each, suppose the first relevant result appears at ranks 1, 3 and nowhere in the top 3. Hit rate at 3 is 2/3. Mean reciprocal rank over the top-3 lists is (1 + 1/3 + 0)/3 ≈ 0.444. When a question has multiple relevant passages, recall should count the fraction of all labeled relevant passages retrieved; document hit rate alone does not capture completeness.

Score the answer separately

A retrieval hit is a prerequisite for many grounded answers, not a guarantee of them. Read the answer against its evidence. Separate factual support from relevance, completeness and format adherence.

An evaluation pass

1. Prepare held-out questions — Keep final evaluation cases separate from training examples and day-to-day prompt tuning.
2. Record retrieved evidence — Store passage IDs, rank, version and the applicable relevance labels.
3. Inspect the answer — Check important claims, omitted conditions, source mapping and missing-evidence behavior.
4. Compare operations — Measure latency and cost under a documented environment and workload.
5. Review regressions — Compare the same cases against the baseline; inspect failures rather than reporting only one average.

Use a simple human rubric

For an early project, explicit human review is easier to audit than an unexplained overall score. If an LLM helps judge responses, validate its judgments against human labels and track disagreement.

Situation	What it means	Next check
Supported answer	Important factual claims follow from the supplied evidence.	Check each key claim and its citation.
Incomplete answer	What is said is supported but a required condition is missing.	Compare against the expected evidence checklist.
Unsupported answer	At least one key claim lacks supporting evidence.	Identify the specific claim; do not reward confidence.
Appropriate abstention	The question cannot be answered from authorized evidence.	Check that the response explains the missing information.
Unnecessary abstention	The authorized evidence supports an answer, but the system refuses.	Inspect retrieval and answer instructions.

Reproduce the arithmetic

Run exercises/evaluation_demo.py. It computes the toy rankings above with assertions, then prints the result. Run the starter’s existing evaluator separately for actual sample-corpus retrieval results. Neither output is a generated-answer quality score.

What to record

Write the case count, relevance unit, top-k, model and index versions, and whether results came from a toy calculation or the actual starter. Report limitations beside the score.

Avoid evaluation leakage

If you tune repeatedly on the same questions, performance on them stops being a strong estimate of new-question performance. Reserve a final set and add real failures to a development set without silently changing the benchmark.

No universal passing percentage

Choose release requirements based on product risk and the cost of failure. A 100% score on seven simple retrieval questions does not establish safety, permission correctness or real-world answer reliability.

A practical workflow

1. Define relevance and success criteria.
2. Label answerable, unanswerable, conflicting and access-sensitive cases.
3. Measure retrieval and generated answers separately.
4. Compare the baseline using unchanged cases and documented conditions.

Practice

Create six test questions: two direct facts, one paraphrase, one requiring two passages, one missing fact and one restricted-source case. Write the expected evidence and a supported-answer rubric before running a model.

Check your understanding

Does Recall@3 measure answer correctness?

No. It measures a retrieval property under a stated relevance definition. Judge generated claims separately.

Why can a document hit still be a poor retrieval result?

The returned passage may omit the needed fact or exception even though its document ID is correct.

Can an LLM judge replace all human review automatically?

No. Validate its judgments, inspect disagreement and choose review requirements appropriate to the product.

Sources

• Original RAG research — Lewis et al.: Research background on combining retrieval and generation. Examples and exercises in this guide are original teaching material.
• Stanford: evaluation of unranked retrieval sets: Primary documentation for further reading. The worked examples here are original teaching illustrations.

Continue learning

Read the original guide on Stack & Craft for the illustrated version and related lessons.

The Local RAG Document Assistant includes the companion exercises.