Measure RAG Retrieval Before You Tune the Prompt

You make the instruction more precise, add an example, and lower a generation setting. The AI answer still misses a crucial passage. Then you inspect the retrieval results: the passage never reached the generator. The team has been tuning the part of the system that could not see the evidence it needed.

Measure whether the retriever returns the relevant documents before using answer quality to judge a prompt change. This tutorial builds a small Python evaluator for recall at a cutoff and the rank of the first relevant result. It scores saved retrieval IDs, so you can compare retrieval changes without making a model call.

Disclosure: This article was drafted using generative AI. The Python example was executed against the synthetic fixtures described below.

Retrieval-augmented generation, or RAG, combines retrieved material with language generation. The original RAG research paper describes a system connecting a neural retriever and a generator. This tutorial evaluates only a retrieval stage; it makes no claim about the factual accuracy of the eventual answer.

What must be labeled before measurement?

Create a set of query IDs and a list of documents judged relevant to each query. Information retrieval evaluations often call these judgments qrels. Keep a separate ranked list of IDs returned by the retriever for each query.

For this small example, relevance is binary: a document either belongs in the known relevant set or does not. Use a stable document or chunk identity and keep the judgment unit consistent. A document-level label cannot automatically tell you whether the returned chunk contains the supporting sentence.

Save this synthetic fixture as retrieval_fixture.json:

{
  "qrels": {"q1": ["d1", "d2"], "q2": ["d3"], "q3": ["d4"]},
  "runs": {
    "q1": ["d9", "d2", "d1"],
    "q2": ["d3", "d8", "d9"],
    "q3": ["d8", "d9", "d10"]
  }
}

The fixture has three queries. The first has two relevant documents. The second has one, returned first. The third has a known relevant document that the retrieval list misses completely. These are invented IDs for testing the arithmetic, not observations from a deployed assistant.

What do recall and reciprocal rank tell you?

Recall@k is the fraction of known relevant documents appearing in the first k results. With two relevant documents, finding one gives a recall of 0.5. This follows the basic recall definition in Introduction to Information Retrieval, applied to a ranked list truncated at k.

Reciprocal rank@k is 1 / rank for the first relevant result inside that cutoff, or zero when none appears. MRR@k averages those values across queries. It tells you about the first useful result; it does not reward retrieving a second piece of evidence needed to answer a multi-part question.

Query	Relevant IDs in top three	Recall@3	Reciprocal rank@3
q1	d2, d1	1.0	0.5
q2	d3	1.0	1.0
q3	None	0.0	0.0

The averages are approximately 0.6667 and 0.5. Each query has equal weight. This is a macro average, so a query with two relevant documents does not count twice as much as one with a single relevant document.

Implement a bounded evaluator

Use Python 3.12; no third-party packages are needed. Save the following as rag_retrieval_eval.py:

import argparse
import json
from pathlib import Path


def check_ids(value, label, allow_empty=True):
    if not isinstance(value, list) or any(
        not isinstance(item, str) or not item.strip() for item in value
    ):
        raise ValueError(f"{label}: expected a list of nonblank IDs")
    if len(value) != len(set(value)):
        raise ValueError(f"{label}: duplicate IDs")
    if not allow_empty and not value:
        raise ValueError(f"{label}: relevance labels are required")


def evaluate(qrels, runs, k):
    if type(k) is not int or k < 1:
        raise ValueError("k must be a positive integer")
    if not isinstance(qrels, dict) or not isinstance(runs, dict):
        raise ValueError("qrels and runs must be objects")
    if not qrels or set(qrels) != set(runs):
        raise ValueError("nonempty qrels and runs must have identical query IDs")

    scores = {}
    for query_id, relevant in qrels.items():
        check_ids(relevant, f"{query_id} qrels", allow_empty=False)
        check_ids(runs[query_id], f"{query_id} run")
        top = runs[query_id][:k]
        wanted = set(relevant)
        recall = len(wanted.intersection(top)) / len(wanted)
        rr = next(
            (1 / rank for rank, doc_id in enumerate(top, 1)
             if doc_id in wanted),
            0.0,
        )
        scores[query_id] = {"recall": recall, "rr": rr}

    count = len(scores)
    return {
        "k": k,
        "queries": count,
        "mean_recall_at_k": sum(s["recall"] for s in scores.values()) / count,
        "mrr_at_k": sum(s["rr"] for s in scores.values()) / count,
        "per_query": scores,
    }


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("fixture", type=Path)
    parser.add_argument("--k", type=int, default=3)
    args = parser.parse_args()
    data = json.loads(args.fixture.read_text(encoding="utf-8"))
    print(json.dumps(evaluate(data["qrels"], data["runs"], args.k), indent=2))

The validation rejects duplicate IDs, missing relevance labels, invalid cutoffs, and different query sets. A missing retrieval entry must not silently disappear from an average. An explicitly empty result list is valid and scores zero.

The script requires at least one known relevant document per query. Questions genuinely lacking an answer belong in a separate abstention evaluation. Questions that simply lack human labels are incomplete evaluation data. Combining those categories into a zero denominator would obscure what you measured.

For the Ranknod editorial example here, the fixture demonstrates a retrieval diagnostic; it does not measure a brand's AI system or content quality.

Run the comparison and inspect individual failures

Run:

python3 rag_retrieval_eval.py retrieval_fixture.json --k 3

The observed Python 3.12.14 result was mean_recall_at_k: 0.6666666666666666 and mrr_at_k: 0.5, with the per-query values shown in the table. The empty-result and invalid-input behaviors described above were also checked locally. No model or vector database was contacted.

To compare a retrieval change, keep the qrels, corpus snapshot, permissions, query set, and cutoff fixed, then replace only the saved ranked runs. Inspect queries that improve and queries that regress. A better average can conceal a newly broken question your users rely on.

Do not set a universal release threshold from this tiny fixture. A real threshold needs a meaningful evaluation set, an agreed cost of failure, and an understanding of label coverage. In CI, compare a candidate against an approved baseline and send material regressions for review.

Where can these numbers mislead you?

Incomplete judgments make “recall” recall against the known labeled set. An unjudged document may be useful. Review those cases before interpreting every apparent miss as a system defect. Keep labels versioned so a label correction is distinguishable from a retrieval improvement.

Increasing k can improve recall while sending the generator more irrelevant or conflicting material. These two metrics do not measure that cost, retrieval latency, permissions, freshness, or how well the generator uses the results. A first relevant hit also cannot prove that all evidence required by a question is present.

Use the evaluator to localize a failure. When relevant material is absent, investigate indexing, chunking, filtering, and ranking. When it is present but the answer still misrepresents it, examine the generation and review stages. The point is to give the next debugging step a reason.