A Small RAG Evaluation Harness: Test Retrieval Before Blaming the LLM

A wrong RAG answer does not automatically mean the language model failed. The system may have retrieved the wrong document, omitted the relevant passage, or supplied conflicting evidence.

This guide builds a small retrieval-only evaluation harness. It separates one question—did we retrieve the expected evidence?—from answer quality. It is an illustrative example, not a measured result from an Acadify client project.

1. Define what the test can and cannot tell you

Start with two retrieval measures:

• Recall@k: the fraction of known relevant documents present in the first k results.
• Reciprocal rank@k: the inverse rank of the first relevant result, or zero if none appears.

Recall@k helps detect missing evidence. Reciprocal rank helps distinguish evidence ranked first from evidence buried lower in the results. Neither measures whether the final answer is correct, safe, or grounded.

Use document identifiers at the same granularity as your labels. If your retriever returns chunks but your labels refer to documents, map chunk IDs to document IDs before scoring.

2. Create reviewed cases

Each answerable question needs a set of expected document IDs. Use questions representative of the actual workflow, and have someone review the labels.

For an unanswerable question, an empty set does not prove that any retrieved document is irrelevant. Here it means there is no known answer in the approved corpus. We report these separately and exclude them from recall and ranking averages.

3. Run a deterministic example

The following uses only the Python standard library. Replace the fixture retrieval function with your own retrieval API when you are ready.

from dataclasses import dataclass
from statistics import mean
from typing import Callable, Sequence

@dataclass(frozen=True)
class Case:
    case_id: str
    question: str
    relevant_ids: frozenset[str]


def score(case: Case, retrieved: Sequence[str], k: int) -> dict:
    if k < 1:
        raise ValueError("k must be positive")

    # Deduplicate while preserving rank. Score unique document results.
    top = list(dict.fromkeys(retrieved))[:k]
    relevant = case.relevant_ids

    if not relevant:
        return {"case_id": case.case_id, "answerable": False,
                "retrieved_ids": top, "recall": None, "rr": None}

    hits = relevant.intersection(top)
    first_hit = next((rank for rank, doc_id in enumerate(top, 1)
                      if doc_id in relevant), None)
    return {"case_id": case.case_id, "answerable": True,
            "retrieved_ids": top,
            "recall": len(hits) / len(relevant),
            "rr": 1 / first_hit if first_hit else 0.0}


def evaluate(cases: Sequence[Case],
             retrieve: Callable[[str], Sequence[str]], k: int) -> dict:
    rows = [score(case, retrieve(case.question), k) for case in cases]
    answerable = [row for row in rows if row["answerable"]]
    return {
        "k": k,
        "answerable_cases": len(answerable),
        "unanswerable_cases": len(rows) - len(answerable),
        "mean_recall": mean(row["recall"] for row in answerable)
                       if answerable else None,
        "mrr": mean(row["rr"] for row in answerable)
               if answerable else None,
        "rows": rows,
    }


cases = [
    Case("refund", "How do refunds work?", frozenset({"refund-policy"})),
    Case("cancel", "How can I cancel?", frozenset({"cancel-guide"})),
    Case("missing", "What is next year's price?", frozenset()),
]
fixture = {
    "How do refunds work?": ["faq", "refund-policy", "refund-policy"],
    "How can I cancel?": ["billing", "faq"],
    "What is next year's price?": ["current-pricing"],
}

result = evaluate(cases, lambda question: fixture[question], k=2)
assert result["mean_recall"] == 0.5
assert result["mrr"] == 0.25
assert result["unanswerable_cases"] == 1
print(result)

The fixture deliberately retrieves the refund document at rank two and misses the cancellation document. The averages describe these two answerable fixture cases only. They are not an estimate of production performance.

4. Inspect failures before changing the system

For each failed case, check:

1. Is the expected document present in the indexed corpus?
2. Is the label still correct for the corpus version?
3. Was the relevant text split across chunks?
4. Did metadata filters exclude it?
5. Did the query need terminology the embedding model did not match?
6. Did reranking push the evidence outside the context budget?

Change one component at a time. Run the same reviewed cases against the same corpus snapshot so you can attribute differences. Keep a held-out set; repeatedly tuning on the evaluation set can make the score improve without improving the real workflow.

5. Add answer evaluation as a separate stage

After retrieval, test the generated answer with an explicit rubric:

• Are its factual claims supported by the supplied passages?
• Do citations point to evidence supporting the associated claim?
• Does it answer the user's actual question?
• Does it ask for clarification or abstain when evidence is insufficient?
• Does it respect access controls and avoid exposing restricted data?

Automated checks and model-based judges can help triage results. Their judgments still need validation against human review. Passing a retrieval test does not establish answer correctness or authorization safety.

6. Keep the evidence reproducible

Record case ID, corpus/index version, retrieval settings, model version, prompt version and the retrieved IDs. Store passages and outputs only where you have permission; redact confidential or personal data before sharing traces.

For production comparisons, use more representative cases, repeat stochastic runs where appropriate, report the denominator and sampling limitations, and track latency and cost alongside quality.

The practical lesson: locate the failure stage first. A prompt rewrite cannot recover evidence the system never retrieved.

What is harder in your RAG workflow: creating reliable labels, evaluating citations, or reproducing retrieval failures?

---

About the author: Acadify Solution provides software and AI engineering services. Acadify AI Lab is our AI evaluation practice. This article was generated by an AI agent; its dataset and code are illustrative. Learn more at Acadify Solution or Acadify AI Lab.