Most teams ship a RAG pipeline, run a few manual tests, and call it done. Then users start complaining that answers are wrong, incomplete, or making things up. The problem is almost never the language model itself — it's that you have no systematic way to measure what's failing. This article shows you how to build a test harness that catches retrieval and generation failures before they reach production.
Why RAG evaluation is harder than it looks
RAG pipelines fail in two independent ways: the retriever surfaces the wrong chunks, or the language model ignores the retrieved context and generates text from its training data instead. These failures look identical to the user (a wrong answer) but have completely different fixes.
Standard ML accuracy metrics don't apply here. There's no single ground-truth label — just a pipeline where retrieval quality and generation quality can silently degrade whenever you change your embedding model, index configuration, or prompt. Most teams don't discover the regression until it's already in front of users.
You need three measurements at minimum: context recall (did the retriever surface what was needed?), faithfulness (does the answer stay within the retrieved context?), and answer relevancy (is the answer actually responsive to the question?).
Building the data structures
Start with minimal Python dataclasses to represent a test case and a result:
from dataclasses import dataclass
@dataclass
class RAGTestCase:
question: str
expected_answer: str
ground_truth_contexts: list[str] # passages that MUST appear in retrieval
@dataclass
class RAGResult:
question: str
retrieved_contexts: list[str]
generated_answer: str
def context_recall(result: RAGResult, test_case: RAGTestCase) -> float:
"""
Fraction of ground-truth contexts covered by retrieved contexts.
Uses substring containment; swap for cosine similarity for fuzzy matching.
"""
if not test_case.ground_truth_contexts:
return 1.0
hits = 0
for gt in test_case.ground_truth_contexts:
gt_stripped = gt.strip()
for ret in result.retrieved_contexts:
ret_stripped = ret.strip()
if gt_stripped in ret_stripped or ret_stripped in gt_stripped:
hits += 1
break
return hits / len(test_case.ground_truth_contexts)
Context recall measures the retriever in isolation. If this number is low, tuning your prompt won't help — fix the index or the chunking strategy first.
Measuring faithfulness with a judge model
Faithfulness is harder to compute without a model. You need to check whether every claim in the generated answer is actually supported by the retrieved passages. String matching won't catch paraphrasing or subtle hallucinations.
The standard approach is to use a language model as a judge — send it the answer and the contexts, and ask it to flag unsupported claims:
import httpx
import json
def check_faithfulness(
answer: str,
contexts: list[str],
api_url: str,
api_key: str,
model: str = "gpt-4o-mini",
) -> dict:
"""
Returns {"faithful": bool, "score": float, "unsupported_claims": list[str]}
Score 1.0 = every claim grounded in the provided contexts.
"""
context_block = "\n\n".join(
f"[Context {i+1}]\n{ctx}" for i, ctx in enumerate(contexts)
)
prompt = f"""You are evaluating whether an AI-generated answer is faithful to its source documents.
SOURCE CONTEXTS:
{context_block}
GENERATED ANSWER:
{answer}
Instructions:
- Identify every factual claim in the GENERATED ANSWER.
- For each claim, check if it is explicitly supported by the SOURCE CONTEXTS.
- Do NOT penalize for rephrasing — only flag claims that add new information not present in the contexts.
Return a JSON object with:
- "faithful": true if all claims are supported, false otherwise
- "score": float 0.0-1.0 (proportion of claims that are supported)
- "unsupported_claims": list of unsupported claim strings (empty list if faithful)
Return only valid JSON, no markdown."""
resp = httpx.post(
f"{api_url}/v1/chat/completions",
headers={"Authorization": f"Bearer {api_key}"},
json={
"model": model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0,
"response_format": {"type": "json_object"},
},
timeout=30,
)
resp.raise_for_status()
return json.loads(resp.json()["choices"][0]["message"]["content"])
One important detail: set temperature=0 for judge calls. You want deterministic verdicts, not creative interpretations. For high-stakes use cases (compliance, regulated environments), run the judge call twice with different prompt phrasings and take the conservative score.
Building a synthetic golden dataset
The bottleneck in RAG evaluation is almost always data — you need question-context-answer triplets to test against, and most teams don't have them. The fastest path is synthetic generation: use a language model to generate questions from your document corpus, then have domain experts review a random sample.
def generate_test_cases(
documents: list[str],
api_url: str,
api_key: str,
questions_per_doc: int = 3,
model: str = "gpt-4o-mini",
) -> list[RAGTestCase]:
"""Generate synthetic test cases from your document corpus."""
test_cases = []
for doc in documents:
prompt = f"""Generate {questions_per_doc} questions answerable using ONLY the text below.
DOCUMENT:
{doc[:3000]}
Rules:
- Each question must be answerable from this document alone.
- Avoid yes/no questions; prefer specific factual questions.
- Include the correct answer for each question.
Return a JSON array of {{"question": "...", "answer": "..."}} objects."""
resp = httpx.post(
f"{api_url}/v1/chat/completions",
headers={"Authorization": f"Bearer {api_key}"},
json={
"model": model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.4,
"response_format": {"type": "json_object"},
},
timeout=30,
)
resp.raise_for_status()
raw = json.loads(resp.json()["choices"][0]["message"]["content"])
items = raw if isinstance(raw, list) else next(iter(raw.values()), [])
for item in items:
if "question" in item and "answer" in item:
test_cases.append(RAGTestCase(
question=item["question"],
expected_answer=item["answer"],
ground_truth_contexts=[doc],
))
return test_cases
Even 50–100 test cases covering your most important document types is enough to catch regressions. Add real user queries to the dataset as you collect them — production signal is always more valuable than synthetic data.
Running the evaluation and setting CI thresholds
Wire everything together into a runner that produces a structured report:
import statistics
def run_evaluation(
test_cases: list[RAGTestCase],
rag_fn, # callable(question: str) -> RAGResult
api_url: str,
api_key: str,
) -> dict:
records = []
for tc in test_cases:
result = rag_fn(tc.question)
recall = context_recall(result, tc)
faith = check_faithfulness(
result.generated_answer,
result.retrieved_contexts,
api_url,
api_key,
)
records.append({
"question": tc.question,
"context_recall": recall,
"faithfulness_score": faith["score"],
"faithful": faith["faithful"],
"unsupported_claims": faith.get("unsupported_claims", []),
})
return {
"n": len(records),
"avg_context_recall": round(statistics.mean(r["context_recall"] for r in records), 3),
"avg_faithfulness": round(statistics.mean(r["faithfulness_score"] for r in records), 3),
"fully_faithful_pct": round(sum(1 for r in records if r["faithful"]) / len(records), 3),
"failed_cases": [r for r in records if not r["faithful"] or r["context_recall"] < 0.5],
}
For CI gates, use these thresholds as a starting point:
| Metric | Minimum threshold |
|---|---|
| Context recall | ≥ 0.75 |
| Average faithfulness | ≥ 0.85 |
| Fully faithful answers | ≥ 70% |
If retrieval recall drops below 0.75, answers will be systematically incomplete regardless of how good your model is. If faithfulness drops below 0.85, you have a hallucination problem that no amount of prompt tuning will fix — you need better retrieval or a different chunking strategy.
For regulated environments (compliance, security audits), push the faithfulness threshold to ≥ 0.95. An answer that adds unsupported claims in a compliance context is not just wrong — it's a liability. I keep security-focused checklists for AI pipelines that cover evaluation gates alongside other controls for regulated deployments.
The takeaway
RAG evaluation comes down to three things: measure the retriever and generator separately, build a golden dataset before you need it (not after), and gate every model or index change on your metrics.
Libraries like RAGAS and DeepEval implement these patterns with more statistical rigor. But understanding what the metrics actually measure means you can debug a drop in faithfulness score instead of just observing it. That's the difference between a RAG pipeline you can iterate on confidently and one that regresses silently every time someone touches the index.
I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.
Top comments (1)
The point about changing the embedding model is worth making louder than the rest, because it is not a config change - it invalidates every stored vector, and a partially reindexed corpus mixes two spaces that are no longer comparable. Recall drops for reasons no metric on your list will localise unless you version the index alongside the model. Two additions to the three measurements. First, whether the pipeline can decline: a query whose answer is not in the corpus still returns a top hit, because something is always nearest, and unless your test set contains unanswerable questions you never measure that. Second, staleness. Context recall and faithfulness both pass when the retriever surfaces an older revision of the right document and the model faithfully cites it - the answer is grounded, fluent, and out of date.