- Book: RAG Pocket Guide: Retrieval, Chunking, and Reranking Patterns for Production
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You have one number on the dashboard. Call it "RAG quality." It sits at 0.72 and has for three weeks. A user files a ticket: the bot quoted a deprecated config flag and the answer was wrong. You open the trace. The retrieved chunks were correct, the right doc was in context, and the model still made something up. Your single score never moved, because the average of a good retriever and a bad generator looks identical to the average of a bad retriever and a good generator.
That is the whole problem with scoring RAG as one system. A RAG pipeline is two systems stapled together: a retriever that finds context, and a generator that writes an answer from that context. They fail for unrelated reasons. When you collapse them into one metric, you throw away the only signal that tells you which half to fix.
Two systems, two failure surfaces
Walk a single query through the pipeline and the split is obvious.
The retriever takes the query and returns k chunks. It can fail by missing the relevant chunk entirely, by burying it at rank 9 under nine off-topic ones, or by returning stale content. None of those failures involve the LLM at all.
The generator takes those k chunks plus the query and writes an answer. It can fail by ignoring the context and answering from pretraining, by inventing a fact the context never stated, or by answering a different question than the one asked. None of those failures involve the retriever.
So you need at least two metrics that move independently:
- Retrieval metrics answer: did the right context reach the model?
- Generation metrics answer: given the context it got, did the model write a faithful, on-topic answer?
A correct answer with bad retrieval is luck. A wrong answer with good retrieval is a generator bug. You cannot tell those apart from one blended score.
Retrieval metrics: recall@k and MRR
For retrieval you need a labelled set: for each query, which document IDs are actually relevant. This is the unglamorous gold-set work, and there is no shortcut. A few hundred queries with hand-labelled relevant doc IDs is enough to start.
Recall@k asks: of all the relevant documents, what fraction showed up in the top k? If a query has 3 relevant docs and 2 of them land in your top 5, recall@5 is 0.67. This is the metric that catches "the answer was never in context." If recall@k is low, no generator on earth can save the answer.
MRR (Mean Reciprocal Rank) asks: how high up did the first relevant doc land? If the first relevant doc is at rank 1 the reciprocal rank is 1.0; at rank 4 it is 0.25. MRR catches the ordering problem: the right chunk is in context, but it is at position 9 and the model anchored on the noise above it.
def recall_at_k(retrieved_ids, relevant_ids, k):
top_k = retrieved_ids[:k]
hits = sum(1 for d in top_k if d in relevant_ids)
return hits / len(relevant_ids) if relevant_ids else 0.0
def reciprocal_rank(retrieved_ids, relevant_ids):
for rank, doc_id in enumerate(retrieved_ids, start=1):
if doc_id in relevant_ids:
return 1.0 / rank
return 0.0
def mrr(all_queries):
# all_queries: list of (retrieved_ids, relevant_ids)
scores = [
reciprocal_rank(r, rel)
for r, rel in all_queries
]
return sum(scores) / len(scores) if scores else 0.0
Recall@k and MRR together tell you two different things. Recall says "the right doc was somewhere in the top k." MRR says "and it was near the top." You want both high. Low recall means you fix chunking, embeddings, or the retriever. High recall with low MRR means you add a reranker.
Generation metrics: faithfulness and answer relevance
Retrieval metrics only need string-matching against doc IDs. Generation metrics judge free text, so they need an LLM-as-judge. The two that carry the weight are faithfulness and answer relevance.
Faithfulness asks: is every claim in the answer supported by the retrieved context? This is the hallucination detector. The judge extracts the individual claims from the answer, then checks each one against the context. Faithfulness is the fraction of claims that the context supports.
import json
from openai import OpenAI
client = OpenAI()
FAITH_PROMPT = """You score faithfulness.
Given CONTEXT and ANSWER, extract each factual
claim in the ANSWER. For each claim, mark
supported=true only if the CONTEXT states or
directly implies it. Return JSON:
{"claims": [{"claim": "...", "supported": true}]}"""
def faithfulness(context: str, answer: str) -> float:
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": FAITH_PROMPT},
{"role": "user", "content":
f"CONTEXT:\n{context}\n\n"
f"ANSWER:\n{answer}"},
],
response_format={"type": "json_object"},
temperature=0.0,
)
claims = json.loads(
resp.choices[0].message.content
)["claims"]
if not claims:
return 0.0
supported = sum(1 for c in claims if c["supported"])
return supported / len(claims)
A faithfulness score below 1.0 means the answer asserts something the context never said. That is the deprecated-config-flag bug from the opening: good retrieval, unfaithful generation.
Answer relevance asks: does the answer actually address the question? An answer can be fully faithful to context and still useless if it answers a tangent. The standard trick: have the judge generate questions that the answer would answer, then measure how close those are to the real question.
RELEVANCE_PROMPT = """Given an ANSWER, write the one
question it most directly answers. Return JSON:
{"implied_question": "..."}"""
def implied_question(answer: str) -> str:
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system",
"content": RELEVANCE_PROMPT},
{"role": "user", "content": answer},
],
response_format={"type": "json_object"},
temperature=0.0,
)
return json.loads(
resp.choices[0].message.content
)["implied_question"]
def answer_relevance(question, answer, embed_fn):
implied = implied_question(answer)
q_vec = embed_fn(question)
i_vec = embed_fn(implied)
return cosine_similarity(q_vec, i_vec)
Tools like Ragas and DeepEval ship these metrics so you do not hand-roll the prompts in production. The point of writing them out here is that the definitions are not magic: faithfulness is claim-support fraction, answer relevance is question round-trip similarity. Know what the number means before you trust it.
Reading the two-by-two
Once retrieval and generation are scored separately, the diagnosis falls out of a small table.
| Retrieval | Generation | What is broken |
|---|---|---|
| Low | High | Retriever. Fix chunking, embeddings, or add a reranker. |
| High | Low | Generator. Fix the prompt, the model, or context-stuffing. |
| Low | Low | Retriever first. A bad generator on bad context tells you nothing. |
| High | High | Ship it. Watch for regressions. |
The low-low case is the one teams get wrong. They see a bad faithfulness score and start prompt-engineering the generator, but the model is hallucinating because the context never contained the answer. Faithfulness was a red herring. Fix recall@k first, then re-measure faithfulness on the queries that now have correct context. Generation metrics are only trustworthy when retrieval is already good, because the judge scores the answer against whatever context it was handed.
This is also why the blended single score is worse than useless. A 0.72 that holds steady while retrieval quietly drops and the generator quietly improves looks like a stable system. It is two opposite regressions cancelling on your dashboard.
Wiring it into one eval run
The mechanical part is keeping the two metric families in one harness so every config change reports both. One change, one row, both halves visible.
def evaluate(query, relevant_ids, retrieve, generate,
embed_fn, k=5):
retrieved = retrieve(query)
retrieved_ids = [c["id"] for c in retrieved]
context = "\n\n".join(
c["text"] for c in retrieved[:k])
answer = generate(query, context)
return {
"recall_at_k": recall_at_k(
retrieved_ids, relevant_ids, k),
"rr": reciprocal_rank(
retrieved_ids, relevant_ids),
"faithfulness": faithfulness(context, answer),
"answer_relevance": answer_relevance(
query, answer, embed_fn),
}
Run that over the gold set, aggregate recall_at_k and mean rr into the retrieval column, average faithfulness and answer_relevance into the generation column, and you get two numbers that move for different reasons. When you swap the embedding model, retrieval moves and generation holds. When you change the answer prompt, generation moves and retrieval holds. That independence is the whole point. If a change moves both, you changed two things at once and you should split the experiment.
Re-run on every change that touches either half: new embedding model, new chunk size, new reranker, new generation prompt, new base model. The gold set is the asset. The metrics are cheap once the labels exist.
The takeaway
A RAG system has two failure surfaces and you need a metric pointed at each. Recall@k and MRR tell you whether the right context arrived. Faithfulness and answer relevance tell you whether the model used it well. Average them into one score and you lose the only information that tells you where to spend the next week.
Score the halves separately. Read the two-by-two. Fix retrieval before you trust any generation number.
The RAG Pocket Guide goes deep on the eval side: building the gold set, picking retrieval metrics that match your query types, and running LLM-as-judge faithfulness without fooling yourself. If your dashboard has one number and you cannot say which half is broken, the eval chapter is where to start.
Top comments (0)