3 bugs I hit wiring DeepEval into my RAG pipeline (and what each one taught me about RAG evaluation)

Everyone shows you the happy path of RAG evaluation: import a metric, pass a test case, get a green check, tweet the screenshot. Then you wire it into CI, point it at a real pipeline, and the numbers make no sense — a metric fails while the answer is obviously correct, or passes while the answer is quietly missing half the data.

I recently rebuilt a small RAG pipeline (OpenAI embeddings → Pinecone →'gpt-4o-mini' over a CSV of customer orders) and turned DeepEval into a real pass/fail gate on top of it. This post is the three bugs that actually taught me how RAG evaluation works — not the docs, the bugs.

If you only take one thing away: a green eval suite is worthless if you don't understand what each metric is measuring.Let's earn the green.

The setup: eval as a gate, not a vibe

The whole point of an eval metric (as opposed to an eval score) is a threshold. Below it, the build should fail. DeepEval leans into this — every metric has a 'threshold' and an 'is_successful()', so 'assert_test()' turns your RAG quality into a CI check.
Here's the shape of the harness. Each sample carries the question, the answer the pipeline produced, the chunks it retrieved, and the ground truth:

python
from dataclasses import dataclass

@dataclass
class EvalSample:
    question: str
    answer: str            
    contexts: list[str]    
    ground_truth: str      

And the translation into a DeepEval test case — this innocent-looking function is where bug #1 lives:

python
from deepeval.test_case import LLMTestCase

def _to_test_case(s: EvalSample) -> LLMTestCase:
    return LLMTestCase(
        input=s.question,
        actual_output=s.answer,
        retrieval_context=s.contexts,     
        expected_output=s.ground_truth,
        context=[s.ground_truth],         
    )

Two fields look interchangeable and are not: 'retrieval_context' is what your retriever pulled, 'context' is the ground-truth reference. Confusing them is bug #1.

Bug #1: the Hallucination metric that punished a perfect answer

My 'FaithfulnessMetric' said '1.0' — the answer was fully grounded in the retrieved chunks. My 'HallucinationMetric' on the same test case said 0.833 and failed. Two grounding metrics, opposite verdicts. One of them was lying.

The mistake was mine. I had built the test case like this:

python
# WRONG
context=s.contexts,   # all 6 retrieved chunks

Here's the trap. 'HallucinationMetric' in DeepEval is lower-is-better, and it checks the answer against 'context' treated as factual ground truth. My retriever returned 6 chunks for a question that only needed 1. So DeepEval looked at the answer, compared it to all 6 "facts," and correctly concluded the answer "contradicted" the 5 orders it (rightly) ignored. The metric wasn't broken — I had fed it noise and called it truth.

The fix is one line, and it encodes a real conceptual distinction:

python
# RIGHT
context=[s.ground_truth],   

Score dropped to '0.000' and passed, 3/3.

The lesson: 'retrieval_context' and 'context' are different inputs on purpose.

• Faithfulness asks: is the answer supported by what we retrieved? → feed it 'retrieval_context'.
• Hallucination asks: does the answer contradict known truth? → feed it 'context' = ground truth.

Your retrieved chunks are evidence, not truth. Passing noisy retrieval in as "truth" makes a correct answer look like a liar. This distinction is also a great interview answer, by the way.

Bug #2: the retrieval recall miss no generation metric could see

Question: "Which orders were returned?" The pipeline answered confidently, listed the returned orders, cited them — and silently dropped order 1011.
Faithfulness: perfect.
Answer relevancy: perfect.
Every generation-side metric was green, because the answer was faithful and relevant to the chunks it got. The bug was upstream: dense top-k retrieval never fetched 1011's chunk, so the generator never had a chance.

This is the single most important thing about RAG evaluation:

Faithfulness measures the generator. It cannot see a retrieval recall miss. If the chunk never arrives, a perfectly faithful answer will confidently omit it.

Two fixes, both worth knowing:

1. Metadata-filtered retrieval for anything that's really an aggregation ("which / how many / list all"). For structured data, don't make cosine similarity guess a category — filter on it:

# 'status=Returned' -> {"status": {"$eq": "Returned"}}
retriever = vectorstore.as_retriever(
    search_kwargs={"k": top_k, "filter": {"status": {"$eq": "Returned"}}},
)

Once "Returned" is a hard filter instead of a fuzzy vector match, recall stops depending on luck.

1. Measure retrieval separately with retrieval-only metrics — hit@k, MRR, precision@k, recall@k — so a recall miss shows up as a retrieval number, not as a mysteriously incomplete answer. 'ContextualRecall' in DeepEval is the gate for exactly this.

The lesson: a RAG eval that only scores the final answer is half an eval. Score the retriever and the generator as separate stages, or retrieval failures will keep hiding behind faithful-sounding prose.

Bug #3: the red metric I refused to turn green

'ContextualRelevancyMetric' came back 0/3. My instinct was to lower its threshold until it passed. I didn't — because it wasn't wrong.

At 'top_k = 6' on a question that needs one order, five of the six retrieved chunks are irrelevant. The metric measures signal-to-noise in what you retrieved, and my signal-to-noise was genuinely 1/6. A red 'ContextualRelevancy' next to a green 'Faithfulness' isn't a contradiction — it's a diagnosis: the answer is correct, but the retriever is dragging in junk that costs tokens, latency, and (at higher noise levels) accuracy.

The honest fixes are architectural, not cosmetic: lower k, add a reranker, or route aggregation queries to a metadata filter (see bug #2). Gaming the threshold would have deleted the one signal telling me my retriever needed a reranker.

The lesson: not every red metric is a bug in your eval. Some are your eval doing its job. The discipline is telling "this metric is misconfigured" (bug #1) apart from "this metric is reporting a real weakness" (bug #3) — and never moving a threshold just to get a green build.

Bonus: when the built-in metrics can't express what you mean — GEval

The metrics above are general. Real apps have bespoke rules. Mine was: the answer must cite the correct '[order NNNN]' id for every fact it states. No built-in metric knows what "the correct order id" means for my data.

That's what 'GEval' is for — a custom LLM-as-judge metric built from a natural-language rubric. It's a direct implementation of the G-Eval paper: you describe the criterion, DeepEval turns it into chain-of-thought evaluation steps, and the judge LLM scores 0–1 against a threshold.

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

citation_correctness = GEval(
    name="CitationCorrectness",
    criteria=(
        "Decide whether the answer cites the correct [order NNNN] id(s) "
        "for the facts it states. Penalize missing or wrong citations."
    ),
    evaluation_params=[            # which fields the judge is allowed to read
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
        LLMTestCaseParams.RETRIEVAL_CONTEXT,
    ],
    threshold=0.7,
)

Two things to respect if you use it:

• It's an LLM judge, so it has bias, cost, and variance. Pin the judge model, set temperature 0, and prefer explicit 'evaluation_steps' over a one-line 'criteria' when you need run-to-run consistency.
• Validate it against human labels before you trust it as a gate. A custom metric is only as good as its correlation with what you actually care about. And don't reach for an LLM judge where a regex or schema check would be cheaper and exact.

Putting it together as a CI gate

Once the metrics are configured correctly, the gate is boring in the best way:

from deepeval import assert_test

def test_orders_pipeline_quality():
    samples = build_eval_samples_from_pipeline(...)
    metrics = build_deepeval_metrics(threshold=0.7)  

    for s in samples:
        assert_test(_to_test_case(s), metrics)

Wire that into a CLI or 'pytest' step and every change to your prompt, chunking, or k gets scored before it ships. My suite currently sits at 18/21 (86%) — and I left the 3 reds visible, because two of them (bug #2's recall, bug #3's relevancy) are telling me exactly what to fix next.

The five things I'd tell past me

1. A green suite you don't understand is worse than no suite — it launders bugs as quality.
2. 'retrieval_context' ≠ 'context'. Evidence is not truth. (Bug #1)
3. Faithfulness can't see a retrieval recall miss. Score retrieval and generation as separate stages. (Bug #2)
4. Some red metrics are your eval working, not failing. Don't move thresholds to feel good. (Bug #3)
5. For bespoke rules, 'GEval' is your escape hatch — but pin the judge and validate it against humans.

If you're building RAG in 2026, the pipeline is the easy 20%. The eval — configured so its numbers actually mean something — is the 80% that decides whether you can ship changes without praying.

What's the eval metric that fooled you the longest? I'll trade you my Hallucination-metric story for yours in the comments

Comments

[Luis]

I appreciated how you highlighted the distinction between 'retrieval_context' and 'context' in the DeepEval test case, and how this subtle difference can significantly impact the results of the 'HallucinationMetric'. The example you provided, where the retriever returned 6 chunks but only one was relevant, effectively illustrated this point. It's interesting to note that the 'HallucinationMetric' is lower-is-better and checks the answer against 'context' as factual ground truth, which can lead to unexpected results if not used correctly. Have you considered exploring other metrics, such as 'RecallMetric', to evaluate the retrieval recall and identify potential misses like the one you described in Bug #2?