Your RAG Evals Are Lying to You. The Bug Is in the Retrieval, Not the LLM.

I rebuilt my RAG evaluation pipeline three times before I admitted the LLM was never the problem.

The retrieval was. And the eval was measuring retrieval noise, not model quality. Every "the model is getting worse" conclusion I'd drawn over six weeks was actually the same retrieval returning different chunks on different runs of the same query.

If you've ever shipped a RAG change, watched your eval dashboard wobble, and concluded "I need to fine-tune" or "I need a bigger model" — this is the post I wish I'd read first.

The Symptom: Flaky Evals

Here's what was happening. I had a "golden" eval set: 200 question/answer pairs, each with a known-correct chunk ID from a known-correct corpus. I ran the same eval three times in a row, same model, same temperature=0, same corpus, same prompt.

The numbers were different every time. Not by a little — by enough to flip a regression test from green to red on the same commit.

Run 1:  recall@5 = 0.78  faithfulness = 0.84
Run 2:  recall@5 = 0.71  faithfulness = 0.81
Run 3:  recall@5 = 0.74  faithfulness = 0.79

The first thing I did was suspect the LLM. The second thing I did was lower temperature to 0. The third thing I did was pin a specific model version. None of it moved the variance meaningfully.

The fourth thing I did was log the retrieved chunk IDs. That's when it became obvious.

Question: "How do I configure SSL offloading on the edge gateway?"
Run 1:  chunk_ids = [a17, b04, c91, d22, e55]
Run 2:  chunk_ids = [a17, b04, c92, d22, e58]   ← c91 → c92, e55 → e58
Run 3:  chunk_ids = [a17, b05, c91, d22, e55]   ← b04 → b05

Same query. Same corpus. Different chunks. The LLM was doing the best it could with whatever it got handed, and the evaluation was measuring the hand, not the head.

The Three Sources of Retrieval Non-Determinism

Once I started looking, I found three places where the same query could return different chunks on the same corpus.

1. Approximate nearest neighbor (ANN) tie-breaking.

Most vector indexes — HNSW, IVF, ScaNN — return slightly different orderings when two vectors are nearly equidistant from the query. The scores look identical to four decimal places, but the order flips. If your top-k=5 happens to land on a tie boundary, the swap propagates.

2. Hybrid search with parallel retrievers.

If you combine BM25 and dense retrieval, you usually have two parallel calls and a fusion step. Depending on which finishes first, batching order, and any concurrent index modifications, the fusion weights can shift fractionally. Enough to swap the 4th and 5th results.

3. Chunking drift after re-ingestion.

This is the one nobody suspects. If your ingestion pipeline chunked the corpus with any randomness — overlap windows, sentence-boundary snapping, or a parallel chunker — then the chunk IDs in your eval set may not match the chunk IDs in your live index after a re-ingest. The eval is grading against a chunk that no longer exists.

The Fix: Deterministic Retrieval for Evaluation

The fix isn't to make retrieval deterministic in production. Production retrieval is allowed to be slightly stochastic — the model can compensate, and a small amount of variation is fine. The fix is to decouple eval retrieval from production retrieval.

from dataclasses import dataclass
from typing import Optional
import hashlib

@dataclass
class EvalRetrievalConfig:
    """Pinned retrieval settings for reproducible evals."""
    seed: int = 42
    ef_search: int = 200           # HNSW: higher = more accurate, less random
    nprobe: int = 32               # IVF: higher = more accurate
    oversample_factor: int = 4     # Retrieve 4x, then deterministically rerank
    rerank_temperature: float = 0.0
    chunk_version: Optional[str] = None  # Hash of chunking config + corpus

    def config_hash(self) -> str:
        """Hash the retrieval config so mismatches are detectable."""
        key = f"{self.seed}|{self.ef_search}|{self.nprobe}|{self.oversample_factor}|{self.rerank_temperature}|{self.chunk_version}"
        return hashlib.sha256(key.encode()).hexdigest()[:12]

The trick is oversample_factor. You retrieve 4× as many candidates as you need, then deterministically rerank them with the LLM. The top-k of 20 candidates is much more stable than the top-k of 5 raw ANN results, because the LLM reranker is itself mostly deterministic at temperature=0.

def deterministic_retrieve(
    query: str,
    index,
    cfg: EvalRetrievalConfig,
) -> list[RetrievalResult]:
    # Pin ANN search parameters — these are the main source of variance
    index.set_ef_search(cfg.ef_search)
    index.set_seed(cfg.seed)

    # Retrieve 4x more candidates than we need
    raw = index.search(query, k=5 * cfg.oversample_factor)

    # Deterministic rerank via cross-encoder or LLM at temp=0
    reranked = reranker.rerank(
        query=query,
        candidates=raw,
        temperature=cfg.rerank_temperature,
    )

    return reranked[:5]

Pin the Chunk Version

This is the part I missed twice. Your eval set has chunk IDs in it. If you change your chunker — even by adjusting chunk_size from 512 to 480 — every chunk ID becomes invalid.

The fix: hash the chunking config and the corpus, and refuse to run the eval if the hash doesn't match.

def corpus_chunk_hash(chunker_config: dict, corpus_path: str) -> str:
    """Hash the chunker config + a content hash of the corpus."""
    config_str = json.dumps(chunker_config, sort_keys=True)
    content_hash = hashlib.sha256(open(corpus_path, 'rb').read()).hexdigest()
    return hashlib.sha256(f"{config_str}|{content_hash}".encode()).hexdigest()[:16]

# At eval time:
expected = eval_set["chunk_version"]
actual = corpus_chunk_hash(current_chunker_config, corpus_path)
if expected != actual:
    raise EvalStaleError(
        f"Chunk version mismatch: eval was built against {expected}, "
        f"current is {actual}. Re-annotate the eval set or revert the chunker."
    )

This sounds pedantic until you spend a week chasing "eval regression" that turned out to be a chunker config that someone bumped in a PR. Now the eval fails loudly with a clear message instead of silently disagreeing with itself.

What the Numbers Look Like After

Once I pinned ef_search, added oversample rerank, and gated on chunk version, the same eval set returned:

Run 1:  recall@5 = 0.78  faithfulness = 0.84
Run 2:  recall@5 = 0.78  faithfulness = 0.84
Run 3:  recall@5 = 0.78  faithfulness = 0.84

Identical to the last decimal. Now when the eval moves, it's because the model moved, the corpus moved, or the chunker moved — not because the retrieval was flipping coins.

The real win came after this: I finally started trusting the eval. Three changes I'd dismissed as "noise" — a reranker swap, a chunker size tweak, a hybrid weight adjustment — turned out to be real improvements. The eval had been right about them; the variance had just been too high to see the signal.

The Question to Ask Before "My Model Is Worse"

When your RAG eval moves and you're tempted to blame the model, ask instead:

1. Did I run the same query twice and check if the chunk IDs match?
2. Is my ef_search (or equivalent) the same on every run?
3. Is the chunker config unchanged since the eval was built?
4. Am I oversampling before reranking, or is the LLM seeing raw ANN top-k?

If any of those is "I don't know" or "no," the eval is not measuring what you think it's measuring. The model is innocent until the retrieval is proven guilty.

The eval set is the source of truth, and the source of truth has to be reproducible. If it isn't, every "improvement" you ship is a coin flip — and the ones that look like improvements in the dashboard might be the noise your eval was hiding all along.

What I Learned

Retrieval is the layer that gets blamed last and fails first.

I burned six weeks thinking I had a model problem because the eval was wobbly. Once I made the eval reproducible — pinned ANN search params, oversampled before rerank, and gated on chunk version — the wobble disappeared and the real improvements surfaced. Three of them I had rolled back as "didn't help" turned out to be solid 5-10% wins.

If you're building RAG in production and your evals are flaky, the bug is almost never the LLM. It's the retrieval. And the fix is a hundred lines of config, not a thousand lines of fine-tuning.

---

If you've hit a different source of retrieval non-determinism, I'd genuinely like to know — especially anything around hybrid fusion or graph-augmented retrieval. The "set the seed" advice works for plain ANN. The other paths are less settled.