RAG evals that fail CI: a 30-question golden set caught 3 of 4 retrieval regressions

Originally published on RecallRun, where the full scripts and raw output are attached.

How do you stop a retrieval change from silently making your RAG worse? Put a small golden set in CI and fail the build when metrics drop below a committed baseline.

Result: four files. On 30 questions it flagged 3 of 4 accidental regressions I injected. The one it passed was a real quality change that stayed inside the tolerance I set (hit@5 fell 3.3 points). One regression (chunks of 200 characters) raised hit@1 and MRR while cutting hit@5 by 13.3 points, so gating on a single metric would have missed it.

Setup

• Retriever under test: heading-aware chunks (800 characters) with the heading path prepended, ranked by BM25. Same chunker as in my chunking benchmark, and the 30 questions are the paraphrase set from the hybrid search post. BM25 only, so CI needs no model download and runs in about 0.2 s.
• Corpus: FastAPI English docs (151 Markdown files). Python 3.12, pytest 9.1, stdlib only.
• Golden set: golden.jsonl, one line per question: {"question": ..., "expected_doc": ...}. A hit means the expected doc appears in the top-k documents.
• Metrics: hit@1, hit@5, MRR (cut off at rank 10).
• Gate: pytest fails if any metric is more than 0.05 below baseline.json. With 30 questions, one question is 0.033, so one flipped answer passes and two fail.
• Regressions injected: each is a one-field config change: smaller chunks, larger chunks, no heading prefix, and a tokenizer bug (lowercasing removed).

I did not test a citation-rate metric here. The harness only scores retrieval.

Results

Differences are in percentage points versus baseline, computed from the unrounded values.

Change	hit@1	hit@5	MRR	Gate
Baseline	0.367	0.933	0.583	pass
chunk size 800 to 200	0.500 (+13.3)	0.800 (-13.3)	0.628 (+4.5)	fail (hit@5)
chunk size 800 to 3000	0.367 (0.0)	0.900 (-3.3)	0.601 (+1.8)	pass
heading prefix off	0.300 (-6.7)	0.867 (-6.7)	0.513 (-7.0)	fail (all 3)
lowercase off	0.400 (+3.3)	0.733 (-20.0)	0.553 (-3.0)	fail (hit@5)

What the numbers mean

The 200-character case is why you gate on several metrics. Small chunks make the best chunk more specific, so the right doc more often lands at rank 1 (+13.3 points). But 6 questions dropped out of the top 5 entirely. If your gate were "MRR must not fall", this change would pass and ship. My reading of the cause: short chunks split the answer away from the words the question uses, so some docs have no chunk that scores at all. I did not test that explanation directly.

The tokenizer bug moved MRR only 3.0 points yet lost 20.0 points of hit@5, which is 6 more questions failing. An average hid it; the per-metric threshold did not.

Removing the heading prefix hit every metric by roughly 7 points, so it failed all three tests at once.

3000-character chunks passed the gate with a 3.3-point hit@5 drop. That is one question. It sits inside the tolerance on purpose, and it also shows the cost: a 30-question set cannot separate a real 3-point drop from noise.

Caveats

• 30 questions is a smoke test. Every question is worth 3.3 points, so differences under about 5 points are not reliable. Grow the set from real user queries and from every bug report.
• The questions and corpus are the ones I used to tune earlier posts, so the baseline is optimistic.
• The expected answer is a document, not a passage. Coarse, but stable while you re-chunk.
• BM25 only. A dense or hybrid retriever adds nondeterminism (model versions, ANN settings), so pin those and expect to widen the tolerance.
• The injected regressions are mine; real ones are less tidy.

Code

The retriever config is a frozen dataclass, so a regression is a one-field diff. The harness and the gate:

# harness.py
import json, os
HERE = os.path.dirname(__file__)

def load_golden(path=os.path.join(HERE, "golden.jsonl")):
    return [json.loads(l) for l in open(path) if l.strip()]

def evaluate(retriever, golden):
    rows, h1, h5, rr = [], 0, 0, 0.0
    for g in golden:
        ranked = retriever.search(g["question"], k=10)
        rank = ranked.index(g["expected_doc"]) + 1 if g["expected_doc"] in ranked else None
        h1 += rank == 1; h5 += rank is not None and rank <= 5; rr += 1 / rank if rank else 0
        rows.append({"question": g["question"], "expected": g["expected_doc"], "rank": rank})
    n = len(golden)
    return {"hit@1": h1 / n, "hit@5": h5 / n, "mrr": rr / n, "n": n}, rows

# test_golden.py
"""Fail the build if any retrieval metric drops below the committed baseline.
Update baseline.json deliberately (in the same PR, reviewed) when you improve the pipeline."""
import json, os, pytest
from rag import Retriever, Config
from harness import load_golden, evaluate

HERE = os.path.dirname(__file__)
TOLERANCE = 0.05   # 30 questions = 0.033 each: one flipped answer passes, two fail
CONFIG = Config(**json.loads(os.environ.get("RAG_CONFIG_OVERRIDE", "{}")))

@pytest.fixture(scope="module")
def result():
    return evaluate(Retriever(CONFIG), load_golden())

@pytest.mark.parametrize("metric", ["hit@1", "hit@5", "mrr"])
def test_metric_not_regressed(result, metric):
    metrics, rows = result
    base = json.load(open(os.path.join(HERE, "baseline.json")))[metric]
    misses = [r["question"] for r in rows if r["rank"] is None or r["rank"] > 5]
    assert metrics[metric] >= base - TOLERANCE, (
        f"{metric} {metrics[metric]:.3f} < baseline {base:.3f} - {TOLERANCE}; "
        f"{len(misses)} questions missed top-5, e.g. {misses[:3]}")

baseline.json is committed and holds {"hit@1": 0.3667, "hit@5": 0.9333, "mrr": 0.5828}. The workflow:

# .github/workflows/rag-eval.yml
name: rag-eval
on:
  pull_request:
    paths: ["rag/**", "golden.jsonl", "baseline.json", "prompts/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install pytest
      - run: bash fetch_corpus.sh
      - run: pytest -q test_golden.py

Rule of thumb

1. Commit a golden set and a baseline; fail the build on any single metric dropping, not on an average.
2. Set tolerance from your set size: at least two questions' worth, so one flip is not a red build.
3. Print the missed questions in the failure message so the fix starts from a list.
4. Raise the baseline only in the same reviewed PR that improves the pipeline.
5. Trigger the job on changes to chunking, embeddings, prompts and the golden set itself.

The full rag.py, golden.jsonl, the demo runner and raw results.txt are attached to the original post. If you gate RAG changes differently, I'd like to hear what metric you trust.