You set a threshold on your retrieval quality. recall@k >= 0.8. The suite runs at 0.94, comfortably above, and stays green.
Then someone changes the chunking strategy, and recall drops to 0.85.
Your gate says nothing, because 0.85 is still above 0.8. You have lost nine points of recall, which is one question in eleven that can no longer be answered, and the check you installed to catch that reports success.
Two different questions
A threshold asks whether the system is good enough. You set it once, from what you believed acceptable at the time, and it rarely changes.
A regression check asks whether this run is worse than the last one. It has no opinion about what good is.
Most eval tooling gives you the first. The second catches the change nobody meant to make, because real systems do not fall off a cliff. They lose two points here and three there, each individually invisible, and a suite with headroom absorbs all of it until someone notices the product got worse and nobody can say when.
The headroom is the problem. The more comfortably you pass your threshold, the more quality you can lose without being told.
What the comparison needs
Three things, and the third is the one people skip.
A stored baseline, somewhere to record what "last time" was. A local SQLite file is enough.
ragbench run --dataset cases.jsonl --exec "python my_rag.py" --label main
A comparison on the branch.
ragbench gate --baseline main --threshold recall@k=0.8
recall@k 0.5000 -0.5000
x recall@k fell 0.5000, from 1.0000 to 0.5000
Exit code 1. The pull request is red and you know which metric moved before anyone reads the diff.
A tolerance band, which decides whether the gate survives contact with reality. Retrieval scores move slightly for reasons that are not your change: an embedding service updates, a tie breaks differently, a timestamp shifts an ordering. A gate that fires on that is wrong, and a gate that fires on noise gets disabled inside a week.
So drops inside the tolerance are reported as warnings. The default is one percentage point:
export const DEFAULT_TOLERANCE = 0.01;
Set it from your own observed run-to-run variance. Run the same commit five times, see how much it moves, set the band above that.
The failure mode nobody plans for
What should happen when a threshold names a metric the run never produced?
The comfortable answer is to skip it. The metric is not there, so there is nothing to compare, so pass.
That looks like success and is not. Gate on recall@k, lose the relevance labels from your dataset, and the metric stops being produced, the threshold stops being checked, and the gate reports green forever while checking nothing.
So a threshold on an absent metric fails:
recall@k has a threshold of 0.8 but was not produced by this run
Treating it as passing hides that the check never ran, and the value of a gate is that green means something.
The same reasoning covers the first run. There is no baseline yet, so the gate passes, and it says so out loud instead of reporting a clean comparison that never happened.
Things worth refusing outright
Three dataset rules that are all the same rule.
A case with nothing to check against is rejected. No expected answer, no relevant documents, no required phrases. It passes every metric vacuously and lifts your average.
A duplicate id is rejected. Predictions join to cases by id, so a duplicate makes half your suite vanish without a word.
A case with no prediction scores as empty instead of skipped. Skipping lets a system that answered nothing score identically to one that answered everything correctly.
Each is a way for a suite to look healthier than it is, which is the only real failure mode of an evaluation harness.
npx ragbench gate --baseline main --threshold recall@k=0.8
Deterministic metrics, no LLM judge, no API key, local SQLite history. ragbench.
Top comments (0)