Your baseline scored 0.000? Before you publish the win, here is the checklist I now run, because a zero from a baseline is almost never a result. It is usually your harness.
This week I watched a builder on r/Rag do something rare: he benchmarked his own memory library against plain RAG expecting a win, got three null results, and published the confounds instead of the victory. Two of his bugs are so common, and so quiet, that I think every RAG or agent-memory benchmark should gate against them by construction. This post is the checklist that came out of that thread.
Bug 1: your arms have different context budgets
His first run compared a memory arm retrieving k=20 sentence-level hits against a BM25 arm returning whole sessions. Same "top-k" on paper. In characters, one arm got 1.3k of context and the other got 11.9k. BM25 looked dramatically better.
Once he matched the budget, accuracy went 0.28 to 0.59 for the memory arms and the ranking flipped. The original result was a budget difference wearing a granularity costume.
The fix is embarrassingly simple: print the context size, in characters or tokens, next to every accuracy number, per arm. If a comparison does not state budget parity, the number does not mean anything. Chunked-vs-whole-document comparisons are especially prone to this because "top-k" hides a 10x budget difference in plain sight.
Bug 2: the baseline that scored 0.000 with clean logs
He ran a competing memory tool as a baseline. It scored 0.000. He re-ran it with a stronger extraction model. 0.000 again, logs clean. Very tempting to publish "competitor discards memories under load."
It was his bug, twice. His harness truncated each session to 6,000 characters before ingestion, cutting off the injected evidence. And he was passing limit= to an API whose parameter is top_k=, so his setting was silently ignored.
That second one deserves its own paragraph, because it is everywhere in this ecosystem: APIs that accept unknown kwargs without raising. They make a broken configuration look like a clean loss. In a benchmark, treat any client that swallows unknown parameters as hostile. Do not trust that a parameter did something. Assert it: if you move top_k from 5 to 20, the retrieved count has to move. If it does not, your harness is configuring nothing.
The three gates that make results trustworthy
Out of that exchange came a harness structure I now consider the minimum for memory and RAG benchmarks. Three separate liveness gates, each with its own abort message, all pre-registered before the run.
1. A positive control per arm. Every arm gets a probe it cannot fail, built from the corpus's own ground truth. If any arm scores below a threshold you wrote down before the run, the whole benchmark aborts as "dead harness," not "clean loss." This is the gate that catches truncation bugs, auth failures, and empty stores. A zero is never evidence until this gate passes.
2. An evidence-in-context ceiling. For each probe, check whether the gold evidence is actually present in the retrieved context. Aggregated, this is the recall ceiling you report next to accuracy. In his broken run it was 3.5 percent. You cannot out-rank evidence that was never retrieved, and no reranker or prompt change will save a run whose ceiling is on the floor. Cheap to compute, needs no LLM calls.
3. A parameter-efficacy assertion. For each knob the benchmark claims to vary, assert that observable behavior changes when the knob moves. This is the gate that catches the swallowed-kwargs class of bug, and it is the one nobody runs.
Why three gates and not one
The temptation is to fold these into a single sanity check. Do not. The whole value of a gate is that its failure names the broken layer. Evidence-in-context failing means retrieval never delivered. The positive control failing means the store or the answerer is dead. The efficacy assertion failing means your configuration is fiction. One merged gate tells you "something is wrong" and leaves the first hour of debugging still ahead of you.
There is a general lesson under all of this. A benchmark harness is software that lies by default, because every bug it has produces a plausible-looking number instead of a crash. The discipline that fixes it is the same one that fixes agent evaluation generally: never trust a claim the system makes about itself unless something the system cannot touch verifies it. A test that has never been proven to fail will happily pass forever.
Credit where it is due: the null-result writeup that prompted this is worth reading in full, including the part where he catches himself half-writing a false finding about a competitor before his own positive control saved him. Publishing that takes more spine than publishing a win.
I build a local-first assistant for the Mac, which is why I spend my days in retrieval evals. No product pitch here; the checklist stands on its own.
Top comments (3)
The silent kwarg swallow is the one that gets teams repeatedly — the parameter did nothing, the logs said nothing, and the result looks like a clean loss from the library under test. The fix you describe (assert retrieved count moves when top_k moves) is exactly right, and it generalizes: any config change that doesn't produce an observable state change in the output should abort the run, not proceed quietly. I'd add one more pre-run gate that compounds with yours: a judge stability probe using a held-out example with a known answer, run before and after any model swap, because LLM-as-judge drift between versions has burned more benchmark comparisons than any retrieval bug I've seen. Once your harness has the positive control, the parameter assertion, and the judge stability check, the result is evidence — without all three, it's a story you're telling yourself.
the judge probe is the one i'd have missed.
one wrinkle: a held-out example with a known answer catches a judge that changed its verdict, but the sneakier drift is in strictness. it still nails the easy probe while its threshold shifts on borderline cases, and borderline is where the score movement lives. so the probe passes and the comparison is still corrupted.
the version i'd trust is a small calibration set spanning the band, a few clear passes, clear fails, and genuinely ambiguous ones, then comparing the judge's score distribution across the swap rather than its accuracy on the easy one.
The strictness-drift case is the one I keep underestimating — a verdict on a clear example can stay constant while the threshold on ambiguous ones shifts, and that shift is exactly what makes cross-model comparisons unreliable without you noticing. Comparing score distribution across the calibration band rather than just accuracy on the easy probe is the right move; I've been using pass/fail on the held-out example as a proxy, and your version catches the failure mode mine doesn't. I'm going to rebuild the probe set to include genuinely borderline examples and compare the distributional spread, not just the pass-fail verdict.