The gate had been green for eleven weeks. In the same eleven weeks we rolled back two prompt changes, hotfixed a retrieval config on a Saturday, and shipped a summarizer that started dropping the second half of long documents. Nobody was confused about whether the gate was working. Everyone assumed it was, because it was the thing standing between a pull request and production, and it kept saying yes.
The question I couldn't answer, when someone finally asked it in a retro, was how often the gate says yes to something it should have caught. I'd never measured it. I'd tuned it, twice, both times to make it stop paging people. That's a different activity.
An eval gate is a classifier. Its input is a diff, its output is one bit, and like every classifier it has a false negative rate and a false positive rate. Ours had both. Neither had a number attached.
Building a labeled set out of history
You can't measure recall without known positives, and the useful thing about having shipped regressions is that you have a supply of them.
I went back fourteen months and pulled every change that we knew, after the fact, had degraded model behavior in production. Three sources: incident tickets with a root cause pointing at a prompt, model, retrieval or tool change; reverts in git where the revert message named a quality problem; and bug reports that a later commit fixed by changing model-facing code. Deduped, that gave 41 merges. For each one I recorded the commit and one sentence on what got worse, because the sentence turns out to matter more than the commit.
The control set was 60 merges from the same window that touched model-facing code and had no incident, revert or quality bug within thirty days after. That thirty-day window is the weakest part of the whole exercise, and I want to be honest about which way it bends. A regression nobody noticed looks exactly like no regression. Some unknown number of my 60 controls are bad merges we never caught, so my false alarm rate is, if anything, overstated. Recall is worse: an unnoticed regression is disproportionately one the gate let through, so it never made it into the 41 at all, and the 56 percent below is an optimistic reading. Both numbers are soft in the direction of making the gate look better than it is. I decided I could live with that rather than build nothing.
Then I replayed the gate at each of those 101 commits.
# replay.py: run the eval suite as it existed at a commit, against that
# commit's model-facing code, and record the gate's verdict.
import subprocess, sys, json, pathlib
from collections import Counter
def replay_once(sha: str, suite_ref: str = "eval-suite-at-commit") -> dict:
# suite_ref="eval-suite-at-commit" replays the suite as it was THEN.
# suite_ref="HEAD" replays today's suite against the old code, which
# answers a different and also useful question. Run both.
subprocess.run(["git", "worktree", "add", "-f", "/tmp/replay", sha], check=True)
try:
if suite_ref != "eval-suite-at-commit":
subprocess.run(["git", "-C", "/tmp/replay", "checkout", suite_ref, "--", "evals/"], check=True)
out = subprocess.run(
[sys.executable, "-m", "evals.run", "--json", "--baseline", f"{sha}~1"],
cwd="/tmp/replay", capture_output=True, text=True,
)
finally:
# without this, one exception leaks the worktree and every later
# iteration dies on "fatal: '/tmp/replay' already exists"
subprocess.run(["git", "worktree", "remove", "-f", "/tmp/replay"], check=False)
if out.returncode != 0:
# otherwise stdout is empty, json.loads raises, and the loop dies
# 40 commits in without telling you which one broke
raise RuntimeError(f"{sha}: {out.stderr[:400]}")
return json.loads(out.stdout)
def replay(sha: str, runs: int = 5) -> dict:
# judge-graded cases are not deterministic. majority verdict over 5.
rs = [replay_once(sha) for _ in range(runs)]
votes = Counter(r["blocked"] for r in rs)
blocked, n = votes.most_common(1)[0]
return {"sha": sha, "blocked": blocked, "votes": f"{n}/{runs}",
"delta": sum(r["mean_delta"] for r in rs) / runs}
for sha in pathlib.Path("labeled/bad.txt").read_text().split():
print(json.dumps(replay(sha)))
Replaying the suite as it existed at the commit is the honest version and it's the one that gave me the number below. Replaying today's suite against old code is the kinder version, and the gap between the two is a decent measure of how much your eval set has learned since.
Two practical notes, because both cost me a day. Pinned model versions matter: two of the 41 replays came back green in a way that had nothing to do with the gate, because the provider had since retired the model the commit ran against and our fallback silently picked a newer one. Both are in the 18 misses below, re-run against the nearest surviving pinned version, and both are misses I hold loosely. And three commits came back 3 to 2 on the majority vote rather than unanimous. A stricter rule would have demanded all five agree. I didn't use one, which again bends toward the gate.
56 percent
The gate blocked 23 of the 41 known regressions. It fired on 9 of the 60 control merges.
Recall of 56 percent was lower than anyone on the team guessed. I asked four people to write their guess down before I showed the number, which I recommend, because it converts an argument about methodology into an argument about who was closest. The guesses were 85, 90, 80 and 75.
The 15 percent false alarm rate surprised no one, because false alarms are the only part of a gate's behavior that anybody experiences. That asymmetry is the whole problem. A gate that wrongly blocks you interrupts your afternoon and you file it under "the eval thing is flaky again". A gate that wrongly passes you does nothing at all, which feels identical to working.
Where the 18 went
I read every miss. They sorted into three piles, and the sizes changed what we did next.
Eight had no case in the eval set that exercised the behavior that broke. The long-document summarizer is the clearest one: our longest eval input was about 2,800 tokens and the failure started somewhere past 6,000. The gate didn't miss this. It was never asked.
Six had a case, the case got worse, and the aggregate didn't move enough to cross the threshold. Our scores run 0 to 100 and our threshold was a two-point drop in the mean across 180 cases, so one case going from pass to fail moves it by about half a point. Six regressions were smaller than the resolution of the thing measuring them.
Four ran against a stubbed dependency, so the failure couldn't occur inside the harness at all. Our retrieval stub returned a fixed document set. Three of these four were retrieval regressions.
Those three piles want three different fixes, and only the first one is "write more evals", which is the thing everybody reaches for by default.
The threshold was set by whoever was most annoyed
Then I swept the threshold across the same labeled set, which took an afternoon because the replays were already cached.
1-point mean drop: regressions caught (of 41) 29, false alarms (of 60) 21
2-point (what we shipped): regressions caught (of 41) 23, false alarms (of 60) 9
3-point: regressions caught (of 41) 18, false alarms (of 60) 5
4-point: regressions caught (of 41) 14, false alarms (of 60) 3
I remembered both times we moved that threshold. Neither time did anyone say "we are trading recall for quiet". The first move followed a week with four false alarms. The second followed a single loud one during a launch. Both were reasonable in the moment and both were made by whoever was on call, with no visibility into the column on the left, because that column didn't exist.
This is the part I'd put in front of anyone who runs a gate on a continuous score. You have an operating point whether or not you chose it, and if you have never measured recall then every threshold change you have ever made has been a one-sided negotiation.
What we changed
We split the gate in two.
The blocking tier holds cases that are cheap, deterministic, and tied to something that actually broke once. Exact-match and structural assertions, no judge. It blocks on any case flipping from pass to fail, not on an aggregate drop, which kills the entire second pile above. It's small on purpose: 61 cases at the time of writing.
A case earns a place in it by meeting three conditions, and we wrote them down because otherwise the tier grows until it is the old suite again. It has to be traceable to a specific production failure, by ticket number, which is what keeps it from being somebody's hunch. It has to be deterministic, meaning the same input produces a byte-identical assertion result across five runs on the same model pin, which is what excludes the judge. And it has to run in under 400 milliseconds, because the whole tier runs on every pull request and a slow gate gets bypassed. Cases that fail the last two conditions aren't thrown away, they go to the reporting tier.
The reporting tier holds everything else, including all the judge-graded scoring. It posts a comment on the pull request and blocks nothing.
Moving the judge out of the blocking path is not free, and this is where I have to show the whole ledger rather than the good half. It removed five of the nine false alarms, because judge variance was most of our false alarm budget. It also cost us eight catches: eight of the original 23 were judge-graded cases that moved the aggregate past the threshold, and those now only leave a comment. Against that, per-case blocking recovers the six in pile two, all six of which were assertion cases and so qualify for the tier, unstubbing retrieval recovers two of the four in pile three, and the eight incident-derived cases we added cover pile one. Twenty-three, minus eight, plus six, plus two, plus eight, is 31.
We also stopped stubbing retrieval in the blocking tier and pinned a frozen corpus snapshot instead. Slower, and it caught two of the four stub misses on replay.
Every incident that traces back to model-facing code now owes the blocking tier a case, written from the incident's own reproduction, before the incident closes. That rule is the only reason pile one shrinks over time, and it works because writing the case is easiest on the day you understand the failure.
So: 31 of 41 caught, 4 of 60 false alarms. I don't fully believe the 31. Eight of those cases exist because these specific regressions are in my labeled set, so the back-test is partly grading a suite built from its own answer key. The honest number arrives in a year, from incidents that had no chance to influence the set. What I do believe is the 4, because nothing on the false alarm side was tuned against this data.
Is 76 percent good? I have no idea what good looks like here, and I haven't found anyone publishing theirs. What I know is that 56 wasn't the number anyone was operating on, and that the fourteen months of history I replayed are all months in which nobody had measured the thing we were trusting. Calling it a control was the mistake.
What I'd check first
Pull your last ten quality incidents and ask, for each one, whether the eval set contains a case that would have gone from pass to fail. Not "a case about that area". A case that flips.
Look at when your blocking threshold last changed, and what happened in the week before. If the answer is a false alarm, you moved your operating point along a curve you have never plotted.
Check what your eval harness stubs out. Anything you replaced with a fixture is a class of regression your gate is structurally incapable of seeing, and retrieval is usually the expensive one.
Top comments (0)