TL;DR: I broke out per-case results for the incident-harvested part of our eval suite, 611 of our 1,400 cases, and asked which cases have ever discriminated between two shipped versions. Of the 528 with enough history to judge, 356 had passed every version and 59 had failed every version. 113 cases were carrying all of the signal. That is not a bug in harvesting, it is what harvesting does, and it is measurable.
Yesterday I published that our eval gate caught 23 of 41 known regressions when we back-tested it (post). This is the other half of that measurement. That one asked what the gate misses. This one asks which cases were ever going to fire at all.
The breakdown I did not have
Our suite is about 1,400 cases. 611 of them are the behavioural set, the ones harvested from production incidents rather than written by hand, which I wrote about starting in June. That set has grown fast, because it grows every time something breaks.
We stored a pass rate per run. We did not store a result per case per version, so I could not answer the question a colleague asked me in a review: when the number moves two points, which cases moved?
Adding it was an afternoon. Then a second pass, because the first one was wrong: I bucketed every case including ones added last month, and a case that has only ever seen three versions has passed all of them for uninteresting reasons. Requiring at least five shipped versions of history drops 83 cases as too new to judge.
Of the 528 that remain, across 23 versions shipped since April:
- 356 passed every version. Never once failed.
- 59 failed every version. Known-failing, mostly things we decided not to fix.
- 113 actually varied.
So 21 percent of the judged set was carrying the signal. A case that every version passes has no discriminating power, because power here just means the ability to come out differently on two runs.
Why a harvested set rots this way
This is a consequence of the harvesting idea, and the mechanism is dull.
You harvest a case from an incident. At the moment you write it, the case fails, because the bug is live. You fix the bug. From the next version on, the case passes. Forever.
The case did its job once, on the day it was written. After that it is a regression guard, and a regression guard is worth its runtime only if the regression can plausibly come back. Some can. Most of ours could not, because the fix was a schema change or a prompt restructure that nothing was going to walk back.
So the set accumulates cases at the rate you have incidents, and it accumulates cases that can no longer discriminate at very nearly the same rate, one fix behind.
The 356: six minutes and no headroom
Runtime. Those 356 cases are roughly 6 minutes of the batched suite's 22. Not fatal on its own, but it is 6 minutes spent re-confirming things that have not changed since April.
A reported number with almost nowhere to go. This is the one that bothers me, and it needs a denominator stated out loud, because I have quoted three different ones in past posts.
One 94 was 481 of 512 cases, on the run that shipped a PII leak. A second 94 was our ticket-routing eval over 600 rows, and that one turned out to be manufactured, because the few-shot index was built from the eval set. The 96 to 97 was an intent-classification eval sitting above a 90 percent gate, one eval, not a suite. None of the three is this number, and the suite has roughly tripled since. The number in this section is the behavioural set's own pass rate, over the 528 cases with enough version history to judge. Different denominators, different scales, and mixing them is how you end up comparing a 96 to an 89 and concluding something regressed.
On that 528: with 356 guaranteed passes the floor was 67.4 percent before any model did anything, and with 59 guaranteed failures the ceiling was 88.8. The entire achievable range was 21.4 points, and we spent the quarter inside a 4 point window near the top of it, between 84 and 88.
That 21.4 is not a coincidence: the achievable range in points is the discriminating share, 113 of 528 either way. Every case you add that everything passes shortens it by one.
That holds whatever your numbers look like. I have argued before that aggregates hide individual regressions, and this is the prior question, which is how much room the aggregate had to begin with.
What I got wrong first
My first move was to delete the 356. My colleague caught it in review and was right.
Some of those cases guard behaviour that is currently correct because the guard exists. No version failing them since April is evidence the guard works, not evidence the case is useless. Deleting a PII redaction case because it has always passed is how you find out in November that it stopped passing in September.
What we did instead was use discrimination as a tiering criterion, for the tiers we already had rather than a new structure:
- 113 discriminating plus 59 known-failing stay in the reporting tier from yesterday's back-test post, the one that scores but does not block.
- 356 always-pass move to the nightly tier, minus those of the 61 already in the blocking tier, which stay where they are: that tier blocks on a case flipping pass to fail and does not care whether the case has ever flipped before.
- Re-score discrimination monthly, because a dead case revives when the code around it changes. Three did in the first month.
The batched merge suite lost about 6 minutes, on the rough assumption that per-case cost is uniform. It is not: the 356 skew deterministic and the judge-scored cases dominate wall clock, so treat 6 as an upper bound rather than a measurement. Nothing was deleted.
What happened to the reported number
We now report the pass rate over the 113 discriminating cases, so the constant block is out of the denominator.
The number did not get lower. It got wider. On the old denominator our quarter sat between 84 and 88 percent; recomputed over the 113, the same quarter swings between roughly 77 and 96. The fixed point is around 86, so it moved down about as often as up.
That is correct, and it took some explaining. The wider number moves when the model moves, which the old one could not do.
The script
Feed it per-case results per version. If you do not store those, that is the actual first task.
import collections
# (version, case_id, passed): substitute your own rows
results = [
("v1", "case_a", True), ("v2", "case_a", True), ("v3", "case_a", True),
("v4", "case_a", True), ("v5", "case_a", True),
("v1", "case_b", True), ("v2", "case_b", False), ("v3", "case_b", True),
("v4", "case_b", True), ("v5", "case_b", False),
("v1", "case_c", False), ("v2", "case_c", False), ("v3", "case_c", False),
("v4", "case_c", False), ("v5", "case_c", False),
# too new to judge: only 3 versions of history, must land in neither bucket
("v3", "case_d", True), ("v4", "case_d", True), ("v5", "case_d", False),
]
MIN_HISTORY = 5 # versions a case must have seen before we judge it
by_case = collections.defaultdict(list)
for _, case_id, passed in results:
by_case[case_id].append(passed)
always_pass, always_fail, discriminating, too_new = [], [], [], []
for case_id, outcomes in by_case.items():
if len(outcomes) < MIN_HISTORY:
too_new.append(case_id)
elif all(outcomes):
always_pass.append(case_id)
elif not any(outcomes):
always_fail.append(case_id)
else:
discriminating.append(case_id)
judged = len(always_pass) + len(always_fail) + len(discriminating)
if not judged:
raise SystemExit("no case has enough history yet; lower MIN_HISTORY or wait")
for name, bucket in [("always pass", always_pass), ("always fail", always_fail),
("discriminating", discriminating)]:
print(f"{name:15s} {len(bucket):4d} ({len(bucket)/judged:.0%} of judged)")
print(f"{'too new':15s} {len(too_new):4d} (excluded)")
print(f"achievable range: {len(discriminating)/judged:.1%} of the scale")
MIN_HISTORY is the line that matters, which is why case_d is in the sample: three versions of history, so it lands in too_new and in none of the three buckets. Without that guard every case added last month reads as dead, and the number that comes back is flattering rather than true. That was my first run.
One thing I have not resolved: this treats all 23 versions as equally informative and they are not. Several were small prompt edits that were never going to move most cases, so a case can look non-discriminating because nothing asked it a hard question. Weighting by how much each version actually changed is the right fix and I do not have a defensible way to do it yet.
What I'd check first
- Do you store a result per case per version, or only an aggregate per run? If it is the second, none of this is answerable, and it is an afternoon to fix.
- What fraction of your set has ever failed, counting only cases with real version history? That fraction is your metric's entire range.
- What is your floor, the number you would report if every discriminating case failed? If it is high, your headline has less room than it looks.
Top comments (0)