TL;DR. Our overall eval pass rate read 0.88 through a model change and looked stable. Sliced by request language, German had fallen to 0.60 while English held near 0.90. The aggregate hid that because German was a rounding error inside the golden set even though it had grown into almost a quarter of real traffic. A bigger golden set does not fix this. Slicing every run by the production distribution, and refreshing the set from real traffic, does.
The dashboard stayed green while users complained
We shipped a prompt change and a model bump on the same afternoon. The eval ran against the golden set. Overall pass rate moved from 0.90 to 0.88. A two-point move sits at our run-to-run noise floor, so we shipped and moved on.
Four days later support forwarded a cluster of complaints, all of them German. Truncated sentences. Wrong register, formal where it should have been plain. English words leaking into German answers. The eval had flagged none of it. The number was still 0.88, green as ever.
Here is the shape of the set that produced that green. The golden set held 400 cases. We built it eighteen months earlier when the product was English-only, so roughly 370 of those cases were English and about 30 were German, added later as an afterthought. Meanwhile German requests in production had climbed from a rounding error to nearly a quarter of traffic after a market launch that quarter. So the eval was 7% German while production was closer to 22% German. A hard regression on German could move the aggregate by about two points and still be a live fire for a large and growing group of real users.
The golden set encoded last year's traffic, not this quarter's, and the single number it produced averaged over a distribution we no longer had.
Why does an average hide a slice?
An aggregate pass rate is a weighted average, and the weights are whatever mix of cases you happened to freeze into the set. If the set is 90% English and English holds at 0.90, the aggregate sits near 0.90 no matter what the other languages do. German at 0.60 carrying a 7% weight pulls the overall down by roughly 0.02. You will read 0.02 as noise. On most runs you would be right, and that is exactly what makes it dangerous.
Two things had to be true at the same time for this to bite, and both were true for us.
The slice regressed. The change helped English and hurt German at once. That is more common than people expect. A prompt edited and spot-checked against English examples can shift tokenization, instruction-following, and register in another language that nobody re-read before the merge. One model swap can lift your largest slice and quietly drop a smaller one on the same commit.
The slice grew. German had gone from a sliver of real traffic to almost a quarter of it, but the eval set never tracked the change. The group that mattered most in production was represented by the fewest cases in the test. The faster a slice grows in the wild, the more badly a stale set under-weights it.
Put those two facts together and the aggregate becomes an average over the wrong distribution. It gives a confident answer to a question we had stopped asking a year earlier. The same trap sits behind any slice key, not just language: a new input-length bucket, a big tenant you just onboarded, an intent that spiked after a UI change. Whichever slice grew fastest since you froze the set is the one the aggregate is now lying to you about.
The check: passrate_by_slice
The measurement fix is small and boring, which is the point. Tag every eval case with the slice keys you care about (language, input-length bucket, tenant, intent), compute pass rate per slice, take the delta against the overall on every run, and sort by that delta so the worst slice lands at the top of the output where you cannot scroll past it.
Here is the whole thing. Standard library, no dependencies, runs on plain Python 3.
import collections
def passrate_by_slice(rows, slice_key, pass_key="passed"):
total, passed = collections.Counter(), collections.Counter()
for r in rows:
s = r.get(slice_key, "unknown")
total[s] += 1
passed[s] += 1 if r.get(pass_key) else 0
overall = sum(passed.values()) / max(1, sum(total.values()))
rows_out = []
for s in total:
pr = passed[s] / total[s]
rows_out.append((s, total[s], round(pr, 3), round(pr - overall, 3)))
return round(overall, 3), sorted(rows_out, key=lambda x: x[3])
# one eval run, sliced by request language. the model regressed on 'de' only.
run = ([{"lang": "en", "passed": True}] * 90 + [{"lang": "en", "passed": False}] * 10 +
[{"lang": "de", "passed": True}] * 60 + [{"lang": "de", "passed": False}] * 40)
overall, table = passrate_by_slice(run, "lang")
print("overall pass rate:", overall)
for s, n, pr, delta in table:
print(f" {s:6} n={n:4} pass={pr:.3f} delta_vs_overall={delta:+.3f}")
Run it and you get:
overall pass rate: 0.75
de n= 100 pass=0.600 delta_vs_overall=-0.150
en n= 100 pass=0.900 delta_vs_overall=+0.150
The overall 0.75 is the number that would have shipped and passed the gate. The de row is the one that matters: pass 0.600, a delta of minus 0.150 against the aggregate, sitting first because the sort pushes the worst slice to the top. English reads fine at 0.900. Same eval, same run, two very different stories, and only one of them was visible before the slice existed.
This toy is exaggerated on purpose. A 0.15 slice gap is loud, and real ones rarely are. In production the per-slice deltas are small and they jitter a little between runs from sampling alone, so a single run in isolation will not tell you much. The signal is a slice delta that drifts in one direction across several runs while the aggregate holds flat. Track each slice run over run and alert on the movement, not on the absolute gap in any one snapshot. A slice that slid from minus 0.01 to minus 0.06 over three runs is worth a look even though 0.06 by itself looks like nothing.
Refresh the set from traffic, not from memory
Slicing tells you where a regression is hiding. It does not close the second gap, which is that a frozen set drifts away from production every week it sits still. The set is fixed at the moment you built it. The traffic it is supposed to represent keeps shifting. The distance between the two only grows, and every point of drift is another slice the aggregate is quietly mis-weighting.
So refresh on a cadence. Every sprint, sample recent production traffic stratified by the same slice keys, label it, and fold a fresh batch into the set. Keep a frozen core of regression cases you never want to break again, the specific failures you have already paid for once. Add current cases that reflect the mix you actually serve today. Retire cases for intents you have dropped. The set should track the live distribution, not embalm an old one.
Reweighting buys you most of the protection before you label a single new case. Score the aggregate against the current production mix instead of the historical set mix. If German is 22% of traffic this month, weight German at 22% of the number. A static set that is 7% German is quietly asserting that German is 7% of your risk, and it is wrong by a factor of three. The reweight is one dictionary of production shares, refreshed whenever the mix moves.
None of this is heavy. The slice function is twenty lines. The refresh is a weekly job that pulls a stratified sample plus a short human pass to accept or reject cases. The reweight is a lookup table. Against that you are weighing a week of a broken language behind a green dashboard, which is what the old setup actually cost us. The corpus you evaluate against has to move at the speed your traffic moves, or the number it gives you ages out from under you.
What I'd check first
- The slice mix versus reality. Compare your set's language, length, tenant, and intent breakdown to last week's real traffic, and treat a wide divergence as the tell.
- The per-slice deltas across runs. Diff the slice output over your last two or three runs; a slice falling while the overall holds flat is the regression.
- The age of the cases. If most of the set predates your last launch, you are grading last year's product and the green number has already expired.
Top comments (1)
The distribution mismatch failure you describe is one that production eval systems keep rediscovering because it is invisible until a support ticket surfaces it. The fix you landed on — slicing by production distribution and refreshing from live traffic — is exactly right, but there is a harder version of this problem: when the failing slice is not a language but a query type that does not have a clean categorical label. In RAG specifically, a prompt change that improves precision on factual lookups can quietly hurt multi-hop reasoning queries, and those do not separate cleanly unless you have instrumented the retrieval path well enough to infer intent. Did you end up using any automated slicing — clustering or topic modeling — to find unexpected dimensions before the German complaints arrived?