Here is the ritual. You ship a prompt change, rerun the eval suite, and open the dashboard. Thirty numbers sit there: faithfulness, answer relevance, context precision, toxicity, latency-adjusted quality, and two dozen more. Twenty-nine are flat. One dropped from 0.86 to 0.81 and the cell is red. Someone says "we regressed on groundedness," and the next hour goes to explaining a number that never needed explaining.
I want to make the boring case that most of these red cells are not findings. They are what you get when you run many comparisons and let each one fire on its own. This has a name, the multiple comparisons problem, and it has arithmetic you can do on a napkin.
Claim 1: the false alarm rate is per-metric, and you have a lot of metrics
Set a threshold for "this metric moved" that would fire 5% of the time by chance when nothing actually changed. That is what a two-sided test at alpha = 0.05 means: a 5% false positive rate per test, under the null.
One metric, one test, 5% chance of a false alarm. Fine. But you are not looking at one metric. Assume for a moment the metrics are independent (they are not, and that matters below). The probability that at least one of n metrics throws a false alarm is:
P(at least one false positive) = 1 - (1 - alpha)^n
n = 1: 1 - 0.95^1 = 0.05 (5%)
n = 20: 1 - 0.95^20 = 0.64 (64%)
n = 30: 1 - 0.95^30 = 0.79 (79%)
At 30 metrics, a run where nothing changed has a 79% chance of showing you at least one red cell. The expected count is just n times alpha: 30 x 0.05 = 1.5 false alarms per clean run. So the "one metric moved" you are staring at is, on average, exactly what a no-op change produces. You did not find a regression. You found the metric that lost this round of a lottery you run every deploy.
Claim 2: your metrics are correlated, which does not save you
The obvious objection: eval metrics are not independent, so the formula above is wrong. Correct, it is wrong. It is not wrong in your favor.
Correlation changes the shape of the false-alarm distribution but not its center. The expected number of false positives is n times alpha regardless of dependence, because expectation is linear and does not care whether the tests move together. Correlation mostly changes the variance. When metrics are highly correlated, false alarms clump: a clean run tends to show either zero red cells or several at once, because the correlated metrics fail together. That is worse for interpretation, not better, because a cluster of three red cells looks like a "real pattern" and is the single most convincing way to fool yourself. That cluster is three correlated metrics failing on the same noise, not three independent findings.
So you cannot correlation your way out of this. You have to correct for it.
Claim 3: the fix is a decision made before you look, not a harder stare after
There are two standard corrections, and they answer two different questions.
Bonferroni controls the family-wise error rate, the probability of even one false positive across the whole family. To hold that at 0.05 across 30 metrics, you test each at 0.05 / 30 = 0.00167. It is exact enough (it slightly over-corrects under dependence) and it is strict. Strict is the point when a single false regression would block a release.
Bonferroni is often too strict when you genuinely track many metrics and can tolerate a few false alarms among your flagged ones. That is what the false discovery rate is for. Benjamini and Hochberg's 1995 procedure (Controlling the false discovery rate, Journal of the Royal Statistical Society Series B) controls the expected proportion of your flagged metrics that are false, rather than the chance of any false flag at all. The procedure is short: sort your n p-values ascending, find the largest k where p(k) <= (k / n) x q for your target rate q, and flag everything up to k. It gives you far more power than Bonferroni while still bounding how much of your red is noise. For a dashboard where you expect a handful of real movements among thirty metrics, FDR is usually the right tool. For a release gate where one false block is expensive, family-wise control is.
Either way, the correction is a rule you commit to before opening the dashboard. The failure mode is picking the correction after you have already seen which cell is red, because by then you are choosing the test that lets you believe what you already decided.
Claim 4: separate the metric you are testing from the metrics you are watching
The cleaner fix is upstream of any correction. Most eval suites conflate two things:
Confirmatory metrics are the one or two you changed the prompt to improve. You have a directional hypothesis. You test those, at full alpha, and you are allowed to act on them.
Exploratory metrics are the other twenty-eight you keep on the board for monitoring. These do not get to trigger a "we regressed" conversation on a single run. They get watched over multiple runs, and a real regression in one shows up as a trend across several deploys, not a one-time red cell that is gone next Tuesday. A drop that reverts on the next run without any code change was regression to the mean, which is the second-most-common way a dashboard lies to you.
Decide which bucket each metric is in before the run. Two primary metrics tested at 0.05, twenty-eight secondary metrics under an FDR rule or simply held to a "two consecutive runs" bar. Now your false-alarm budget is one or two tests wide, not thirty.
The rule I actually use
Before the run, write down the one or two metrics the change is supposed to move, and the direction. Everything else is monitoring. For the monitoring set, do not react to a single red cell: apply an FDR correction across the set, or require the movement to survive a second run. When you must gate a release on a broad panel, use family-wise control and accept that you will miss small real effects, because at the gate a false block costs more than a missed 0.01.
None of this makes your eval more sensitive. It makes the eval account for how many comparisons you ran before a cell turned red.
FAQ
Does this mean my dashboard is useless?
No. A dashboard is a fine monitoring instrument. It is a bad hypothesis test. Use it to watch trends across runs, not to adjudicate a single deploy. The moment a single-run red cell triggers a decision, you have turned a monitor into a test without paying for the test.
If I only ever look at one metric, do I need any of this?
No. One pre-specified metric, one test, no correction needed. The problem is entirely created by the number of comparisons you let fire. Two or three metrics tied to a hypothesis tell you more than thirty scanned for whatever happened to move.
Bonferroni or Benjamini-Hochberg?
Bonferroni when a single false alarm is expensive, such as a release gate: it controls the chance of any false positive. Benjamini-Hochberg when you track many metrics and can tolerate a known fraction of false flags in exchange for catching more real ones. They optimize different things on purpose.
My eval scores are not p-values, they are just averages. Does this still apply?
Yes, and it is easier to abuse. A raw score delta with no notion of sampling variability is a comparison with an implicit, undocumented threshold. The multiplicity is still there; you have just hidden the alpha. Put a confidence interval on each delta and the correction problem becomes visible again.
Will correcting make me miss real regressions?
It will reduce power, yes. That is the trade. The answer is not to skip the correction, it is to keep the confirmatory set small so each test keeps full power, and let the large monitoring set be governed by trend, not by a single run.
Open question
Family-wise and FDR corrections both assume the family is well defined: you know how many tests you are running. On a live eval dashboard the family is open-ended. You add metrics over months, you rerun after every deploy, and you look at the board on days when nothing shipped. The true number of comparisons is not 30, it is 30 times every run times every glance. I do not have a clean way to define "the family" for a metric board that is queried continuously by a whole team. Sequential testing methods (alpha-spending, always-valid p-values) are the honest direction, but I have not seen them adapted well to eval suites where the metric set itself keeps growing. If you have a working formulation of the family for a continuously-watched dashboard, I would like to see it.
Top comments (0)