Comparing Two Eval Runs by Their Average Pass Rate Is the Wrong Test

TL;DR. You run version A and version B against the same 500-item eval set. A passes 71.4 percent, B passes 74.0 percent, and you conclude B is better. That reasoning throws away the one fact that matters most: both systems answered the same questions, so their per-item outcomes are correlated, not independent. Reading two separate averages (or eyeballing whether their confidence intervals overlap) is the wrong test for a same-items design. The fix is to pair the outcomes per item and test the difference directly. For pass/fail, that is McNemar's test on the items where the two runs disagree. For graded scores, it is a bootstrap over the per-item deltas. Report an effect size and a confidence interval on the delta, not two averages sitting next to each other.

I have shipped a regression because two dashboard numbers looked close enough. The new prompt scored 68.9 percent, the old one 69.7 percent, and I called it noise and moved on. It was not noise. The new prompt was quietly worse on a specific slice, and pairing the runs would have shown me a tight interval that sat entirely below zero. This has practical stakes, not just cosmetic ones: the unpaired reading can hide a regression that the paired reading would surface.

The rest of this post is one claim, argued criterion by criterion: when two eval runs share the same items, you owe them a paired analysis, and the paired analysis is not hard.

The setup, and why it looks reasonable

Here is the pattern I see in almost every eval report. There is a table with two rows. Row one is the baseline, row two is the candidate, and each row has a single number: mean pass rate, or mean rubric score, or mean judge score. Sometimes there is a 95 percent confidence interval on each row, drawn as a little error bar. The reader compares the two numbers, glances at whether the error bars overlap, and makes a call.

The instinct is not stupid. A mean is a legitimate summary, and an error bar is more honest than a bare point. The problem is narrower and more specific: the two error bars in that picture are each computed as if that run stood alone, and the comparison you actually care about (is B better than A) is a comparison the picture does not draw. The uncertainty you need is the uncertainty of the difference, and the difference has its own variance that depends on how the two runs move together across items.

That last phrase is the whole argument, so here it is in concrete terms.

Criterion 1: Same items means paired data, not two independent samples

Some prompts in your eval set are simply harder than others. A gnarly multi-hop question, an ambiguous instruction, a long context that buries the answer. Both A and B face that same hard prompt. When A fails it, B often fails it too, because the difficulty is a property of the item, shared by both systems.

Statistically, that shared difficulty induces a positive correlation between A's per-item outcome and B's per-item outcome. And the variance of a difference is not the sum of the variances when the two things are correlated:

Var(A - B) = Var(A) + Var(B) - 2 Cov(A, B).

When Cov(A, B) is positive (and with a shared test set it usually is), the variance of the delta is smaller than what you would get by treating the runs as independent. Two consequences follow, and they point in opposite directions depending on which mistake you make.

If you run an unpaired two-proportion test (the kind that assumes independence), you plug in a standard error that ignores that covariance term. You typically overestimate the uncertainty of the difference, which makes you underpowered: you fail to detect improvements that are actually there. If instead you eyeball two marginal confidence intervals and check for overlap, you hit a separate and well-documented trap. Two intervals can overlap while the paired difference is comfortably significant, because the overlap of marginal intervals is not the same question as whether the interval on the difference excludes zero.

This is not a niche observation. Dror, Baumer, Shlomov, and Reichart lay it out for language work in "The Hitchhiker's Guide to Testing Statistical Significance in NLP" (ACL 2018): the structure of your data determines which test is valid, and a shared test set produces dependent measurements that a naive test mishandles. Dietterich made the same point a generation earlier for classifiers in "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms" (Neural Computation, 1998), where he recommends tests that respect the paired structure of predictions on a single held-out set and warns against procedures that quietly assume independence.

So the first criterion is a modeling decision rather than a choice of statistic: same items, therefore paired. Everything below follows from taking that seriously.

Criterion 2: For pass/fail, McNemar's test looks only at the disagreements

When each item is a binary pass or fail, pairing the two runs sorts every item into one of four buckets:

• both pass
• both fail
• A passes, B fails (call it a_only)
• B passes, A fails (call it b_only)

The items where both systems agree carry no information about which system is better. They cancel. All the signal lives in the discordant pairs, the items where exactly one of the two got it right. McNemar's test (McNemar, "Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages," Psychometrika, 1947) asks a single question: among the discordant items, is the split between a_only and b_only further from 50/50 than chance would produce?

The exact version is a binomial test on the discordant count. The large-sample version is a chi-square statistic with a continuity correction, (|b - c| - 1)^2 / (b + c), on one degree of freedom. For the sample sizes typical of an eval set, I reach for the exact binomial and stop thinking about it.

import numpy as np
from scipy.stats import binomtest

rng = np.random.default_rng(0)
n = 500

# Some prompts are just harder for every system. That shared difficulty is
# exactly why the two runs are correlated, and exactly why you must pair.
difficulty = rng.beta(2, 2, size=n)
p_a = np.clip(0.45 + 0.40 * (1 - difficulty), 0, 1)   # system A
p_b = np.clip(0.50 + 0.40 * (1 - difficulty), 0, 1)   # system B, a bit stronger
pass_a = rng.random(n) < p_a
pass_b = rng.random(n) < p_b

a_only = int(np.sum(pass_a & ~pass_b))   # A right, B wrong
b_only = int(np.sum(pass_b & ~pass_a))   # B right, A wrong
n_disc = a_only + b_only

print(f"A={pass_a.mean():.3f}  B={pass_b.mean():.3f}  discordant: {a_only} vs {b_only}")
if n_disc == 0:
    print("no disagreements: McNemar is undefined, there is nothing to weigh")
else:
    p = binomtest(b_only, n_disc, 0.5, alternative="two-sided").pvalue
    print(f"exact McNemar p = {p:.4f}")

Notice the guard on n_disc == 0. It is not decoration. If your two runs are very similar (a small prompt tweak, a temperature change), you can land with almost no discordant items, and both the chi-square and the exact test degrade or become undefined. The honest reading in that case is not "p is large, no difference" but "I do not have enough disagreements to say anything," which is a different sentence and a cue to collect more items.

To make the reframe concrete, take an illustrative outcome from a run like the one above. Suppose across 500 items you see 40 items where only A passed and 53 where only B passed, with 407 concordant. The marginal gap is (53 - 40) / 500 = 2.6 points, which matches a 71.4 versus 74.0 headline. But McNemar is weighing 53 against 40 out of 93 discordant items, and the exact binomial p for that split is about 0.21 (illustrative). That 2.6-point gap is inside the range you would see from shuffling which system got the coin flip on the contested items. The averages made it look decided. The paired test says wait.

Criterion 3: For graded scores, bootstrap the per-item deltas

Plenty of evals do not produce a clean pass or fail. You get a rubric score in [0, 1], a 1-to-5 judge rating, a similarity score, a latency. The same logic holds, and the mechanical move is the same: form the per-item difference first, then reason about the distribution of those differences.

The paired bootstrap does this without assuming the deltas are normal. You compute diff_i = score_B(i) - score_A(i) for each item, then resample those paired differences with replacement many times, recomputing the mean each time, and read a confidence interval off the resampled means. The word "paired" is doing real work: you resample the differences, not the two score columns independently. Resampling the columns separately would break the pairing and reintroduce the very independence assumption you are trying to avoid.

import numpy as np

def paired_bootstrap_delta(scores_a, scores_b, n_boot=10_000, alpha=0.05, seed=1):
    rng = np.random.default_rng(seed)
    diffs = np.asarray(scores_b) - np.asarray(scores_a)   # pair FIRST
    n = len(diffs)
    if n == 0:
        raise ValueError("no items to compare")
    idx = rng.integers(0, n, size=(n_boot, n))            # resample the PAIRS
    boot = diffs[idx].mean(axis=1)
    lo, hi = np.quantile(boot, [alpha / 2, 1 - alpha / 2])
    return float(diffs.mean()), (float(lo), float(hi))

# Illustrative graded scores in [0, 1] on the same 400 items.
rng = np.random.default_rng(7)
difficulty = rng.beta(2, 2, 400)
scores_a = np.clip(rng.normal(0.70 - 0.20 * difficulty, 0.10), 0, 1)
scores_b = np.clip(scores_a + rng.normal(0.02, 0.08, 400), 0, 1)   # correlated with A

delta, (lo, hi) = paired_bootstrap_delta(scores_a, scores_b)
print(f"mean delta (B - A) = {delta:.3f}, 95% CI = [{lo:.3f}, {hi:.3f}]")

Bootstrap resampling of a single test set is not something I am inventing here. Koehn introduced it to machine translation evaluation in "Statistical Significance Tests for Machine Translation Evaluation" (EMNLP 2004), precisely to attach a confidence interval to a metric computed on one fixed test set, and the general method traces to Efron and Tibshirani's "An Introduction to the Bootstrap" (1993). The one discipline to keep is the pairing. If your items come in natural groups (several questions drawn from the same source document, or several turns from the same conversation), even the paired item bootstrap can understate uncertainty, and you want to resample at the level of the group. More on that in the open question.

Criterion 4: Report an effect size and a CI on the delta, not a bare p-value

Suppose the paired test comes back at p = 0.03. You still do not know whether B is better by half a point or by twelve. A p-value answers "could this be zero," not "how big is it and does the size matter to anyone." On a 5,000-item eval, a 0.3-point improvement can clear significance while being operationally meaningless. On a 120-item eval, a genuinely useful 4-point gain can miss the 0.05 line entirely.

So report the delta and its interval as the headline, and treat the p-value as secondary. For the binary case, the difference in paired proportions is (b - c) / n, and a standard error that respects the pairing is

SE = (1 / n) * sqrt(b + c - (b - c)^2 / n),

which for the illustrative 40-versus-53 example gives a 2.6-point delta with a roughly [-1.2, +6.4]-point 95 percent interval. That interval straddles zero, which is the same verdict McNemar gave, but now stated in units a product owner can act on. For the continuous case, the bootstrap interval on the mean delta is already in the right form.

This emphasis is exactly what the American Statistical Association argued for in its 2016 statement on p-values (Wasserstein and Lazar, "The ASA Statement on p-Values: Context, Process, and Purpose," The American Statistician): a p-value does not measure the size of an effect or the importance of a result, and good practice reports estimates with their uncertainty. Dror and colleagues make the same recommendation for language experiments. An effect size with an interval tells you direction, magnitude, and precision in one line. A lone p-value tells you one bit and hides the rest.

Criterion 5: If you compare many metrics at once, correct for it

Modern eval runs are wide. You are not testing one number, you are testing pass rate and faithfulness and a toxicity check and format-adherence and latency and a half dozen rubric dimensions, all at once, all with their own paired test. Each test at the 0.05 level has a 1-in-20 chance of a false alarm under the null. Run twelve of them and the probability of at least one false positive, if nothing truly changed, is 1 - 0.95^12, which is about 0.46. Almost a coin flip that you will "discover" a difference that is not there and chase it for a day.

There are two standard responses. Bonferroni divides your alpha by the number of tests (0.05 / 12 = 0.0042 each), which controls the chance of any false positive but is conservative and will hide real effects. Benjamini and Hochberg's procedure ("Controlling the False Discovery Rate," Journal of the Royal Statistical Society Series B, 1995) instead controls the expected fraction of your flagged results that are false, which keeps more power when several metrics really did move. For eval dashboards, where you would rather not miss a real regression, I default to Benjamini-Hochberg and reserve Bonferroni for the small number of metrics I would gate a release on. Either way, the point stands: the more comparisons you draw from one run, the more of them will look significant by luck, and a wide dashboard without any correction will regularly flag differences that are not real.

When the simple average is fine

I promised a measured claim, so here is the boundary. Comparing two averages is not always wrong, and there are cases where I do exactly that and sleep fine.

If the two runs used different, independently sampled item sets, they are not paired, and an unpaired analysis is the correct one. If the gap is so large that no reasonable interval could touch zero (A at 40 percent, B at 88 percent, on a few hundred items), the paired test will agree with your eyes and you can skip the ceremony for a quick read, then do it properly before you write it down. If you are not making a decision, just watching a number drift over time on a monitoring chart, a smoothed average is a fine early-warning signal and nobody needs a p-value to notice a cliff. And if your eval set is tiny (say under 30 items), no test rescues you; the honest move is to report the raw counts, resist a verdict, and go collect more data.

The through line is the same. The average is a fine description. It becomes the wrong test the moment you use two of them, side by side, on the same items, to decide which system won. That specific move, two averages side by side on the same items to decide which system won, is the one that needs a paired test.

I will also admit what pairing does not fix. It does not repair a biased judge, a leaky eval set, or items that do not represent production. Get a real improvement wrong and no statistic saves you. Pairing only makes sure that, given honest measurements, you read the comparison the measurements actually support.

FAQ

Is a two-proportion z-test fine if my eval set is large enough?
No, and size does not rescue it. The two-proportion z-test assumes the two samples are independent. With a shared test set they are not, because item difficulty is common to both runs. A larger n makes the wrong standard error more precisely wrong, not correct. What large n does buy you is stability for the paired test: McNemar and the paired bootstrap both behave well with more items, and their intervals tighten. So keep growing the eval set, but feed the counts into a paired procedure. The independence assumption is a property of the design, not of the sample size, and you cannot buy your way out of it with more rows.

My two confidence intervals overlap. Doesn't that mean no significant difference?
This is the most common trap in the whole topic. Overlapping marginal intervals do not imply the paired difference is non-significant. The interval you drew on A and the interval you drew on B each describe one run in isolation. The question you care about lives in a third interval, the one on the delta, which depends on how A and B covary across items. Because a shared test set makes them positively correlated, the interval on the difference is often much tighter than the overlap picture suggests. Draw the interval on the delta and check whether it excludes zero. Ignore the overlap of the two marginal bars.

What if I only have the two average pass rates, not the per-item results?
Then you cannot run the correct test, and you should go get the per-item results. This is the practical reason to log outcomes at the item level for every run: the paired analysis needs to know, item by item, whether each system passed. Two summary numbers have already discarded the pairing, and no post-hoc formula reconstructs it. If retrieving the raw outcomes is genuinely impossible, be honest that you can only make an unpaired, underpowered comparison, and treat any close call as undecided rather than shipping on it. Store the per-item pass/fail and scores by default so this never becomes the blocker.

When should I use McNemar's chi-square versus the exact binomial version?
Use the exact binomial test on the discordant count when that count is small, which for eval sets is most of the time. The chi-square form, with the continuity correction (|b - c| - 1)^2 / (b + c), is a large-sample approximation and is fine when the number of discordant pairs is comfortably into the dozens (a common rule of thumb is b + c of at least 25). Below that, the approximation drifts and the exact test is both safer and, in Python, no harder to call. When discordant pairs are near zero, neither version is meaningful; you simply do not have enough disagreements, and the answer is more data, not a smaller p.

How many items do I need for the paired test to detect a real difference?
It depends on the discordance rate, not the total item count, which is the counterintuitive part. Power comes from the items where the two systems disagree, so two nearly identical runs need a large set to accumulate enough discordant pairs, while two clearly different runs reach significance on far fewer items. As a rough planning move, estimate the fraction of items where you expect the runs to differ, multiply by your set size to get expected discordant pairs, and aim for that to be at least in the dozens. If you cannot get there, you are testing a difference too small to resolve at your current scale, which is itself a useful finding.

I track 15 metrics per run. Do I really need a multiple-comparisons correction?
Yes, if you are making decisions on whichever metric lights up. Fifteen independent tests at 0.05 give roughly a 1 - 0.95^15, about 0.54, chance of at least one false positive under the null. That is worse than a coin flip. You do not need to correct metrics you are only monitoring, but any metric that can trigger a decision (block a release, revert a prompt) belongs in a corrected family. I use Benjamini-Hochberg to control the false discovery rate across the dashboard, because it keeps more power than Bonferroni when several metrics genuinely moved, and I reserve the stricter Bonferroni cut for the two or three release-gating metrics where a single false alarm is expensive.

Open question

Pairing solves the correlation between two systems on the same item. It does not, by itself, solve the correlation between items that are not independent of each other.

Real eval sets are full of hidden clusters. Ten questions generated from the same source document. Eight turns sampled from the same conversation. A batch of items authored by the same annotator with the same blind spots. When items cluster like this, the effective number of independent observations is smaller than your row count, and both McNemar and the item-level paired bootstrap will hand you an interval that is too tight, because they treat correlated items as if they were independent draws. You end up overconfident in the opposite direction from where we started.

The candidate fixes are known in name: a cluster bootstrap that resamples whole groups instead of individual items, or a mixed-effects model with a random intercept per cluster. What I do not have is a clean, agreed-on recipe for the messy case where the clustering is partial and unlabeled, where some items share a document and others do not, and where nobody logged the grouping at eval-authoring time. How much does ignoring soft clustering actually inflate false positives on a typical agent eval, and is the cluster bootstrap worth the complexity for sets in the low thousands?

I have opinions and no proof. If you have run this comparison on your own evals, with the grouping tracked and the intervals computed both ways, I would genuinely like to see the numbers. That is the next thing I want to measure, and I would rather learn it from your data than guess.