Paired eval, wrong test: two prompts scored on one 100-item set give paired outcomes, so ranking them needs McNemar, not the two-proportion Wald SE. On the same data, Wald read 1.01 SE and said 'collect more'; McNemar read 2.65 SE and allowed the ranking. Same 100 items, opposite decision.
I shipped a little eval helper that refuses to rank two configurations when their gap is smaller than two standard errors. The idea is good. The refusal is honest. Last week it looked at two prompts scored on the same 100-item set and printed this:
RANK: INDISTINGUISHABLE - gap 7.00 pp against 6.95 pooled SE = 1.01 SE < 2.0. Ranking "prompt B" above "prompt A" is NOT allowed.
Seven points of apparent improvement, called noise. The honest reading of that line is "collect more items." And it was wrong. Not a rounding error, not a close call. The right test on that exact data says the ranking is already decided, and I could have stopped.
The bug was not in the arithmetic. It was in which test the arithmetic ran. My helper was treating two paired runs as if they were two independent samples.
TL;DR
- Two prompts on one task set are paired: each task gives a pair
(pass_A, pass_B), and both-pass / both-fail tasks carry zero information about the difference. - My harness used the Wald SE of a difference of two proportions,
pooled = sqrt(se_A**2 + se_B**2). That is the independent-samples formula; it ignores the pairing entirely. In my showcase that cost real power. - The paired test is McNemar:
SE = 100*sqrt(b+c)/n, using only the discordant countsbandc. On the same 100 items it returned 2.65 SE, not 1.01, and ranking was allowed. With just 7 discordant pairs the tool also prints the exact binomial,p=0.0156(z-equivalent 2.42), so the decision does not ride on the normal approximation. - On 535 real paired observations from my own sweep, one pair reads 2.65 SE under Wald and 6.24 under McNemar with
c=0: a strictly nested result that Wald reports as a hair over the line and has no way to flag as deterministic. - The fix is about fifteen lines. The hard part is not the formula, it is noticing your runs are paired.
What this is and is not. The two-prompt table below is a constructed example: I picked the counts to sit exactly on the line where the two tests disagree, so you can reproduce it with four numbers. The 535-observation table comes from a synthetic marker fixture I wrote, not from anyone's production system. Where a number is forced by my constants rather than measured, I say so in the same paragraph. That habit is half the point.
Why two prompts on one set are paired
Run prompt A and prompt B on the same 100 eval tasks. Task 7 either trips both of them, or neither, or exactly one. That last group is the only one that tells you which prompt is better. The tasks where both pass and the tasks where both fail are shared difficulty: they move both pass rates together and say nothing about the gap.
Here is the case that caught me, laid out as a 2x2:
B pass B fail
A pass 55 0 <- b = 0 (A pass, B fail)
A fail 7 38 <- c = 7 (A fail, B pass)
A marginal = 55/100 B marginal = 62/100 n = 100 pairs
Prompt A passes 55, prompt B passes 62. Same items. Ninety-three of the hundred tasks are concordant: 55 both pass, 38 both fail. Seven tasks are discordant, and all seven go the same way, B passes where A failed. Zero go the other way. B's pass set contains A's pass set completely. That is a strong statement, and it is invisible to a test that only looks at the two marginal rates.
What my harness printed, and why it was the wrong number
The refusal came out of rank(). Under the hood it builds the pooled standard error the way you were taught for two independent proportions:
pooled = sqrt(p1.se ** 2 + p2.se ** 2)
Each se is the binomial standard error of one marginal rate. For 55/100 and 62/100 that pools to 6.95 points, the 7-point gap divides to 1.01 SE, and the guard files it under "indistinguishable." Fair, if the two samples were drawn independently. They were not. The 93 concordant tasks are the same 93 tasks in both columns, and the formula charged me full variance for them anyway.
The cost of believing that line is real budget. At 1000 items each, the same seven-point gap finally clears the bar:
RANK: "prompt B" > "prompt A" - gap 7.00 pp = 3.18 SE >= 2.0. Ranking is allowed.
So the harness asked for ten times the eval spend to reach a verdict the correct test already had at 100 items. For anyone paying per token to run a judge model over a set, that is a straight line from a stats mistake to a bill.
What McNemar does instead
McNemar's test (Quinn McNemar, Psychometrika, 1947, the standard test for paired nominal data) throws away the concordant pairs and looks only at b and c. The standard error of the difference becomes 100*sqrt(b+c)/n, and the test statistic is |c-b|/sqrt(b+c). I added it to the same library as mcnemar(name_a, b, name_c, c, n). On the exact same 100 items:
McNEMAR: discordant pairs prompt A=0, prompt B=7 (concordant 93 of n=100).
SE = 2.65 pp (100*sqrt(b+c)/n); marginal gap 7.00 pp = 2.65 SE >= 2.0 -> ranking "prompt B" over "prompt A" is allowed.
small discordant count (b+c=7 < 25): the normal approximation is anti-conservative here. Exact two-sided binomial p=0.0156 (z-equiv 2.42), continuity-corrected z=2.27. all three clear the 2.0 bar; the decision is unchanged.
NESTED (b=0 or c=0): every discordant item favours the same side; the difference is deterministic (strict nesting on this sample), not a coin-flip margin.
2.65 SE against the same 2.0 threshold. Allowed. Same data, opposite decision. And note the third line. With only 7 discordant pairs, 2.65 is a normal approximation, and the tool will not let me lean on it: beside it sit the exact two-sided binomial, p=0.0156 (z-equivalent 2.42), and the continuity-corrected z at 2.27. All three clear the 2.0 bar. Same decision, reached without trusting the largest number in the row. The NESTED line fires because one discordant count is zero, which means the two prompts never disagreed in both directions: on this sample B dominates A item by item. That is a qualitatively different thing from a noisy 7-point margin, and a test that pools marginals cannot see it.
One honesty note, because it matters. NESTED is not a license to rank anything with a zero in it. A pair with b=0, c=1 is also nested, and the tool prints it at 1.00 SE, nowhere near the bar. The ranking decision still rides on the z value. In this case the z is 2.65 and the nesting is strict, so both agree. I flag them separately on purpose.
The same divergence on 535 real observations
The constructed example is clean but you should distrust anything I can tune to land on the line. So here is the same phenomenon on data I did not hand-pick: a sweep of a monotone-but-not-total marker fixture, P_PATHS=12 W_PER_TICK=3 T_TICKS=400 SEEDS=20, where a false reject is a landed write that a single-integer witness wrongly rejected.
The cells of that sweep share one underlying draw. Path and outcome come off the LCG independently of the axis knobs, so the raw (tick, path, outcome) at each sampled position is identical across cells. The run checks that before it does anything else, across all 33 sampled points per seed:
(tick,path,outcome) skew 6:6 vs 7:5 identical: True (33 sampled points)
(tick,path,outcome) streams 8 vs 4 identical: True (33 sampled points)
ALL PAIRED (raw observations identical across cells): True
Same observations, only the verdict moves. That is the definition of paired, and it is why the two-proportion SE is the wrong tool here too. Cross-tabbing the false-reject indicators gives real b/c counts. Both tests, side by side:
pair | b / c / n | Wald (rank) | McNemar (paired)
------------------------------------------------------------------------------
6:6 vs 7:5 | 142 / 128 / 535 | 0.87 SE | 0.85 SE
7:5 vs 8:4 | 39 / 3 / 535 | 2.31 SE | 5.55 SE
8:4 vs 9:3 | 39 / 0 / 535 | 2.65 SE | 6.24 SE NESTED
streams=8 vs streams=4 | 91 / 32 / 535 | 3.98 SE | 5.32 SE
streams=2 vs streams=1 | 229 / 0 / 535 | 20.01 SE | 15.13 SE NESTED
Read the third row. 8:4 versus 9:3 is 39 discordant pairs, all in one direction, c=0. Strict nesting again, on real counts. Both tests clear the 2.0 bar here, so both allow the ranking. But Wald reports 2.65, a hair over the line, while McNemar reports 6.24 and flags NESTED. One of those tells you the result is deterministic on this sample; the other cannot tell "barely over the threshold by luck" apart from "decided." Here is that pair with both guards printing in full:
PAIR 8:4 vs 9:3 marginal FR: 8:4=172/535 9:3=133/535
8:4: 32.1% (k=172 n=535 SE=2.02)
9:3: 24.9% (k=133 n=535 SE=1.87)
RANK: "8:4" > "9:3" - gap 7.29 pp = 2.65 SE >= 2.0. Ranking is allowed.
McNEMAR: discordant pairs 8:4=39, 9:3=0 (concordant 496 of n=535).
SE = 1.17 pp (100*sqrt(b+c)/n); marginal gap 7.29 pp = 6.24 SE >= 2.0 -> ranking "8:4" over "9:3" is allowed.
Two honest observations about that table. First, the two tests never disagree on direction: whichever version each one ranks higher, they agree, on all five pairs. Second, at this 2.0 threshold none of the five flip the decision. Row one, 6:6 vs 7:5, is 0.87 under Wald and 0.85 under McNemar, both below the bar, both refused. Rows two through five sit above the bar for both. What moves is the reported SE, sometimes a lot, 2.31 against 5.55, and whether the c=0 nesting gets named at all.
And the gap runs both ways, which is why "McNemar is always more powerful" would be the wrong lesson. On 6:6 vs 7:5 Wald reads 0.87 against McNemar's 0.85, and on streams=2 vs streams=1 it reads 20.01 against 15.13: when the discordant pairs are many and lopsided, the independent-samples SE is not the conservative one, it understates the variance instead. The rule is not that one test wins, it is that you owe your data the test its design calls for.
So where is the decision flip? Near the bar. This particular sweep happens to spread its pairs away from the 2.0 line, so the two tests agree on every call even while their SEs diverge. The constructed 100-item example sits right on the line, which is where the difference between the tests stops being cosmetic and turns into a yes or a no. I did not engineer that to cheat; it is where most config bake-offs I have watched actually get decided, on a handful of items either way.
One thing I am deliberately not claiming
You may have noticed those false-reject levels, 75.9%, 64.9%, and want me to say something about them. I will not, and the same library is why. Run the construction-independence probe on those cells and it comes back:
FORCED BY CONSTRUCTION [streams=8]: conditional 75.9% (k=406 n=535 SE=1.85) is indistinguishable from unconditional 74.7% (k=493 n=660 SE=1.69) (0.48 SE < 2.0)
The level equals the unconditional pass-the-gate rate, 74.7% at n=660, because the fixture draws outcome and stamp from separate LCG steps. So the level is an artifact of my constants, not a measurement, and I quote none of them as findings. What survives that probe is the discordance structure, b and c, which is a genuine between-cell comparison. The McNemar inputs are real; the levels they sit next to are not. Running the probe and reporting the result is the only reason I trust the distinction.
The fix, in about fifteen lines
There is nothing clever in it. It is the discordant count and a square root:
disc = b + c
conc = n - disc
...
se = 100.0 * sqrt(disc) / n
gap = 100.0 * abs(c - b) / n
n_se = abs(c - b) / sqrt(disc) # = gap / se, n cancels
...
if b == 0 or c == 0:
# NESTED: strict set-nesting, deterministic on this sample
(Truncated by me: the wrapper handles b==c, the no-discordant case, and localized output; full function in measure.py.)
The formula is the easy part. The part that actually protects you is upstream and it is not in this function at all: the library cannot know your two runs are paired. You have to establish that yourself, by confirming the observations are the same items in the same order and only the outcome moved. That is the check the sweep runs before it calls mcnemar() at all, and if the raw observations had differed it would have refused to treat them as paired. A paired test on unpaired data is its own mistake.
The limits, plainly. SE = 100*sqrt(b+c)/n is the normal approximation to McNemar; for a handful of discordant pairs you want the exact binomial instead, which is why the function prints it automatically once b+c drops below 25, and I would not read the plain SE without it. The z threshold of 2.0 is a convention I carried over from the rest of the harness, not a law. And none of this touches the failure mode that actually costs you money, a silent false accept, because that is a question about your ground truth, not your arithmetic.
What does yours do
Everything here ran locally on a synthetic fixture: Python 3.13.5, stdlib only, offline, no keys, no funds, three runs byte-identical, exit 0, empty stderr. Two earlier pieces work the same eval harness from other angles: reading eval results by severity class instead of a flat pass rate and a static probe that found contamination points without running the agent. Both belong to the same family of pre-execution gates for AI agents.
I publish the runs that corrected my own reading, not only the ones that confirmed it. Follow along if that is your kind of thing. And a real question I do not have a clean answer to: when your last A/B put two agent versions at 76 and 74 out of 100 on the same set, did your harness run a paired test, or did it pool two marginals and quietly ask you for more data? I would like to know what yours does.
AI disclosure. I drafted this with an AI writing assistant and edited every line; the test choice, the sweep, and the reading are mine. Every output block is pasted from one real local run on 2026-07-21. measure.py sha256 b1b3702bccb6ab46…, paired_test.py sha256 fa687e05e8e69512…, run output sha256 d8e4e521b1f35ac0…. The library default stays Russian so the sha256 of two already-published runs keeps verifying; English is opted into explicitly.
Top comments (0)