Most of what I have written about LLM judges argues in one direction: your improvement is probably not real. Small eval sets, selection across many experiments, position bias, clustered examples. All of that pushes toward scepticism and I still believe it.
This post argues the other way, because there is a failure mode that runs opposite and I do not see it discussed. Judge noise biases your measured improvement toward zero. The interval gets wider, which everyone expects, and the point estimate also moves, systematically, in the same direction every time. A change that genuinely helped can read as flat, you will record a null result, and the noisier your judge the more confident that mistake becomes.
This is old statistics wearing new clothes. Spearman named it attenuation in 1904: an observed correlation is bounded above by the geometric mean of the reliabilities of the two instruments measuring it, so unreliable measurement makes real relationships look weaker than they are ("The proof and measurement of association between two things", American Journal of Psychology 15, 72-101, whose section II.2 is titled "'Attenuation' by errors"). An LLM judge is an instrument with reliability well under 1, and we report its output as though it were a ruler.
The arithmetic, which is embarrassingly simple
Take a binary judge: pass or fail. Suppose its flip probability is the same conditional on either true label, so it is as likely to turn a true pass into a reported fail as the reverse. Call that flip probability e.
If a model's true pass rate is p, the rate your judge reports is:
p_observed = p(1 - e) + (1 - p)e
= e + p(1 - 2e)
Now measure two models, true rates p_A and p_B, with the same judge. Subtract:
p_observed(B) - p_observed(A) = (1 - 2e) * (p_B - p_A)
The e term cancels. What survives is a multiplier on your effect: every measured gap is the true gap times (1 - 2e).
That factor bites at values of e that are entirely ordinary:
| Judge flip rate e | Disagreement with your labels | Attenuation factor | A true 5-point gap measures as |
|---|---|---|---|
| 0.00 | 0% | 1.00 | 5.0 points |
| 0.10 | 10% | 0.80 | 4.0 points |
| 0.15 | 15% | 0.70 | 3.5 points |
| 0.20 | 20% | 0.60 | 3.0 points |
| 0.25 | 25% | 0.50 | 2.5 points |
I checked the algebra against simulation rather than trusting it: 6,000 simulated eval runs of 1,600 examples each, true rates 0.70 and 0.75, e = 0.15, mean observed gap 3.5 points against a predicted 3.5. Note what that does and does not establish. The bias is the same at every eval-set size, so it confirms the algebra and says nothing about how many examples you need. That question comes next, and it is worse than the table suggests.
It costs you twice, and the second cost is a noise term with two sources
Shrinking the effect is the visible cost. The second depends on which test you are running, and I had it wrong at first.
The intuitive story is that noise pushes both observed rates toward 0.5, since p_observed - 0.5 = (1 - 2e)(p - 0.5), and 0.5 is where per-example variance is greatest. That is true and it is the right picture to hold. But it is the noise term of an unpaired comparison, and unpaired is the wrong test here. Eval examples are not independent draws: some prompts are hard for every system, that shared difficulty correlates the two arms, and I have argued before that comparing two runs by their average pass rate is the wrong test for exactly this reason.
So what does judge noise do to a paired comparison? The arithmetic is prettier than I expected. Write the reported label as A' = A + F_A(1 - 2A), with F_A a Bernoulli flip at rate e independent of everything else. Expand E[A'B'] and E[A']E[B'] and the cross terms are identical, so they cancel:
Cov(A', B') = (1 - 2e)^2 * Cov(A, B)
The factor that attenuates your effect appears squared in the covariance between arms. That is an identity rather than a simulation result, and the ratio does not depend on how correlated your arms were to begin with. I checked it at three different difficulty spreads and got covariance ratios of 0.495, 0.490 and 0.493 against a predicted (1 - 2e)^2 = 0.490, with the correlation itself falling from 0.27 to 0.12 in the middle case.
It is worth saying up front what that identity is for, because it does more work later than it looks like it is doing here. Pairing helps you only through the covariance, so a judge flipping labels at rate e keeps exactly (1 - 2e)^2 of whatever pairing was buying you. That one fact is why every number in this post moves when your examples get more or less uniform in difficulty, and it is the last section of this piece.
Which is why there are two costs on the statistic and not three. McNemar's test looks only at the discordant pairs, so its noise term is the discordant-pair rate, and that rate is
p_discordant = p_A + p_B - 2 p_A p_B - 2 Cov(A, B)
so judge noise inflates it from both directions at once: the marginal rates slide toward 0.5, and the covariance is destroyed. I decomposed it one factor at a time, in both orders, and they come out close to even: of the total inflation, about 48 percent is the marginal shift and about 52 percent is the covariance loss. Which means the "toward 0.5" intuition is not superseded by the covariance identity, it is one of two roughly equal ingredients in the same noise term. On my model that rate went from 0.294 to 0.400 at e = 0.15, and the whole effect on the test statistic is:
z' / z = (1 - 2e) * sqrt(p_discordant / p_discordant')
= 0.70 * sqrt(0.294 / 0.400) = 0.60
Two factors on the statistic: the effect shrinks, and the discordance inflates. What I got wrong at first was adding a third, by counting the marginal-variance story separately from the discordance it is half responsible for.
So here is the power calculation for the design this actually is: a per-example difficulty term, marginal pass rates calibrated to 0.70 and 0.75 (0.699 and 0.749 after calibration), McNemar's exact test on the discordant pairs, 6,000 simulated runs per cell.
| Eval set size | Power, perfect judge | Power, judge at e = 0.15 |
|---|---|---|
| 200 | 0.21 | 0.10 |
| 400 | 0.41 | 0.19 |
| 800 | 0.72 | 0.32 |
| 1,600 | 0.95 | 0.58 |
| 3,200 | 1.00 | 0.87 |
Those cells carry about a point of Monte Carlo error, so read them to two figures and no further. To reach 80 percent power on that effect I needed roughly 1,000 examples with a perfect judge and roughly 2,750 with the noisy one, about 2.75 times the eval set to buy back what a 15 percent flip rate cost.
At n = 800, a size that felt generous to me for years, that puts a real 5-point improvement below a one-in-three chance of surfacing. Most teams who try that change will shelve it on the evidence.
One caveat, because that table has a knob in it I nearly published without stating. Everything above assumes one level of between-arm correlation, which is to say one answer to how much your examples vary in difficulty. Change it and the whole table moves: at n = 800 a perfect judge's power runs from 0.59 at uncorrelated arms to 0.90 at a correlation of 0.57.
I expected the ratio between the two columns to be the stable thing. It is not, and it moves in the direction that makes this worse rather than better. Eval-set sizes for 80 percent power, across the same range:
| Between-arm correlation | Perfect judge | Judge at e = 0.15 | Ratio |
|---|---|---|---|
| 0.00 | 1,254 | 2,888 | 2.3x |
| 0.10 | 1,129 | 2,763 | 2.5x |
| 0.27 | 918 | 2,551 | 2.8x |
| 0.45 | 693 | 2,327 | 3.4x |
| 0.57 | 544 | 2,178 | 4.0x |
Those are closed-form rather than simulated, so the trend is exact rather than approximate. The first row is worth checking by hand, because at zero correlation pairing buys nothing and McNemar reduces to the unpaired test: the ratio there is just the variance inflation times the attenuation term, 1.13 * 2.04 = 2.3, which is the two factors from the previous section multiplied together.
So the more your examples differ in difficulty, the more a noisy judge costs you, and the covariance identity says why: pairing pays off precisely when the arms are correlated, and (1 - 2e)^2 is the fraction of that payoff you keep. At a correlation of 0.57 a perfect judge clears 80 percent power on about 540 examples and a judge flipping 15 percent of labels needs almost exactly four times as many. The 2.75x I quoted above is the middle of that range, not a constant. Measure your own arm correlation, which you can do directly from a paired run, and read the ratio off the row you land on.
What I do about it
Estimate e, and be honest about what you are estimating. You probably have a human-labelled set from validating the judge. What that gives you is the disagreement between two fallible instruments, which is a different quantity from e. If your humans flip at rate e_h and the judge at e_j, then for the observed disagreement d:
1 - 2d = (1 - 2*e_h) * (1 - 2*e_j)
I confirmed that on four million simulated labels and it holds to four decimal places. Which means dividing your gap by (1 - 2d) over-corrects by exactly 1/(1 - 2*e_h). Humans flipping 5 percent of labels inflates your effect by 11 percent; at 10 percent human error you are inflating by 25 percent. Two annotators agreeing does not make them right, which I have written about before, and it is precisely the assumption this correction smuggles in. So treat the corrected number as an upper bound unless you truly believe your labels are error-free.
Report the disattenuated effect only next to the raw one. Divide the observed gap by (1 - 2e): an observed 3.5 points at e = 0.15 implies a true gap near 5. The uncertainty is computable and belongs in the report. The standard error on (1 - 2e) from m human labels is 2*sqrt(e(1-e)/m), so at e = 0.15 with 100 labels the correction factor alone carries about 20 percent relative error, putting your corrected 3.5 points somewhere near 4.2 to 6.3 before any sampling error in the gap itself. At 500 labels that tightens to roughly 4.6 to 5.5.
Spearman's own worked example in the 1904 paper takes an observed 0.38 and corrects it to approximately 1, which he calls absolute and complete. The abuse is as old as the method.
Check whether you have noise or bias, and use the right null. Attenuation assumes the judge's errors do not depend on what you are measuring. If it prefers longer answers and model B is more verbose, that is a confound with a direction, and dividing by reliability amplifies it. I will not re-derive that argument here beyond the one thing people get wrong in code: the expected split of the disagreements is not even. Under symmetric noise E[flip_up] = (1-p)*e*n and E[flip_down] = p*e*n, so the expected ratio is (1-p) : p. At a pass rate of 0.70, perfectly symmetric noise produces a 30:70 split, and any test against 50/50 will call that lopsided and wrongly tell you not to correct. I ran exactly that mistake against simulated symmetric noise at pass rates from 0.5 to 0.9 and it produced a false "do not correct" verdict at every rate above 0.5.
import numpy as np
from scipy.stats import binomtest
def diagnose(human, judge, alpha=0.01):
"""human, judge: equal-length boolean arrays of pass/fail labels.
Returns e_hat, its standard error, and whether the disagreement is
consistent with symmetric noise (attenuation) rather than a directional bias."""
human, judge = np.asarray(human, bool), np.asarray(judge, bool)
m = human.size
flip_up = int((~human & judge).sum()) # human fail, judge pass
flip_down = int((human & ~judge).sum()) # human pass, judge fail
e_hat = (flip_up + flip_down) / m
se_e = np.sqrt(e_hat * (1 - e_hat) / m)
# The null is the HUMAN FAIL RATE, not 0.5: symmetric noise flips
# (1-p) of the failures up and p of the passes down.
n_disagree = flip_up + flip_down
pval = binomtest(flip_up, n_disagree, 1 - human.mean()).pvalue if n_disagree else 1.0
return {"e_hat": e_hat, "se_e": se_e, "attenuation_ok": pval > alpha, "p_value": pval}
rng = np.random.default_rng(0)
truth = rng.random(20_000) < 0.70 # symmetric noise, e = 0.15
judge = np.where(rng.random(20_000) < 0.15, ~truth, truth)
print(diagnose(truth, judge)) # e_hat ~ 0.15, attenuation_ok True
Do not assume cutting e beats buying examples. I believed this and it is wrong. Going from e = 0.15 to e = 0.08 is worth about 1.7 times the eval set, and at a fixed budget doubling the examples beat cutting e at every size I tested: at n = 800 that is 0.57 power against 0.49. The honest reason to attack e is that tightening a rubric or splitting a compound criterion is usually cheaper than labelling another 800 cases, and it also recovers some of the pairing benefit noise destroyed. Both interventions are legitimate. And the sample-size requirement scales as the inverse square of (1 - 2e), not of e, which is worth getting right before quoting it at anyone.
Stop reading a flat result as evidence of no effect unless you know your power at the effect size you cared about. A null with unknown power carries no information, and attenuation biases even that absence toward the conclusion you are about to draw.
FAQ
Does this apply to continuous scores, not just pass or fail?
Not in the same way, and this is the part I had wrong until I simulated it. Under classical additive error, where the measured score is the true score plus independent noise, a raw mean difference is not attenuated at all, because the noise has mean zero. What attenuates is a correlation or a standardised effect size, and there the correction is division by the square root of the reliability when the other variable is measured without error, or by the geometric mean of both reliabilities when neither is. The binary case above is not a special case of that. It attenuates the raw gap precisely because a flip is bounded rather than additive, so the expected judge score is 0.64 where the true rate is 0.70. Identify your error process before reaching for a correction.
My judge disagrees with humans 5 percent of the time. Am I fine?
Mostly. The factor is 0.90, so you lose about a tenth of every effect. Worth knowing when you are arguing over a 1-point difference, not worth restructuring around. Confirm the 5 percent was measured on traffic-like cases rather than your cleanest examples.
Can I average several judge calls and skip the maths?
Averaging independent calls reduces the random component of e, so it is a real intervention, and it recovers some of the lost pairing correlation as well. The systematic component is untouched by averaging, and it was a bias rather than attenuation to begin with.
Does a paired test fix it?
No, and that is the finding that surprised me most. Pairing remains the right design and it still helps. But the (1 - 2e) multiplier is a property of the instrument rather than of the comparison, so pairing cannot reach it, and per-judgment noise actively erodes the between-arm correlation pairing depends on.
Were my past null results wrong?
Some, and you can work out which. Take the effect you were hoping for, your e, and your eval-set size, and compute the power you actually had. Any null where power was under about 0.5 tells you nothing in either direction. Three of mine qualified.
Open question
The awkward case is a judge whose error rate depends on what it is reading. Everything above assumes one e applied to both arms. If model B produces output that is easier to grade correctly then e_B < e_A, the arms attenuate by different factors, and the observed gap mixes a real quality difference with a difference in how legibly each model fails. Those two are not separable from the judge's output alone.
Estimating e per model on a small human-labelled slice was my first attempt, and at the sample sizes I could afford the two estimates were too noisy to propagate, which the standard error above explains: separating a 3-point difference in e needs a couple of thousand labels per arm, not the few hundred I assumed when I started. If you have a design that identifies model-dependent judge reliability without a full human pass over both arms, I would like to read it.
Top comments (0)