A preference judgement is not one measurement. It is a collision between a customer, a request and a person, and only one of those three is in the spreadsheet twice. Take 4,000 human judgements of an LLM feature that is exactly as good as the control, and the A/A false-positive rate depends entirely on which column you decided was independent.
| what you called independent | A/A false-positive rate |
|---|---|
| every rating | 65.6% |
| by customer | 12.0% |
| by rater | 39.4% |
| by customer and rater | 7.2% |
Five standard errors spanning 3.82x off one file, and on a single seed the same −4.46pp verdict carries z = 6.15 or z = 1.24.
The classical correction is one line, and the parameter nobody configures is inside it.
SE_naive = sd / sqrt(n)
deff = 1 + rho * (m_eff - 1)
m_eff = sum(n_i**2) / sum(n_i) # NOT the mean
Mean requests per customer is 8. Under Zipf traffic, which is what real traffic is, m_eff is 112.9.
So the headline: everyone argues about the formula and the answer is in the sample. Same 4,000 ratings, same $4,800, same estimand - cap how much of the eval one customer may supply at 8x its fair share and the estimate's true standard deviation falls 0.0307 to 0.0175 while power rises 39.4% to 58.0%. That is worth 3.08x the ratings, and nothing was bought: the biggest customer goes from 12.7% of the sample to 1.6%, and Kish's effective cluster count from 35.4 of 500 customers to 154.3.
Full detail, computed in the browser: https://dev48.infy.uk/ai/days/day72-preference-eval-variance.html
Three things I had written down that did not survive
The 1/p² dilution rule is the first. At a 15% trigger rate the ratings needed rise 1.36x, not 44.44x, because a preference metric forces a tie on the requests the feature never touched and a forced tie carries no variance. The rule is right on a continuous outcome and wrong on this one.
mSPRT is the second. Always-valid inference is sold as the licence to look at every observation; run it on data that is not independent and it spends 43.0% at one look and 46.0% at ten. It fails before you peek at all. It is valid against peeking and against nothing else.
The third is the one I most expected to go the other way. Equal-per-customer sampling looks like the clean fix for skewed traffic - and it makes clustering by customer worse, 12.0% to 17.2%, because flattening the traffic moves the variance onto the axis you were not correcting.
Two more that cost money. The published small-cluster repair, a wild cluster bootstrap-t, is exact on the axis it was written for and blind to the other: 6.6% to 2.4% on Zipf traffic with no rater panel, 16.8% to 15.6% with a real panel. And the obvious two-way extension is provably the wrong quantity - its resampling variance is the sum of squared cell sums, the exact term the two-way formula subtracts, verified against the identity and 11.92x too small.
Nine designs at the same $4,800 and seven are invalid. 1,232 verifier assertions, 500 Monte-Carlo worlds per estimate, one file, no network.
One AI idea a day, computed rather than quoted: https://dev48.infy.uk/aifromzero.php
Top comments (0)