You iterate on a prompt, run the eval set, keep the best variant. Repeat for a quarter. The number that survives is the maximum of forty noisy measurements, and the maximum of noisy measurements is an upward-biased estimate of the thing you care about. The size of that bias is computable, it is larger than most of the "wins" teams celebrate, and there is a decade-old literature on how to reuse a holdout without fooling yourself that I almost never see cited in LLM eval writeups.
The selection machine
The workflow looks responsible. A fixed eval set of 250 examples, a binary pass judgment per example, every candidate prompt measured on the same set. Best score ships. It looks like discipline.
The problem is the word "best." Each measurement is the true pass rate plus sampling noise. Pick the maximum of forty measurements and you have preferentially picked positive noise. The winner's measured score overstates its true score even when some candidates are genuinely better than others; the statistics community has called this the winner's curse for decades.
How big the bias is
For a pass rate near p on n examples, one measurement has standard error sqrt(p(1-p)/n). At p = 0.85 and n = 250 that is 2.26 points. Now suppose, worst case, all k candidates are actually identical: every measured difference is pure noise. The expected maximum of k standard normal draws is easy to compute numerically:
- 5 experiments: expected max 1.16 sigma, so +2.6 points at n = 250
- 10 experiments: 1.54 sigma, +3.5 points
- 40 experiments: 2.16 sigma, +4.9 points
- 100 experiments: 2.51 sigma, +5.7 points
Forty tries against a 250-example set buys you an expected phantom improvement of about five points. If your quarterly review says the prompt work moved the pass rate from 85 to 89, the null hypothesis says: that is what selection looks like when nothing improved at all.
Twelve lines to check my arithmetic:
import numpy as np
rng = np.random.default_rng(0)
n, p, k, sims = 250, 0.85, 40, 100_000
# k identical candidates, each measured once on n examples
runs = rng.binomial(n, p, size=(sims, k)) / n
best = runs.max(axis=1)
print(f"true rate: {p:.3f}")
print(f"mean best-of-{k}: {best.mean():.3f}")
print(f"inflation: {(best.mean() - p) * 100:.1f} points")
This prints about +4.7, just under the normal-approximation figure above for k = 40 (the binomial's discreteness and ties shave off a couple of tenths). Change k and re-run: the bias keeps growing, roughly like sqrt(2 ln k), so it slows with persistence but never stops.
The part the field already solved
Dwork, Feldman, Hardt, Pitassi, Reingold, and Roth worked out the general problem in 2015 under the name adaptive data analysis (the reusable holdout line of work; arXiv:1506.02629). Their observation: the moment your next experiment depends on the last measurement, the holdout stops being fresh, and classical guarantees quietly void themselves. Their mechanism, Thresholdout, answers holdout queries through a noise-adding gate so the set survives many more adaptive looks.
You do not need to implement Thresholdout to benefit from the diagnosis. Four practices capture most of the value:
- Two sets, two roles. Iterate freely against a dev set. Confirm on a frozen set you touch only when shipping. The confirm set's job is to be boring: it answers a handful of times per quarter, not forty.
- Selection-adjusted bars. If you compared k variants, the winner must clear the baseline by more than the expected max of k noise draws, not by more than zero. For k = 40 at n = 250, that bar is about five points, which is sobering and correct.
- Count your touches. k is the one quantity in this whole analysis you control and can log exactly. Teams version their prompts and their datasets and keep no record of how many times the eval set has been queried. Make k a first-class number in the eval report.
- Retire burned sets. After enough adaptive looks, refresh the eval set from new traffic and re-baseline. Not because the data drifted (that is a separate failure with its own essay) but because your process has memorized this particular sample's noise.
FAQ
Does a bigger eval set fix this? It helps: sigma scales as 1/sqrt(n), so quadrupling the set halves the inflation. It does not change the shape: the bias still grows with k. A bigger set lowers the noise floor. The selection effect sitting on top of it remains.
Is this just p-hacking with a new name? It is the same statistical family. P-hacking selects among analyses of one experiment; this selects among experiments measured on one reused set. The adaptive data analysis literature treats both as instances of the same leakage.
Has anyone measured whether this happens in practice, or is it theory? Measured, with a twist. Recht, Roelofs, Schmidt, and Shankar built a fresh CIFAR-10 test set in 2018 (arXiv:1806.00451) expecting to catch years of test-set reuse as overfitting. The drop they found pointed at distribution shift instead, and adaptivity-driven overfitting was smaller than theory allows. Two honest readings: benchmark-scale selection has correction mechanisms (many teams, public leaderboards, diverse models), and a single team hammering a private 250-example set with forty variants of one prompt is a much purer selection machine than ImageNet ever was. The math above is the worst case; your k and your n decide how close you sit to it.
Open question
Is there a principled touch budget for an eval set: a k beyond which refresh is mandatory, stated in advance the way significance levels are? I have a heuristic (refresh when the selection-adjusted bar exceeds the effect sizes you care about) but I have not seen a treatment that turns it into a rule teams actually adopt. If you have seen one in the wild, I want to read it.
Top comments (0)