TL;DR. To catch a drop from a 0.90 pass rate to 0.85 at 80% power (one-sided, alpha 0.05), you need about 253 examples. A 50-example set has roughly 35% power, so it misses that regression about two times in three. The move that matters is not "collect more data" as a slogan. Size the set to the effect you actually care about, report a confidence interval on each run instead of a bare point delta, and prefer per-criterion binary labels over vague graded scores.
The two-point win that wasn't
A few months ago I sat in a review where a team was pleased with itself. A new prompt had moved their eval pass rate from 87.5 percent to 90 percent, and someone had already written "prompt v3: +2pp" in the changelog. The eval set had forty examples.
Here is what those points were made of. On forty examples every result is a multiple of 2.5 points, so the pass rate can only land on 35 out of 40, or 36, or 37, with nothing in between. The move from 87.5 to 90 was 35 correct becoming 36 correct. One example. A single test case that used to fail now passed, and it could flip back next week when the decoding lands differently.
I asked the obvious question. If we reran the old prompt three more times, would it always score 35 out of 40? Nobody knew, because nobody had rerun it. The honest summary of that meeting is that we had watched one example change its mind, and we had written it into the changelog as progress.
This is not a story about one careless team. It is the default failure mode of eval-driven development. We compare two numbers, see a gap, and our brains supply a cause. The possibility we skip is that the gap sits entirely inside the noise.
Power is the number nobody computes
The quantity that decides whether your eval can see a regression is statistical power: the probability that your test declares a difference when a real difference of a given size exists. Power depends on three things. The effect size you want to catch, the baseline rate, and the number of examples. Most teams pick the effect size implicitly ("a five-point drop would matter"), never write down the baseline variance, and let the sample size be whatever happened to be sitting in the folder.
Jacob Cohen spent a career arguing that this is backwards, most durably in "Statistical Power Analysis for the Behavioral Sciences" (2nd ed., 1988). The discipline he asked for is boring and effective: decide the smallest effect that would change a decision, then size the study so you can actually see it. An eval set is a study. The same arithmetic applies, whether the outcome is a clinical result or a pass or fail from a grader.
For a pass rate the math is the two-proportion normal approximation. You have a baseline rate p0, a rate you would not want to miss p1, and you solve for the n that gives you, say, 80 percent power. For the case that comes up constantly, catching a slide from 0.90 to 0.85, the answer is about 253 examples. Not forty. Not fifty.
Run the numbers yourself
Here is the whole calculation in standard-library Python. No dependencies, so you can paste it into a scratch file and swap in your own baseline and the drop you care about.
import math
def _phi(x): # standard normal CDF
return 0.5 * (1.0 + math.erf(x / math.sqrt(2.0)))
def _z(p): # inverse normal CDF (Acklam)
a = [-3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02,
1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00]
b = [-5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02,
6.680131188771972e+01, -1.328068155288572e+01]
c = [-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00,
-2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00]
d = [7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00,
3.754408661907416e+00]
plow, phigh = 0.02425, 1 - 0.02425
if p < plow:
q = math.sqrt(-2 * math.log(p))
return (((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) / ((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1)
if p <= phigh:
q = p - 0.5; r = q*q
return (((((a[0]*r+a[1])*r+a[2])*r+a[3])*r+a[4])*r+a[5])*q / (((((b[0]*r+b[1])*r+b[2])*r+b[3])*r+b[4])*r+1)
q = math.sqrt(-2 * math.log(1 - p))
return -(((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) / ((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1)
def power_one_proportion(n, p0, p1, alpha=0.05): # one-sided
z_a = _z(1 - alpha)
return _phi((abs(p0 - p1) * math.sqrt(n) - z_a * math.sqrt(p0*(1-p0))) / math.sqrt(p1*(1-p1)))
def n_for_power(p0, p1, power=0.80, alpha=0.05):
z_a, z_b = _z(1 - alpha), _z(power)
return math.ceil(((z_a*math.sqrt(p0*(1-p0)) + z_b*math.sqrt(p1*(1-p1))) / abs(p0-p1))**2)
p0, p1 = 0.90, 0.85 # detect a 5-point drop in pass rate
for n in (50, 100, 250):
print(f"n={n:4} power to catch {p0}->{p1} = {power_one_proportion(n, p0, p1):.2f}")
print("n needed for 80% power:", n_for_power(p0, p1))
n= 50 power to catch 0.9->0.85 = 0.35
n= 100 power to catch 0.9->0.85 = 0.51
n= 250 power to catch 0.9->0.85 = 0.80
n needed for 80% power: 253
Read it slowly. A fifty-example set has about 35 percent power against a five-point drop, which means it misses that regression roughly two times in three. At a hundred examples you reach 51 percent, a coin flip. You do not clear 80 percent until about 250, and the closed-form requirement is 253.
Two caveats carry weight here. This is a normal approximation and a one-sided test, appropriate when you only care about catching a drop rather than an improvement. And it describes a single run measured against a fixed baseline. If you evaluate the same items before and after (a paired design), the right test is McNemar on the items that flipped, and it needs fewer examples because it cancels the item-to-item difficulty that otherwise inflates your variance. If instead you compare two independent runs, that is a two-sample problem, and you need roughly twice as many per run. The forty-example set was never in the conversation.
When more labels are not an option
"Collect more data" is easy to say and expensive to do, because the labels are the cost, not the inputs. Three things help when you are genuinely stuck at a small n.
Report an interval, not a point. Every pass rate is an estimate with a standard error, and near the boundaries where eval scores live (roughly 0.85 to 0.97) the ordinary Wald interval behaves badly. Use the Wilson score interval instead, from Wilson, E. B. (1927), "Probable inference, the law of succession, and statistical inference," JASA. For 36 out of 40, the Wilson 95 percent interval runs from about 0.77 to 0.96. Print that next to the number. An interval that wide makes the two-point win argue against itself, and nobody has to say a thing.
Prefer per-criterion binary labels over graded scores. A vague one-to-five rubric hides two problems: raters disagree about whether an answer is a three or a four, and that disagreement is pure measurement noise that widens every interval you compute. Splitting the judgment into specific binary criteria (did it cite a source, did it follow the format, did it answer the question) is easier to label reliably, and each example then yields several outcomes instead of one. The honest caveat is that criteria inside one example are correlated, so k criteria are not k independent examples. You still come out ahead, because the reliability gain is real and the correlated-labels problem is smaller than the rater-noise problem it replaces.
Fix the effect size before the run, not after. Decide in advance the smallest regression you care about, size the set to it, and write the decision rule down. Five points at 80 percent power means 253 examples. If you can only get 80, then say plainly that you are powered to catch about a ten-point drop and nothing smaller, and stop reading two-point moves as signal. A known blind spot is a manageable risk. An unknown one ships regressions to users.
The lesson I keep relearning is that eval sets fail quietly. A set that is too small does not error out. It returns a number with two decimal places and a confident sign, and that number is noise that reads exactly like a real measurement.
FAQ
Does a bigger or better model change the sample size?
Only through its pass rate. The arithmetic depends on the effect size and the baseline rate, not on which model produced the answers. A stronger model with a higher baseline actually needs fewer examples to catch the same absolute drop, because the variance p(1-p) shrinks as the rate approaches 1. Catching a five-point slide from 0.97 costs far fewer labels than the same slide from 0.90.
What if my eval is paired, the same items scored under both versions?
Then use McNemar's test on the discordant pairs, the items that passed under one version and failed under the other. It ignores the items that agree, and it needs fewer examples than the independent-samples formula because it removes item difficulty from the variance. The catch is that if almost nothing flips, you have almost no information no matter how many items you scored. Paired designs concentrate all the signal in the disagreements.
One-sided or two-sided?
One-sided is right when you will only act on a drop, which is the regression-detection case. If you will act on a move in either direction, use two-sided and expect to need somewhat more, since the critical value rises from 1.645 to 1.96. Do not reach for one-sided just to shrink the sample size on paper.
Open question
The clean part of this is the single-metric, binary-outcome case. The messy part is the multi-criterion eval, where each example carries five or ten correlated binary judgments and I want one honest power calculation for the whole thing. Treating the criteria as independent overcounts the evidence, and treating each example as one outcome throws information away. The right answer sits somewhere in between, governed by the intra-example correlation, and the design-effect corrections from cluster sampling are the closest tool I know of. The trouble is that they need a correlation estimate you usually do not have until you have already run the eval several times. I do not have a tidy recipe for sizing a correlated multi-criterion eval up front, and non-binary rubric scores make it harder still. If you have solved this in a way that survives contact with real data, I would like to read it.
Top comments (0)