You changed the system prompt, reran the 300-item suite, and accuracy went from 76.0% to 78.0%. Two points. You shipped it. Three weeks later someone reran the same suite on the same prompt and got 75.7%, and now nobody trusts the eval.
Nothing broke. The 2-point win was never there. At n=300 and p≈0.77, the standard error of a single accuracy number is about 2.4 percentage points — your "improvement" is smaller than one standard error of either measurement. This is LLM eval noise, and it is the single most common reason teams ship prompt and model changes that do nothing.
TL;DR
- At n=300 items and ~77% accuracy, the standard error on accuracy is ~2.4pp, so a 95% CI spans roughly ±4.8pp. A 2pp delta is inside the noise floor.
- Never compare two accuracy numbers. Compare per-item outcomes on the same items and run McNemar's exact test — only the items where the two systems disagree carry signal.
- Concretely: 18 flips to correct, 12 flips to wrong (net +2pp on 300 items) gives p ≈ 0.36. Not evidence.
- Detecting a true 2pp gain needs ~7,000 items per arm unpaired, but only ~1,000–2,000 items paired, because pairing cancels item-difficulty variance. Pairing is a 4–7x cost reduction, not a nicety.
- Log per-item, per-run records (
item_id,run_id,prompt_version,seed, raw output, grade) so any two runs can be paired retroactively. Without that, you cannot recover the statistics later.
Why does a 2-point eval gain usually mean nothing?
Because accuracy on a finite suite is a random variable, and its spread is larger than most engineers expect.
Treat each item as a Bernoulli trial with success probability p. The standard error of the measured accuracy is sqrt(p(1-p)/n). At p=0.77:
| n | SE | 95% CI half-width |
|---|---|---|
| 100 | 4.2pp | ±8.2pp |
| 300 | 2.4pp | ±4.8pp |
| 1,000 | 1.3pp | ±2.6pp |
| 5,000 | 0.6pp | ±1.2pp |
Now compare two systems the naive way, as independent proportions. The SE of the difference is sqrt(p1q1/n + p2q2/n). For 76% vs 78% at n=300 that's about 3.4pp, so z = 0.02/0.034 ≈ 0.58 and p ≈ 0.56. A coin flip.
The naive comparison is also the wrong test, and its wrongness is expensive. It assumes the two runs sampled two independent sets of items. They didn't — they ran the same items. That shared structure is exactly what you should be exploiting.
How do you tell LLM eval noise from a real gain?
Pair on items and count only the disagreements. If both systems get item 47 right, item 47 tells you nothing about which system is better. The signal lives entirely in the discordant cells.
Build the 2x2 table:
| new correct | new wrong | |
|---|---|---|
| old correct | a | c |
| old wrong | b | d |
Under the null hypothesis "the change has no effect," each discordant item is a fair coin: it was equally likely to flip either way. So b should follow Binomial(b+c, 0.5). That's McNemar's test, and with small counts you want the exact binomial version, not the chi-square approximation.
from math import comb
import random
def mcnemar_exact(b: int, c: int) -> float:
"""Two-sided exact McNemar.
b = items the new system fixed, c = items it broke."""
n = b + c
if n == 0:
return 1.0
k = min(b, c)
tail = sum(comb(n, i) for i in range(k + 1)) / 2 ** n
return min(1.0, 2 * tail)
def paired_bootstrap(old, new, iters=10_000, seed=0):
"""95% CI on the accuracy delta, resampling items (not runs)."""
rng = random.Random(seed)
n = len(old)
d = [nw - od for od, nw in zip(old, new)]
means = []
for _ in range(iters):
s = sum(d[rng.randrange(n)] for _ in range(n))
means.append(s / n)
means.sort()
return means[int(0.025 * iters)], means[int(0.975 * iters)]
# 300 items: 228 -> 234 correct. 18 fixed, 12 broken.
print(mcnemar_exact(18, 12)) # 0.3616
p = 0.36. Your 2-point win is one plausible sample from a null world. Note that pairing did improve things — the unpaired p was 0.56 — it just wasn't enough at this n.
The bootstrap gives you the number worth reporting to a PM: a paired CI on the delta, which for these counts straddles zero comfortably. Report the interval, not the point estimate. "+2.0pp, 95% CI [−1.6, +5.6]" is honest; "+2 points" is not.
One correction people miss: if your suite has multiple items drawn from the same source document, conversation, or customer, resample clusters, not items. Item-level bootstrap on clustered data understates variance, sometimes badly.
How many eval items do you actually need?
Enough that the effect you care about exceeds the noise floor — and pairing changes the answer by 4–7x.
Unpaired, to detect 76% → 78% at α=0.05 and 80% power, the standard two-proportion formula gives roughly 7.85 × (p1q1 + p2q2) / δ² ≈ 7,000 items per arm. Most teams do not have 7,000 graded items, and if they do, they cannot afford to run them on every prompt tweak.
Paired, the sample size depends on the discordance rate π_d, not the accuracy level:
- π_d = 10% (systems disagree on 1 item in 10): ~2,000 items
- π_d = 5%: ~1,000 items
That inversion is the key intuition. The more similar your two systems are, the cheaper it is to detect a small difference between them, because there are fewer noisy discordant items diluting the signal. A prompt tweak that changes 4% of outputs is statistically easier to evaluate than a model swap that changes 40% — even for the same net delta.
Practical consequence: measure π_d first. Run both systems, count disagreements, then decide whether your suite is large enough to answer the question. If π_d is 30% and your suite is 200 items, stop and go collect more data instead of running the test.
Should you resample k times per item instead of adding items?
Only if within-item variance is a real share of the total. Decompose it:
Var(mean_score) = (σ²_between_items + σ²_within_item / k) / n
σ²_between comes from items being of different difficulty. σ²_within comes from the model being stochastic on a given item: temperature > 0, tie-breaking near the top of the distribution, and the fact that even nominally deterministic serving isn't bit-identical across batch shapes.
Estimate σ²_within directly — run k=5 samples per item on a 100-item subset and look at the per-item variance:
def variance_split(runs):
"""runs: {item_id: [0/1, ...] with k samples each}"""
ks = [len(v) for v in runs.values()]
k = ks[0]
means = {i: sum(v) / k for i, v in runs.items()}
within = sum(
sum((x - means[i]) ** 2 for x in v) / (k - 1)
for i, v in runs.items()
) / len(runs)
gm = sum(means.values()) / len(means)
total = sum((m - gm) ** 2 for m in means.values()) / (len(means) - 1)
between = max(0.0, total - within / k)
return between, within
If most items score 0.0 or 1.0 across all k samples, σ²_within is near zero and resampling buys you nothing — spend the budget on more items. If items cluster around 0.4–0.6, the model is genuinely coin-flipping and k=5 will tighten your estimate more cheaply than 5x the items, because you skip the grading and authoring cost of new items.
For agentic or multi-step tasks, σ²_within is usually large: one bad tool call early cascades. For single-turn classification with a strict grader, it's usually small.
What about the 12 prompt variants you tested?
If you tried 12 variants and shipped the best one, your reported delta is biased upward by construction. This is the winner's curse, and it is severe: the expected maximum of 12 independent standard normal draws is about 1.6σ. With a per-comparison SE of 3.4pp, that's roughly a +5pp phantom gain from selection alone, before any real effect.
Two fixes, both cheap:
- Split your suite. Select on a dev split, then confirm the single winner on a held-out split you touch once. The confirmation number is unbiased; the selection number is not.
- Shrink the reported estimate. If you must report from the selection set, report the CI and state how many variants were compared. A reviewer who knows you tried 12 will discount appropriately.
Do not apply Bonferroni across variants and call it done — that controls false positives but leaves you with no unbiased effect size, which is the number you actually need for a ship decision.
What should the eval harness log?
Per-item, per-run rows. Anything less and you cannot pair retroactively.
{"run_id":"r-2026-07-27-a","item_id":"sql-0412","cluster_id":"doc-88","model":"claude-sonnet-4-5","prompt_version":"v3","temperature":0.0,"sample_idx":0,"grade":1,"latency_ms":1840,"output_hash":"9f2c...","grader":"exact-match"}
{"run_id":"r-2026-07-27-b","item_id":"sql-0412","cluster_id":"doc-88","model":"claude-sonnet-4-5","prompt_version":"v4","temperature":0.0,"sample_idx":0,"grade":0,"latency_ms":2110,"output_hash":"1ab7...","grader":"exact-match"}
With item_id and cluster_id present, any two run_ids become a paired comparison with a JOIN. Store the raw output too — when McNemar says "12 items broke," you want to read those 12 items. Half the time the grader is wrong, not the model, and grader noise is its own variance term that silently inflates σ²_within.
When should you ship a statistically insignificant gain?
When the change is free and the interval is centered positive. Statistical significance is a claim about evidence, not a shipping policy. If a prompt edit costs nothing in latency or tokens and the paired CI is [−1.6, +5.6], the expected value is positive and shipping is fine — just don't put "+2%" in the changelog.
Flip it for regressions. If a change doubles latency or cost, you need a one-sided non-inferiority test with an explicit margin: "new is not worse than old by more than 1pp." That's a different, usually harder, question — and it's the one that should gate anything expensive.
The short answer
A 2-point accuracy gain on a few-hundred-item suite is almost always LLM eval noise, because the standard error of accuracy at n=300 is roughly 2.4pp — larger than the effect you're claiming. To separate signal from noise, run both systems on the same items, tabulate only the disagreements, and apply McNemar's exact test plus a paired bootstrap CI on the delta. Pairing cuts the items needed to detect a 2pp effect from about 7,000 to about 1,000–2,000, and the fewer outputs your change touches, the cheaper the test gets. Log per-item results with stable item IDs so every future run can be paired against every past one, hold out a split for confirming whichever variant won, and report intervals instead of point estimates.
Top comments (0)