TL;DR. Almost every eval harness reports a pass rate with an error bar, and almost every one of those error bars comes from the normal approximation: p̂ plus or minus 1.96 times the square root of p̂(1 - p̂)/n. That formula is taught first, implemented everywhere, and reasonable near a pass rate of 50 percent. It falls apart at the extremes, which is precisely where any model worth shipping lives. At 49 of 50 passing it produces an upper bound of 1.0188, a probability above one. At 50 of 50 it produces the interval [1.0, 1.0], a claim of perfect certainty from fifty observations. Worse than either artifact: when the true pass rate is 98 percent and n is 50, its actual coverage is 63.5 percent. The reframing is that this is a solved problem, and has been since 1927. Invert the score test instead of approximating around the estimate, and you get the Wilson interval, which is one argument change in the library you already have installed.
The regression I shipped because I misread an interval
Three years ago I owned the eval suite for a document extraction pipeline. Fifty held-out documents, hand-labeled, each either extracted correctly or not. We were shipping a prompt change and the numbers looked clean: 49 of 50 passing, and the harness printed a 95 percent confidence interval of [0.941, 1.000].
I read the lower bound and did what most people do with a lower bound. I treated it as the pessimistic case. I told the stakeholder that the worst plausible outcome was about 94 percent, that we would be fine at 94 percent, and we shipped on a Thursday.
Production accuracy landed near 91 percent. That was outside the interval I had quoted, and I spent a week looking for the distribution shift that explained it. There was a small one. It was not the story. The story was that my interval had no business excluding 91 percent in the first place. Running the same 49 of 50 through the Wilson interval returns [0.895, 0.996]. The lower bound is 89.5 percent, not 94.1 percent. Wilson's interval contained the truth. Mine did not, and the gap between the two was not a rounding difference. It was 4.6 percentage points of false confidence, manufactured by a formula that should not have been applied to that data.
The distribution shift got the postmortem. The interval got nothing, because nobody in the room, including me, thought of the error bar as a thing that could itself be wrong. That is the failure mode I want to argue against here, criterion by criterion.
What the Wald interval is, and where it comes from
The estimator is uncontroversial. With k passes out of n cases, p̂ = k/n.
The interval most harnesses report is the Wald interval:
p̂ ± z · sqrt( p̂(1 - p̂) / n )
The derivation is a normal approximation to the binomial, with one extra move that does the damage: the standard error is evaluated at p̂, the number you happened to observe, rather than at p, the parameter you are trying to bracket. That substitution is harmless when p̂ is near 0.5 and n is large. Near the boundaries it is not harmless at all, because the standard error sqrt(p(1-p)/n) is itself a function of p that collapses to zero as p approaches 1. Plug in p̂ = 1.0 and the formula obediently reports that it has no uncertainty.
So the question is not whether the Wald interval is defensible in general. It is whether it is defensible in the regime where eval results actually land. Below are five criteria any interval estimator should satisfy, and how Wald does against each.
Criterion 1: the interval must stay inside the parameter space
A pass rate is a proportion. It lives in [0, 1]. An interval that includes values above 1 is not reporting uncertainty about a proportion, it is reporting that the model has been arithmetically mangled.
At 49 of 50, the Wald interval computes to [0.9412, 1.0188]. The upper bound is a probability of 1.0188.
Here is the part that keeps this from being obvious, and the reason I did not catch it for years. Your library hides it. In statsmodels, the proportion_confint function ends with an explicit clip for exactly two methods:
if method in ["normal", "agresti_coull"]:
ci_low = np.clip(ci_low, 0, 1)
ci_upp = np.clip(ci_upp, 0, 1)
So proportion_confint(49, 50, method="normal") returns [0.9412, 1.0000], and the 1.0188 never reaches your logs. The docstring says so plainly, and nobody reads it, because who reads the docstring for a confidence interval.
The clip is cosmetic, and I can show that precisely: clipping the upper bound from 1.0188 down to 1.0 only removes values in the range (1.0, 1.0188] from the interval, and no true proportion can live there. The set of true values the interval covers is identical before and after the clip. Coverage does not move by a single decimal place. What the clip accomplishes is that a number which would have announced the method's failure now looks like an ordinary, tidy upper bound of 1.0.
Criterion 2: the interval must not vanish when the data is unanimous
Run a perfect eval. 50 of 50. The Wald interval is [1.0, 1.0], width zero.
Read that as an epistemic claim and it says: having observed fifty successes, I am now certain, to the exclusion of all alternatives, that this system never fails. Not "very likely above 95 percent". Certain. A zero-width interval assigns probability zero to a true rate of 0.999.
Fifty of fifty is a perfectly ordinary eval outcome. It is also the outcome where the Wald interval fails hardest, and the failure has a sign: it always errs toward overconfidence, never toward caution.
Wilson at 50 of 50 returns [0.9287, 1.0]. Clopper-Pearson returns [0.9289, 1.0]. Both say the sensible thing: fifty consecutive passes is real evidence, it is consistent with a true rate around 93 percent, and you cannot rule out roughly a 1-in-14 failure rate on this evidence.
This connects to a heuristic some readers will know, the "rule of three": with zero failures in n trials, the upper bound on the failure rate is about 3/n. At n = 50 that gives 6 percent, implying a lower bound near 94 percent, which is close to Wilson's 92.9 percent but not equal to it. The reason is that the rule of three is the one-sided 95 percent bound. I checked: 1 - 0.05^(1/50) = 0.0582, and 3/50 = 0.06. The two-sided version needs 0.025 in each tail, which gives 1 - 0.025^(1/50) = 0.0711, closer to 3.7/n. Useful heuristic, commonly misquoted by half a tail.
Criterion 3: nominal coverage must resemble actual coverage
This is the criterion that matters, and the one nobody checks.
A 95 percent confidence interval makes a frequentist promise: across repeated experiments, the interval contains the true parameter 95 percent of the time. That promise is testable. The binomial has finitely many outcomes, so you do not even need simulation. Enumerate every k from 0 to n, ask whether the interval built from that k contains the true p, and weight by the binomial probability of observing that k. The answer is exact.
I ran that enumeration. At n = 50, nominal 95 percent:
| true p | Wald | Wilson | Clopper-Pearson |
|---|---|---|---|
| 0.50 | 0.935 | 0.935 | 0.967 |
| 0.80 | 0.938 | 0.951 | 0.967 |
| 0.90 | 0.879 | 0.970 | 0.970 |
| 0.95 | 0.920 | 0.962 | 0.988 |
| 0.98 | 0.635 | 0.922 | 0.982 |
| 0.99 | 0.395 | 0.911 | 0.986 |
At a true pass rate of 98 percent, an interval labeled "95 percent confidence" contains the truth 63.5 percent of the time. At 99 percent, it manages 39.5 percent. Those are not slightly optimistic numbers. An interval that misses the parameter more than a third of the time is not delivering what its label promises, and at 99 percent it misses more often than it hits.
Note the shape of the failure. Near p = 0.5 the Wald interval is fine (0.935, close enough to nominal). The degradation is monotone in how good your model is. The better the system under test, the more the interval lies, and it lies in the direction of telling you the system is more reliable than it is. Any team whose models improve over time is walking into this, and the error bar gets quieter about it every quarter.
None of this is news to statisticians. Brown, Cai and DasGupta laid it out in 2001 in Statistical Science, in a paper whose abstract describes the Wald interval's coverage as "erratic" and "chaotic," and states that "common textbook prescriptions regarding its safety are misleading and defective" (Brown, Cai and DasGupta, 2001, Statistical Science 16(2), 101 to 133). They recommend Wilson or the equal-tailed Jeffreys interval for small n, and Agresti-Coull for larger n. The paper is 25 years old. The formula it warns about is still the default in most eval code I read.
Criterion 4: more data must not make the interval worse
The standard defense of Wald is that it is asymptotic, so just use enough data. There are rules of thumb attached: n ≥ 30, or np ≥ 5, or np ≥ 10.
Test the rule. At p = 0.98, the binding constraint is the failure count, n(1 - p), because that is the small one. The np ≥ 5 rule, applied to failures, demands n ≥ 250.
At n = 250 and p = 0.98, Wald coverage is 0.873.
The rule is satisfied, and the interval is still nowhere near 95 percent. So the rule of thumb does not work, which is what "misleading and defective" meant.
It gets less comfortable. Coverage is not even monotone in n. Holding p = 0.98 fixed and computing exact coverage as n grows:
| n | Wald coverage |
|---|---|
| 125 | 0.916 |
| 142 | 0.941 |
| 150 | 0.800 |
| 225 | 0.935 |
| 250 | 0.873 |
Adding eight test cases, from 142 to 150, drops coverage from 0.941 to 0.800. More data, worse interval. I verified each of these two ways, by exact enumeration over the binomial and by a one-million-replication Monte Carlo, and they agree to three decimals (the Monte Carlo returns 0.9408, 0.8003 and 0.8732 for n = 142, 150 and 250).
The mechanism is not mysterious once you see it. k is discrete. As n changes, the lattice of achievable k values slides relative to the true p, and whether p sits inside the interval built from the most probable k flips on and off. That produces a sawtooth, which is why Brown, Cai and DasGupta call the behavior chaotic rather than merely biased. There is no threshold n above which you are safe, because the function you would be thresholding oscillates.
Which brings me to the honest caveat, and I would rather state it than have it found. Wilson oscillates too. Every interval for a discrete parameter does. Sweeping n from 50 to 600 at p = 0.98, Wilson's coverage ranges from 0.918 to 0.980, while Wald's ranges from 0.635 to 0.951. The difference is that Wilson's oscillation is centered near the nominal level, so its errors go in both directions and stay small. Wald's ceiling across that entire sweep is 0.951. It essentially never over-delivers, and its floor is 0.635. Wilson is not exact. It is well-behaved, which is a different and more achievable property.
Criterion 5: the interval should be asymmetric when the estimate is near a boundary
Wald intervals are symmetric by construction, because the formula is "estimate plus or minus a fixed quantity." Near a boundary, symmetry is the wrong shape.
At 49 of 50, the truth cannot be far above 0.98, since only 1.0 is available up there. It can quite easily be below, because 0.95 and 0.93 and 0.91 all produce 49 of 50 with unremarkable probability. The interval should be lopsided: short on the top, long on the bottom.
Wilson does this automatically. At 49 of 50 it returns [0.8950, 0.9965], which extends 0.085 below the point estimate and 0.0165 above it, roughly five times more room downward than upward. That asymmetry is not a patch bolted onto the formula. It falls out of the derivation, because Wilson inverts the score test: rather than approximating the standard error at p̂, it asks which values of p would fail to be rejected, evaluating the standard error at each candidate p. Solving that quadratic in p gives
center = (p̂ + z²/2n) / (1 + z²/n)
half-width = (z / (1 + z²/n)) · sqrt( p̂(1 - p̂)/n + z²/4n² )
Notice the center is not p̂. It is p̂ pulled toward 0.5, which is why the interval never escapes [0, 1] and never collapses at p̂ = 1: the z²/4n² term inside the square root keeps the width positive even when p̂(1 - p̂) is exactly zero. Wilson published this in 1927 (Wilson, 1927, Journal of the American Statistical Association 22(158), 209 to 212). It predates the eval harness by about ninety years.
The code
Pasteable, and the output below is what it actually prints:
# pip install statsmodels scipy
import numpy as np
from scipy.stats import binom, norm
from statsmodels.stats.proportion import proportion_confint
Z = norm.ppf(0.975) # 1.959963...
def wald_by_hand(k, n):
"""The textbook normal-approximation interval, unclipped."""
p = k / n
se = np.sqrt(p * (1 - p) / n)
return p - Z * se, p + Z * se
def show(k, n):
print(f"\n{k}/{n} passing (point estimate = {k/n:.3f})")
lo, hi = wald_by_hand(k, n)
print(f" Wald, by hand : [{lo:.4f}, {hi:.4f}] width={hi-lo:.4f}")
lo, hi = proportion_confint(k, n, alpha=0.05, method="normal")
print(f" Wald, statsmodels : [{lo:.4f}, {hi:.4f}] width={hi-lo:.4f}")
lo, hi = proportion_confint(k, n, alpha=0.05, method="wilson")
print(f" Wilson : [{lo:.4f}, {hi:.4f}] width={hi-lo:.4f}")
lo, hi = proportion_confint(k, n, alpha=0.05, method="beta")
print(f" Clopper-Pearson : [{lo:.4f}, {hi:.4f}] width={hi-lo:.4f}")
show(49, 50)
show(50, 50)
def exact_coverage(n, true_p, method):
"""Actual coverage, computed by enumerating every possible k.
No simulation: the binomial has finitely many outcomes."""
total = 0.0
for k in range(n + 1):
lo, hi = (wald_by_hand(k, n) if method == "wald"
else proportion_confint(k, n, alpha=0.05, method=method))
if lo <= true_p <= hi:
total += binom.pmf(k, n, true_p)
return total
print("\n\nActual coverage of a nominal 95% interval, n=50")
print(f"{'true p':>8} {'Wald':>8} {'Wilson':>8} {'Clopper':>9}")
for p in [0.50, 0.80, 0.90, 0.95, 0.98, 0.99]:
print(f"{p:>8.2f} {exact_coverage(50, p, 'wald'):>8.3f} "
f"{exact_coverage(50, p, 'wilson'):>8.3f} "
f"{exact_coverage(50, p, 'beta'):>9.3f}")
Output:
49/50 passing (point estimate = 0.980)
Wald, by hand : [0.9412, 1.0188] width=0.0776
Wald, statsmodels : [0.9412, 1.0000] width=0.0588
Wilson : [0.8950, 0.9965] width=0.1014
Clopper-Pearson : [0.8935, 0.9995] width=0.1060
50/50 passing (point estimate = 1.000)
Wald, by hand : [1.0000, 1.0000] width=0.0000
Wald, statsmodels : [1.0000, 1.0000] width=0.0000
Wilson : [0.9287, 1.0000] width=0.0713
Clopper-Pearson : [0.9289, 1.0000] width=0.0711
Actual coverage of a nominal 95% interval, n=50
true p Wald Wilson Clopper
0.50 0.935 0.935 0.967
0.80 0.938 0.951 0.967
0.90 0.879 0.970 0.970
0.95 0.920 0.962 0.988
0.98 0.635 0.922 0.982
0.99 0.395 0.911 0.986
Two library notes. In statsmodels, method="beta" is Clopper-Pearson (it is named for the Beta distribution used to compute it, which is a naming choice that has cost me time). In scipy, the same intervals are available as binomtest(k, n).proportion_ci(method="wilson") and method="exact" for Clopper-Pearson. I checked both libraries against each other and against the hand-rolled Wilson formula above; all three agree to machine precision.
What I do now
Wilson is the default. One keyword argument, no new dependency, correct behavior at the boundary, coverage that stays near nominal across the range where eval results actually land.
Clopper-Pearson when someone external is going to rely on the number, or when the cost of overstating reliability is asymmetric (safety filters, anything with a compliance story attached). It is called "exact" because it inverts the binomial test directly rather than approximating, but note from the table that its coverage runs to 0.982 and 0.986 where nominal is 0.95. It is conservative, and its intervals are wider than they strictly need to be. That is a deliberate trade, not a free upgrade. Agresti and Coull made this argument in 1998, in a paper titled "Approximate Is Better than 'Exact' for Interval Estimation of Binomial Proportions" (Agresti and Coull, 1998, The American Statistician 52(2), 119 to 126), and their point stands: guaranteed-conservative is not the same as accurate, and if you want coverage near 95 percent rather than above it, the approximate methods do better.
Report the interval, not the point estimate. "98 percent" and "98 percent, 95 percent CI [89.5, 99.7]" are the same measurement, and only one of them communicates that fifty test cases is fifty test cases. If the interval you get is too wide to support the decision you are making, the answer is more test cases, and the width tells you roughly how many. I worked that arithmetic in a separate piece on eval-set size, so I will not repeat it here.
And stop reading the lower bound as the pessimistic case. It is a bound on a plausible range, not a floor. I have made that mistake in production and it cost me a week of looking for a distribution shift that explained three of the nine points I was missing.
FAQ
Does this matter if my pass rate is around 70 percent?
Much less. At n = 50 and p = 0.80, exact Wald coverage is 0.938 against a nominal 0.95, which is a real but tolerable error. At p = 0.50 it is 0.935. The Wald interval was designed for this regime and behaves acceptably in it. The problem is that a 70 percent pass rate is usually a system you are still fixing, not one you are reporting on, and by the time you are writing the number in a document it has moved to 95 percent or higher. The method degrades precisely as your project succeeds, which is an unfortunate property for a measurement tool.
Is Wilson always better than Wald?
Not always, and I would rather not oversell it. At p = 0.50 and n = 50 they return identical coverage of 0.935, because near the center the two derivations nearly coincide. Wilson's advantage appears at the extremes and at small n, and it grows as you move toward the boundary. There is no regime I am aware of where Wilson is meaningfully worse, so "always use Wilson" is a defensible default even though "Wilson is always better" is too strong a claim. The honest version: Wilson is never worse in any way that matters, and is dramatically better exactly where you need it.
Why not just clip the Wald interval to [0, 1] and move on?
Because clipping changes nothing that matters. I showed this above: truncating an upper bound of 1.0188 to 1.0 only excludes values above 1.0 from the interval, and no true proportion lives there. The coverage is bit-for-bit identical before and after. What clipping does is remove the visible symptom, the absurd number that would have prompted you to check the method. statsmodels clips by default for normal and agresti_coull, which means the most common way to compute a Wald interval in Python is also the way that hides its most obvious failure.
What about the 50 of 50 case specifically? Is any interval sensible there?
Wilson and Clopper-Pearson both handle it, returning [0.9287, 1.0] and [0.9289, 1.0] respectively at n = 50. Both are saying the same reasonable thing: fifty consecutive passes is genuine evidence of a high rate, and it is also consistent with a true rate near 93 percent. Wald returns [1.0, 1.0], claiming certainty. If your eval dashboard has ever shown a perfect score with no error bar, or an error bar of zero width, this is why, and it is worth grepping your harness for.
Is Clopper-Pearson the safe choice since it is "exact"?
"Exact" describes the construction, not the coverage. Clopper-Pearson inverts the binomial test directly instead of approximating, which guarantees coverage of at least 95 percent. It overshoots: 0.982 at p = 0.98 and 0.986 at p = 0.99 in the table above, against a nominal 0.95. You are paying for that guarantee with width, and wide intervals have their own cost, since an interval too wide to distinguish two candidate models is not helping you decide anything. Use it when understating reliability is cheaper than overstating it. Otherwise Wilson.
Does the same problem hit the standard error I report on a per-metric basis?
Yes, for anything that is fundamentally a count of successes over trials. Pass rate, exact-match accuracy, tool-call validity, refusal rate, any binary judge verdict aggregated across a test set. If the reported number is k/n and the error bar is p̂ ± z·sqrt(p̂(1-p̂)/n), the analysis here applies unchanged. It does not apply to metrics that are means of continuous scores, like a 1-to-5 rating averaged across cases, which have a different and generally better-behaved sampling distribution.
What should I actually change in my code tomorrow?
Search for sqrt(p * (1 - p) / n) and for proportion_confint( without an explicit method argument, since statsmodels defaults to method="normal". Both are the Wald interval. Change the default to method="wilson". That is the whole migration. If you have historical eval numbers with published intervals, the point estimates are unaffected and only the intervals move, so a backfill is cheap and mostly widens old lower bounds.
Open question
The thing I have not resolved is what to do when the eval set is not a random sample, which is most of the time.
Everything above assumes n independent Bernoulli draws from a fixed distribution. Real eval sets are curated. We add cases because they broke something, we keep cases because they are hard, we group cases by document or by conversation so that the units are correlated rather than independent. Under curation the binomial model is the wrong likelihood, and an interval derived from it, Wilson included, is answering a question about a population that does not exist. Wilson gives you a correct interval for the wrong model. That is an improvement over Wald, which gives you an incorrect interval for the wrong model, but I notice it is a smaller improvement than this whole essay implies.
I have seen three responses. Treat the eval set as the entire population and report no interval at all, which is honest but gives up on generalization. Cluster-bootstrap at the document or conversation level, which handles the correlation but not the curation. Maintain a separate uncurated random sample purely for estimation, which is correct and which I have never seen anyone actually staff.
I do not have a defensible fourth answer. If you have shipped one, I would like to read it.
Top comments (1)
Ran every number in this before commenting — exact binomial enumeration, no simulation — and they all reproduce: Wald 49/50 = [0.9412, 1.0188], Wilson 49/50 = [0.8950, 0.9965], Wilson 50/50 = [0.9287, 1.0000], and the coverage claim lands exactly: 63.5% at p=0.98, n=50.
Your clip point is actually stronger than you stated it. Enumerated, clipping moves coverage by 0.0000000000 pp. Not "barely", not "to a rounding difference" — identically zero, because the clipped mass sits outside the parameter space entirely.
(stdlib only, two runs same sha256 c3ed6bfe8455d78b)
The thing I'd add is what the fix does to a decision, since an interval only matters when something gates on it. Take the rule people actually write: ship if lower95 ≥ 0.95, at n=50.
Wilson is 0.0% everywhere at n=50 — not "stricter", structurally unable to pass. Its best possible case is 50/50, whose lower bound is 0.9287, and 0.9287 < 0.95 always. So fifty cases cannot clear a 95% bar no matter what you observe. Meanwhile the Wald harness was passing 36.4% of the time at p=0.98, entirely on manufactured confidence.
At n=200 the gate becomes possible again: p=0.98 → Wilson 43.1% (Wald 78.7%); p=0.95 → Wilson 0.9%.
That reframes the swap as more than a correction. Wilson doesn't just widen the bar, it tells you your eval set is too small to make the claim your gate is making — the error bar wasn't only wrong, it was concealing that 50 documents can't authorize the ship decision. Which is the version of your postmortem I'd want on the wall: the distribution shift was a detail, the sample size was the finding.
Small nitpick on the rule of three: 0.0711 × 50 = 3.56, so at this n the two-sided version is nearer 3.6/n than 3.7/n.