How many outcomes before you can quote a percentage?

Your model outputs 0.87. Someone puts it on a dashboard labeled "87% likely to succeed." A customer makes a decision with it.

How many resolved outcomes did you actually need before that number meant anything?

There's a closed-form answer, it's older than all of us, and most teams never run it.

The number

For a proportion you want to quote within ±ε at 95% confidence:

n ≥ p̂(1 − p̂) · (z / ε)²

z = 1.96 for 95%. That's it. Run it:

from math import ceil

def min_n(p_hat: float, eps: float, z: float = 1.96) -> int:
    """Resolved outcomes needed to quote p_hat within ±eps."""
    return ceil(p_hat * (1 - p_hat) * (z / eps) ** 2)

for p in (0.5, 0.8, 0.9):
    print(p, min_n(p, 0.05))
# 0.5 385
# 0.8 246
# 0.9 139

385 resolved outcomes to say "50%, give or take 5 points." Not 385 rows in your training set — 385 cases where you found out what actually happened.

The part that hurts

Precision is quadratic. Tightening ε by 5× costs 25× the data:

for eps in (0.10, 0.05, 0.02, 0.01):
    print(eps, min_n(0.5, eps))
# 0.1   97
# 0.05  385
# 0.02  2401
# 0.01  9604

Going from "roughly half" to "50% ± 1 point" costs about 9,500 outcomes. If you're in a domain where outcomes take 90 days to resolve, that's not a sprint — that's years.

Read it backwards

The more useful direction is inverting it. Instead of asking how much data you need, ask what your current data licenses you to say:

def widest_claim(p_hat: float, n: int, z: float = 1.96) -> float:
    """The ± band your evidence actually supports."""
    return z * (p_hat * (1 - p_hat) / n) ** 0.5

print(round(widest_claim(0.5, 50), 3))   # 0.139

At n=50, your honest claim is ±14 points. So "50%" is really "somewhere between 36% and 64%." That's not a probability you should be printing next to a dollar amount.

This reframes the whole thing. You don't have a precision problem, you have a vocabulary problem: at n=50 you're licensed to say bands, not percentages. Say the band.

Wilson, not the naive interval

One correction: the formula above is the normal approximation, and it degrades badly at small n or extreme p̂. It'll happily hand you an interval that extends past 1.0. Use Wilson instead:

def wilson(successes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    if n == 0:
        return (0.0, 1.0)
    p = successes / n
    d = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / d
    half = z * ((p * (1 - p) / n + z**2 / (4 * n**2)) ** 0.5) / d
    return (centre - half, centre + half)

print(wilson(9, 10))   # (0.5958, 0.9821)

Nine out of ten successes is not "90% reliable." It's "somewhere between 60% and 98%," which is a very different sentence to put in front of a customer.

Making it a constraint, not a guideline

Here's the part that matters in production: a rule nobody enforces is a rule that erodes under deadline pressure. Documentation won't save you. Put the gate in the code path.

MIN_N = 385

def render_score(p_hat: float, n_resolved: int) -> str:
    if n_resolved < MIN_N:
        lo, hi = wilson(round(p_hat * n_resolved), n_resolved)
        return f"band: {lo:.0%}–{hi:.0%} (n={n_resolved})"
    return f"{p_hat:.0%} (n={n_resolved})"

Now the honest version is the default path, and overclaiming requires someone to delete code in a reviewed PR. That's a very different organizational fact than a wiki page saying "be careful with probabilities."

One trap: which n?

n is resolved outcomes, and resolution is where the bias sneaks in.

If your wins auto-resolve from downstream data but your losses require somebody to manually mark them, your resolved set is not a random sample of reality. It's skewed toward wins, and every interval you compute on it is confidently wrong. Missing-not-at-random data doesn't announce itself — it just makes your calibration look great.

Before you trust the number, check whether the mechanism that produces outcomes differs by outcome type. That check has caught more bad dashboards for me than any model diagnostic.

Why I care about this

I build billing software. When our system says a claim is likely to be recovered, somebody staffs against that number and a practice makes a payroll decision. The cost of a confidently wrong percentage is somebody's month.

So we ship bands until the volume clears, and the product is architecturally forbidden from saying "probability" before then. It costs us a nicer-looking demo. It has never once cost us a customer.

---

If you want the longer argument — what it means for a system to be structurally forbidden from overclaiming, and why the entity that makes a claim must never be the one that settles whether it was right — I wrote it up here: The Uncapturable Judge.

What's your team's rule for when a score gets to call itself a probability?