You launch the test on Monday. You check the dashboard each morning, because of course you do. On Thursday it reads p = 0.04, you call it, and you ship.
That p-value did not mean what the tooltip said it meant.
Why looking changes the result
A p-value of 0.05 is a promise about a specific procedure: collect a predetermined sample, then test once. Under that procedure, a true null hypothesis produces a false positive 5% of the time.
Checking repeatedly is a different procedure. The statistic wanders as data accumulates, and each look is another opportunity for that wandering to cross the line. You are not measuring whether the effect is real. You are measuring whether it ever looked real.
What it costs, concretely
Simulate two identical variants — no effect whatsoever — and stop the moment p drops below 0.05:
import numpy as np
from scipy import stats
rng = np.random.default_rng(0)
def run(peeks, n_per_peek=200, trials=10_000):
false_positives = 0
for _ in range(trials):
a, b = [], []
for _ in range(peeks):
a.extend(rng.binomial(1, 0.10, n_per_peek))
b.extend(rng.binomial(1, 0.10, n_per_peek)) # identical
if stats.ttest_ind(a, b).pvalue < 0.05:
false_positives += 1
break
return false_positives / trials
for p in (1, 5, 10, 20):
print(f"{p:2d} looks -> {run(p):.1%}")
1 looks -> 5.0%
5 looks -> 14.2%
10 looks -> 19.4%
20 looks -> 25.6%
Twenty looks — four weeks of checking on weekdays — and one experiment in four reports a winner that does not exist. Every one of those will be defended with a screenshot showing p < 0.05.
Three ways out
1. Fix the sample size in advance, and don't look.
Compute it before launching, from the smallest effect worth shipping:
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
effect = proportion_effectsize(0.10, 0.11) # 10% -> 11%
n = NormalIndPower().solve_power(effect, power=0.8, alpha=0.05, ratio=1)
print(f"{n:,.0f} per variant")
This is the honest option, and the one most teams skip because the number that comes back is uncomfortably large. That discomfort is the information: it's telling you the effect you're hoping for needs more traffic than you have.
2. Look, but adjust the threshold.
Sequential testing methods — O'Brien-Fleming boundaries, alpha spending — let you monitor while keeping the overall false positive rate at 5%, by demanding much stronger evidence early on.
3. Stop reporting significance at all.
Report the observed difference and a confidence interval. "+1.2% [-0.4%, +2.8%]" communicates both the estimate and the uncertainty, and it doesn't invite a binary ship/don't-ship reading from a number that was never binary.
The one thing that isn't negotiable
Whatever you choose, decide the stopping rule before you see any data. Peeking is only a problem because the decision to stop depends on what you saw. A fixed rule, even a crude one, removes the mechanism entirely.
The uncomfortable implication
Most teams running weekly experiments on modest traffic are underpowered for the effects they're chasing, and their significant results are dominated by this bias. The fix is not a better dashboard. It is running fewer, larger tests on changes big enough to be worth detecting.
These posts come out of material I build for my Udemy courses — 25 of them now, mostly drill-based, across Go, Python, TypeScript, testing and Three.js. If this was useful, the full list is at udemy-c1f90.web.app. The links on that page carry a coupon I refresh each month, which usually lands around half the list price.
Top comments (0)