Both arms drawn from the same distribution. No effect, by construction. So every "winner" is a false positive and the rate can be counted rather than argued about.
A fixed-horizon test — look once, at the end — rejects 5.27% of 4,000 null experiments. Exactly as advertised.
Then start checking the p-value as the data arrives, which every experimentation dashboard invites you to do:
| looks | false positive rate |
|---|---|
| 1 (fixed horizon) | 5.3% |
| 10 | 18.9% |
| 50 | 31.4% |
Same data. Same test. Only the stopping rule changed.
Run it yourself: https://dev48.infy.uk/ml/day63-ab-testing-peeking.html
The test is fine. The stopping rule is the bug.
A p-value is conditional on a design you fixed in advance: if there were no effect, and you ran to this predetermined sample size, you would see something this extreme 5% of the time.
The moment your rule becomes "keep going until p < 0.05", you are no longer sampling that distribution. You are sampling the minimum of a random walk over many looks — and the minimum of many draws is far more extreme than any single draw.
With enough looks a null experiment is guaranteed to cross 0.05 at some point. Not likely — guaranteed, in the limit. So "we ran it until it was significant" does not describe a weak result. It describes no result at all.
Three fixes, all measured against the same null
Group sequential boundaries. If you must look K times, use a threshold that accounts for K. Pocock applies one constant stricter threshold at every look. O'Brien–Fleming spends almost no alpha early and nearly all at the end — which matches how people want to behave anyway: stop early only for something spectacular.
Always-valid inference (mSPRT). Instead of a p-value whose validity depends on when you stop, build a statistic that is a martingale under the null. Ville's inequality then bounds the probability it ever crosses 1/α, over unbounded looks. I verified that at four different alphas — the tightest was α=0.01 crossing 0.45% of the time.
The cost is power: 97.1% on a 2-point lift against the fixed-horizon test's 99.9%. That is the honest price of not having to commit in advance, and any writeup that omits it is selling something.
A test of mine that was wrong
My first check asserted E[mSPRT] = 1 under the null. That is true — it is a likelihood ratio — but it is not estimable at a few hundred samples. The statistic is extremely heavy-tailed and its mean is carried by rare large values, so the sample average read 0.35 and said nothing about correctness.
The code was right and the test was wrong. Replaced it with the property that is actually estimable at this sample size: Ville's bound.
Top comments (0)