Statistical Significance in A/B Testing (Without the Math Headache)

#webdev #abtest #discuss

“We hit 95% statistical significance!”

Cool — but do you actually know what that means?

Most teams use statistical significance as a finish line without fully understanding what they’re validating. This post breaks down the practical meaning of statistical significance, so you can make better decisions from your experiments — not just prettier dashboards.

What “95% Statistical Significance” Really Means

When a test reaches 95% significance, you’re saying:

“There’s only a 5% chance this result happened due to random variation.”

That 5% is the p-value. Lower p-value = stronger evidence your result is real.

It does not mean:

The result is guaranteed
The improvement is meaningful
The test should automatically ship

Statistics tell you confidence, not judgment.

The 4 Numbers That Actually Matter

1. Confidence Level (usually 95%)
How confident you want to be that the result is real.

2. Statistical Power (usually 80%)
Your chance of detecting a real effect when it exists. Low power = missed wins.

3. Minimum Detectable Effect (MDE)
The smallest improvement you care about. Smaller MDE → more traffic required.

4. Baseline Conversion Rate
Lower baselines need more users to prove the same lift.

Miss one of these, and your test is unreliable before it even starts.

Why Most A/B Tests Fail (Even at 95%)

Stopping early
Peeking daily inflates false positives dramatically.
Too many metrics
Test 10 metrics and one will “win” by chance.
Statistical ≠ practical significance
A 0.1% lift might be real — and still useless.
Not running full cycles
Weekend behavior ≠ weekday behavior.
Underpowered tests
If you needed 10,000 users and ran 1,000, the test was doomed.

When You Should Actually Call a Winner?

Before shipping a change, confirm: