DEV Community

Baneet Singh
Baneet Singh

Posted on

How many visitors does an A/B test actually need? (table + free calculator)

#abtesting #statistics #javascript #webdev

Most A/B tests that "won" never had enough visitors to win anything. Here is the math, a table you can bookmark, and a tiny zero-dependency calculator.

The short answer

At a 2% conversion rate, detecting a 20% relative lift (2.0% to 2.4%) takes about 21,106 visitors per version at 95% confidence and 80% power. Two versions means about 42,000 visitors on that page.

Smaller lifts and lower conversion rates need a lot more:

Current conversion rate Detect +10% Detect +20% Detect +30% Detect +50%
1% 163,092 42,691 19,824 7,747
2% 80,679 21,106 9,795 3,823
3% 53,208 13,911 6,452 2,515
5% 31,231 8,155 3,778 1,468
10% 14,749 3,839 1,772 707

Visitors per version. Two-sided test, 95% confidence, 80% power.

The formula

For two proportions p1 (current) and p2 (with the lift):

n = (z_a/2 + z_b)^2 * (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2
Enter fullscreen mode Exit fullscreen mode

with z_a/2 = 1.96 for 95% confidence (two-sided) and z_b = 0.84 for 80% power.

In JavaScript:

const Z = 1.959964 + 0.841621;
function sampleSize(base, lift) {
  const p2 = base * (1 + lift);
  return Math.ceil((Z * Z * (base * (1 - base) + p2 * (1 - p2))) / (p2 - base) ** 2);
}
sampleSize(0.02, 0.2); // 21106
Enter fullscreen mode Exit fullscreen mode

I put this (plus an inverse-normal function so you can change confidence and power, and a CLI) in a small MIT repo: baneets/ab-test-sample-size.

npx github:baneets/ab-test-sample-size 2 20 50000
# 21,106 visitors per version (42,212 total), 95% confidence, 80% power
# about 3.7 weeks at 50,000 visitors a month
Enter fullscreen mode Exit fullscreen mode

How long that takes

A page with 50,000 visitors a month at 2% needs about 3.7 weeks to detect a 20% lift, and about 14 weeks for a 10% lift. At 10,000 visitors a month, the 20% case takes about 18 weeks.

That is why button-colour tests on low-traffic stores almost never produce a real answer.

Three mistakes that create fake winners

  1. Peeking. Checking a fixed-sample test every day and stopping the first time it looks significant inflates false positives a lot. Either commit to the sample size, or use a method built to be checked as it runs (sequential tests, Bayesian tests, multi-armed bandits).
  2. Tiny effects on small traffic. If your page gets 5,000 visits a month, test big changes (offer, headline, first screen), not small ones. Big lifts need far fewer visitors.
  3. Not running whole weeks. People buy differently on Monday and Saturday. Run at least one full week even if the number is reached sooner.

Bandits vs fixed tests

A classic A/B test splits traffic 50/50 until the sample is reached. A multi-armed bandit (often Thompson sampling) moves traffic toward the version that is winning while the test runs, so fewer visitors see the loser. The catch: a pure bandit can starve your original page, which makes the gain hard to prove. The fix is to keep a holdout share on the original no matter what.

I wrote more on that here: multi-armed bandit vs A/B testing.

Try it in the browser

If you just want the number without installing anything, there is an interactive version here: A/B test sample size calculator.

Disclosure: I build Luria, an AI split-testing tool, which is why I care about this. The math above is standard and works with any testing tool.

Top comments (0)