You shipped a variant, the dashboard shows a lift, and someone asks "is it significant?". Here is the minimum statistics you need to answer honestly, plus free calculators to check your numbers.
Before the test: sample size
Decide how many users each variant needs before you start, otherwise you will be tempted to stop as soon as the graph looks good. Plug your baseline conversion rate, the smallest lift you care about, and your desired confidence into a sample size calculator.
Describe the data
Means hide a lot. Look at the spread too:
- Mean, median, mode, range for a quick summary.
- Standard deviation to see how noisy the metric is.
After the test: is the difference real?
- A p-value calculator tells you how surprising the result would be if there were no real difference. Below 0.05 is the common threshold, but it is a convention, not a law.
- A confidence interval calculator is often more useful: "the lift is between +0.4% and +2.1%" says far more than "p = 0.03".
Common mistakes
- Peeking daily and stopping early.
- Testing ten metrics and reporting the one that "won".
- Ignoring practical significance – a statistically real 0.1% lift may not be worth maintaining.
Run the numbers, write down the interval, and decide with the business context in mind.
Top comments (0)