A small A/B test significance checker in Python, and when not to trust it

A significance checker answers one narrow question: if the two versions of a page were actually identical, how surprising would this difference in conversion rate be? That is useful, but it is easy to read more into the answer than it contains. Here is a short checker using only the Python standard library, followed by the situations where its output should not be trusted.

The checker

This is a two-sided, two-proportion z-test. You give it visitors and conversions for each variant.

from math import sqrt, erf

def normal_cdf(x):
    return 0.5 * (1 + erf(x / sqrt(2)))

def ab_test(conv_a, n_a, conv_b, n_b):
    p_a, p_b = conv_a / n_a, conv_b / n_b
    pooled = (conv_a + conv_b) / (n_a + n_b)
    se = sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
    if se == 0:
        raise ValueError("No variation in the data")
    z = (p_b - p_a) / se
    p_value = 2 * (1 - normal_cdf(abs(z)))
    return {"rate_a": p_a, "rate_b": p_b, "z": z, "p_value": p_value}

# Illustrative inputs only: conversions, visitors for A, then for B
print(ab_test(conv_a=40, n_a=1000, conv_b=55, n_b=1000))

rate_b - rate_a is the observed difference. The p_value is the probability of seeing a difference at least this large by chance alone, assuming there is no real difference. It is not the probability that B is better.

How to use the result

Decide three things before the test starts, and write them down:

1. The single metric that decides the winner.
2. The sample size or end date you will stop at.
3. The p-value threshold you will accept. Many teams use 0.05, but it is a convention, not a law.

Then run the checker once, at the planned end. If the p-value is below your threshold, you have evidence of a difference. If it is above, you have learned that this test could not tell the versions apart, which is not the same as proving they are equal.

When not to trust it

You looked early and stopped when it turned green. A p-value is only valid if you check it once, at the planned stopping point. Checking daily and stopping at the first good result inflates false positives badly. Decide the end date first.

You tested many things at once. Each extra variant or extra metric you compare gives chance another opportunity to produce a "winner". If you compare several variants or metrics, treat any single hit with suspicion, or apply a correction such as Bonferroni.

The sample is small. The z-test relies on an approximation that is poor when conversions are very few. As a rough rule, be cautious if either variant has only a handful of conversions, and use an exact test instead.

Traffic was not split cleanly. If you intended a 50/50 split and the visitor counts are noticeably uneven, something in the setup is wrong, such as redirects, caching or bot filtering. Fix that before reading any result. This is often called a sample ratio mismatch.

The test ran over an unusual period. A campaign launch, a public holiday or a Ramadan shift can change who visits. Run tests across at least one or two full business cycles so weekday and weekend behaviour are both included.

The difference is real but too small to matter. A tiny difference can be statistically significant on large traffic and still not justify the cost of changing the page. Look at the size of the change, not only the p-value.

A practical routine

• Write a one-line hypothesis and the decision metric before launch.
• Fix the end date and keep to it.
• Check the visitor split first, then the result.
• Record the outcome, including the tests that found nothing.
• Treat a significant result as a reason to confirm, for example by re-running or watching the metric after rollout, not as final proof.

Check that conversions are recorded correctly in both variants before you trust any number. If test design is where you are stuck, our conversion rate optimisation service page explains what we do.

Written with AI assistance for the Gorilla Vibe marketing team.