Hi, I'm Mathew. Most A/B test tutorials end with "variant B won!". Real life is messier, and the most useful skill is knowing when the data does not support a conclusion. Let's analyze a real experiment and practice reading it honestly.
The experiment
A mobile game (the well-known Cookie Cats test) moved its progression gate, a point where players hit a pause, from level 30 (gate_30) to level 40 (gate_40). Players were randomly assigned. The question is whether the move changes retention.
One important note: the lesson works with a sample of 2,256 players, so the numbers below belong to that sample.
import pandas as pd
import numpy as np
from scipy import stats
ab_test = pd.read_csv("ab_test_cookie_cats_raw.csv")
ab_test["version"].value_counts()
# gate_40 1135
# gate_30 1121
Columns: userid, version, sum_gamerounds, retention_1 and retention_7 (booleans: did the player come back after 1 day / 7 days).
Always check group balance first. The split is 50.31% vs 49.69%, which is what we'd expect from random assignment.
Retention by group
Because retention is a True/False column, its mean is the retention rate:
ab_test.groupby("version")["retention_1"].mean().round(4) * 100
# gate_30 43.44
# gate_40 43.00
ab_test.groupby("version")["retention_7"].mean().round(4) * 100
# gate_30 19.18
# gate_40 17.71
Day-1 retention is almost identical. Day-7 retention is 1.47 points lower for gate_40. Tempting to call it, right? Not yet.
Is the gap bigger than chance?
For a yes/no outcome across two groups, the chi-square test of independence is the standard tool. Build a contingency table with crosstab:
table_7 = pd.crosstab(ab_test["version"], ab_test["retention_7"])
chi2, p_value, dof, expected = stats.chi2_contingency(table_7)
round(p_value, 4) # 0.3976
Day 1 gives p = 0.8633 and day 7 gives p = 0.3976. With alpha = 0.05, neither is significant. A gap this size could easily appear by random chance in a sample this small.
Engagement needs a different test
Total game rounds is skewed. The mean says 54.83 rounds for gate_30 and 44.70 for gate_40, but the medians are both 15. The max is 1,906 rounds and 97 players played zero. A few heavy players drag the means around, so a t-test on raw means is a poor fit. Use a non-parametric test:
g30 = ab_test.loc[ab_test["version"] == "gate_30", "sum_gamerounds"]
g40 = ab_test.loc[ab_test["version"] == "gate_40", "sum_gamerounds"]
stat, p = stats.mannwhitneyu(g30, g40, alternative="two-sided")
round(p, 4) # 0.487
Not significant either. The mean gap was outlier noise.
Effect size and confidence interval
A p-value says whether, not how much. Report the size of the gap in both forms:
gap_pts = 19.18 - 17.71 # 1.47 percentage points
rel_lift = (19.18 / 17.71 - 1) * 100 # 8.3 percent relative
These tell different stories: "1.47 points" sounds tiny, "8.3% relative" sounds big. Both describe the same observed gap, so state both.
Then a 95% confidence interval on the difference in proportions:
p1, p2 = 0.1918, 0.1771
n1, n2 = 1121, 1135
se = np.sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)
low = (p1 - p2 - 1.96 * se) * 100
high = (p1 - p2 + 1.96 * se) * 100
# (-1.73, 4.67)
The interval runs from -1.73 to +4.67 points. It includes zero, which is the same message as the p-value, but more informative: the true effect could be a small harm to gate_30, nothing, or a few points of benefit.
Looking inside the segments
The notebook also splits by engagement. Among the top 25% most active players, day-7 retention is 60.58% (gate_30) vs 52.46% (gate_40). Among the bottom 25%, it's 1.63% vs 1.52%. So the observed gap sits almost entirely among heavy players. That's a hypothesis worth testing with more data, not a proven result: the notebook doesn't run significance tests on these segments, and slicing a sample into smaller pieces makes noise easier to mistake for signal. Engagement and retention are strongly related overall (correlation 0.527), which is why engagement is a sensible place to look.
A word on the business framing
The notebook includes a "what if" calculation: with 1,000,000 weekly active players, a 1.47-point gap would equal about 14,700 more retained players. That's a useful way to size the stakes, but it only holds if the effect is real, and here the confidence interval includes zero. Present it as a "what this might be worth if confirmed" figure, never as a forecast.
My honest takeaway
- The point estimates lean toward keeping the gate at level 30 (19.18% vs 17.71% day-7 retention).
- But in this sample the difference is not statistically significant (p = 0.3976, CI -1.73 to +4.67 points).
- The sensible next step is more data, or a pre-planned larger test, not a confident announcement.
Learning to say "the evidence isn't conclusive yet" is a real analyst skill. Stakeholders trust people who don't oversell.
A checklist you can reuse
Here's the routine I'd follow on any A/B test:
- Confirm the groups are balanced in size.
- Compare the key rates, and check both short and long horizons (here day 1 and day 7).
- Pick the right test for the metric: chi-square for yes/no outcomes, Mann-Whitney for skewed counts.
- Report effect size in absolute and relative terms.
- Add a confidence interval, because it shows the range of plausible effects.
- Slice by segment only to generate hypotheses, and say so.
- Write the conclusion in plain English, including what you don't know.
Following the same steps every time keeps you honest, and it makes your analysis easy for colleagues to review.
Get the full notebook, dataset and quiz
The full lesson includes the charts, the retention funnel, conditional retention, sanity checks, the saved summary table, and a quiz.
- Full lesson: Marketing Campaign A/B Test Analysis
- Watch the video walkthrough: https://www.youtube.com/watch?v=uZWLJbzC7m0
- Subscribe for more real-data tutorials: https://www.youtube.com/@KarMat-Analytics?sub_confirmation=1
It's all free. Thanks for reading!
Top comments (0)