DEV Community: Mathew Kariuki

How to Analyze an A/B Test in Python (Chi-Square, Effect Size, CI)

Mathew Kariuki — Tue, 20 Oct 2026 13:00:00 +0000

Hi, I'm Mathew. Most A/B test tutorials end with "variant B won!". Real life is messier, and the most useful skill is knowing when the data does not support a conclusion. Let's analyze a real experiment and practice reading it honestly.

The experiment

A mobile game (the well-known Cookie Cats test) moved its progression gate, a point where players hit a pause, from level 30 (gate_30) to level 40 (gate_40). Players were randomly assigned. The question is whether the move changes retention.

One important note: the lesson works with a sample of 2,256 players, so the numbers below belong to that sample.

import pandas as pd
import numpy as np
from scipy import stats

ab_test = pd.read_csv("ab_test_cookie_cats_raw.csv")
ab_test["version"].value_counts()
# gate_40    1135
# gate_30    1121

Columns: userid, version, sum_gamerounds, retention_1 and retention_7 (booleans: did the player come back after 1 day / 7 days).

Always check group balance first. The split is 50.31% vs 49.69%, which is what we'd expect from random assignment.

Retention by group

Because retention is a True/False column, its mean is the retention rate:

ab_test.groupby("version")["retention_1"].mean().round(4) * 100
# gate_30    43.44
# gate_40    43.00

ab_test.groupby("version")["retention_7"].mean().round(4) * 100
# gate_30    19.18
# gate_40    17.71

Day-1 retention is almost identical. Day-7 retention is 1.47 points lower for gate_40. Tempting to call it, right? Not yet.

Is the gap bigger than chance?

For a yes/no outcome across two groups, the chi-square test of independence is the standard tool. Build a contingency table with crosstab:

table_7 = pd.crosstab(ab_test["version"], ab_test["retention_7"])
chi2, p_value, dof, expected = stats.chi2_contingency(table_7)
round(p_value, 4)   # 0.3976

Day 1 gives p = 0.8633 and day 7 gives p = 0.3976. With alpha = 0.05, neither is significant. A gap this size could easily appear by random chance in a sample this small.

Engagement needs a different test

Total game rounds is skewed. The mean says 54.83 rounds for gate_30 and 44.70 for gate_40, but the medians are both 15. The max is 1,906 rounds and 97 players played zero. A few heavy players drag the means around, so a t-test on raw means is a poor fit. Use a non-parametric test:

g30 = ab_test.loc[ab_test["version"] == "gate_30", "sum_gamerounds"]
g40 = ab_test.loc[ab_test["version"] == "gate_40", "sum_gamerounds"]

stat, p = stats.mannwhitneyu(g30, g40, alternative="two-sided")
round(p, 4)   # 0.487

Not significant either. The mean gap was outlier noise.

Effect size and confidence interval

A p-value says whether, not how much. Report the size of the gap in both forms:

gap_pts = 19.18 - 17.71                       # 1.47 percentage points
rel_lift = (19.18 / 17.71 - 1) * 100          # 8.3 percent relative

These tell different stories: "1.47 points" sounds tiny, "8.3% relative" sounds big. Both describe the same observed gap, so state both.

Then a 95% confidence interval on the difference in proportions:

p1, p2 = 0.1918, 0.1771
n1, n2 = 1121, 1135

se = np.sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)
low  = (p1 - p2 - 1.96 * se) * 100
high = (p1 - p2 + 1.96 * se) * 100
# (-1.73, 4.67)

The interval runs from -1.73 to +4.67 points. It includes zero, which is the same message as the p-value, but more informative: the true effect could be a small harm to gate_30, nothing, or a few points of benefit.

Looking inside the segments

The notebook also splits by engagement. Among the top 25% most active players, day-7 retention is 60.58% (gate_30) vs 52.46% (gate_40). Among the bottom 25%, it's 1.63% vs 1.52%. So the observed gap sits almost entirely among heavy players. That's a hypothesis worth testing with more data, not a proven result: the notebook doesn't run significance tests on these segments, and slicing a sample into smaller pieces makes noise easier to mistake for signal. Engagement and retention are strongly related overall (correlation 0.527), which is why engagement is a sensible place to look.

A word on the business framing

The notebook includes a "what if" calculation: with 1,000,000 weekly active players, a 1.47-point gap would equal about 14,700 more retained players. That's a useful way to size the stakes, but it only holds if the effect is real, and here the confidence interval includes zero. Present it as a "what this might be worth if confirmed" figure, never as a forecast.

My honest takeaway

The point estimates lean toward keeping the gate at level 30 (19.18% vs 17.71% day-7 retention).
But in this sample the difference is not statistically significant (p = 0.3976, CI -1.73 to +4.67 points).
The sensible next step is more data, or a pre-planned larger test, not a confident announcement.

Learning to say "the evidence isn't conclusive yet" is a real analyst skill. Stakeholders trust people who don't oversell.

A checklist you can reuse

Here's the routine I'd follow on any A/B test:

Confirm the groups are balanced in size.
Compare the key rates, and check both short and long horizons (here day 1 and day 7).
Pick the right test for the metric: chi-square for yes/no outcomes, Mann-Whitney for skewed counts.
Report effect size in absolute and relative terms.
Add a confidence interval, because it shows the range of plausible effects.
Slice by segment only to generate hypotheses, and say so.
Write the conclusion in plain English, including what you don't know.

Following the same steps every time keeps you honest, and it makes your analysis easy for colleagues to review.

Get the full notebook, dataset and quiz

The full lesson includes the charts, the retention funnel, conditional retention, sanity checks, the saved summary table, and a quiz.

Full lesson: Marketing Campaign A/B Test Analysis
Watch the video walkthrough: https://www.youtube.com/watch?v=uZWLJbzC7m0
Subscribe for more real-data tutorials: https://www.youtube.com/@KarMat-Analytics?sub_confirmation=1

It's all free. Thanks for reading!

pandas groupby and Aggregation: A Practical Guide

Mathew Kariuki — Tue, 13 Oct 2026 13:00:00 +0000

Hi, I'm Mathew. If I had to pick one pandas tool that turns raw rows into real answers, it's groupby. "Revenue per country", "orders per customer", "average per month": they're all the same idea. Let's walk through it with a cleaned slice of a real online retail dataset.

Quick note on size: this lesson uses a small extract (1,394 cleaned rows, all from December 2010), so the focus is on the technique, not on big business conclusions.

Setup

import pandas as pd

df = pd.read_csv("online_retail_sample.csv")
df = df.dropna(subset=["CustomerID"]).drop_duplicates()
df = df[(df["Quantity"] > 0) & (df["UnitPrice"] > 0)]
df["TotalPrice"] = df["Quantity"] * df["UnitPrice"]

We drop rows with no customer, drop duplicates, remove returns and zero prices, then create a TotalPrice column for each line.

1. Split, apply, combine

groupby follows the split-apply-combine pattern. The key thing to know: nothing is computed until you aggregate.

by_country = df.groupby("Country")
print(type(by_country))   # DataFrameGroupBy, no numbers yet

revenue_by_country = (
    df.groupby("Country")["TotalPrice"]
      .sum()
      .sort_values(ascending=False)
)

The top of the result:

Country
United Kingdom    28259.30
Norway             1919.14
France              855.86
EIRE                555.38
Australia           358.25

Read it as a recipe: group by Country, pick TotalPrice, sum it, sort it.

2. Several aggregations at once

Pass a list to agg:

df.groupby("Country")["TotalPrice"].agg(["sum", "mean", "count"])

That works, but the column names are generic. Named aggregation is the version I use daily, because you choose the output column names:

named = df.groupby("Country").agg(
    total_revenue=("TotalPrice", "sum"),
    avg_line_total=("TotalPrice", "mean"),
    num_invoices=("InvoiceNo", "nunique"),
)
named.sort_values("total_revenue", ascending=False).head(5)

Country	total_revenue	avg_line_total	num_invoices
United Kingdom	28259.30	22.63	76
Norway	1919.14	26.29	1
France	855.86	42.79	1
EIRE	555.38	26.45	2
Australia	358.25	25.59	1

Notice the three aggregations use different source columns. Also, be careful with the word "average": mean of TotalPrice is the average line item, not the average invoice. (The notebook calls this column avg_order_value; I renamed it here because each invoice has several lines.) To get a true average order value, you'd sum per invoice first.

3. Grouping by more than one column

Pass a list of columns and you get a multi-level index:

df["InvoiceDate"] = pd.to_datetime(df["InvoiceDate"])
df["Month"] = df["InvoiceDate"].dt.to_period("M")

monthly_country = df.groupby(["Month", "Country"])["TotalPrice"].sum()

That's a long, stacked result. To make it a readable grid, unstack pivots the inner level into columns:

monthly_country.unstack().fillna(0).round(2)

Since this extract only covers one month (2010-12), the grid is a single row. With a full year you'd get one row per month and one column per country, ready for a chart.

4. transform: put the group stat back on every row

agg collapses each group to one row. transform does the same calculation but returns a result the same length as your original data, so you can compare each row with its own group:

df["CountryAvgOrder"] = df.groupby("Country")["TotalPrice"].transform("mean")
df["AboveCountryAvg"] = df["TotalPrice"] > df["CountryAvgOrder"]

df["AboveCountryAvg"].value_counts()
# False    1069
# True      325

So 325 of the 1,394 lines are worth more than their own country's average. The pattern "compare each row with its group" is useful for flagging outliers, calculating share of group total, or filling missing values with a group mean.

5. pivot_table and crosstab

pivot_table is groupby plus unstack in one call, spreadsheet style:

pivot = df.pivot_table(
    values="TotalPrice", index="Month", columns="Country",
    aggfunc="sum", fill_value=0,
)

crosstab is the one to reach for when you want counts of combinations:

top_countries = revenue_by_country.head(4).index
subset = df[df["Country"].isin(top_countries)]
pd.crosstab(subset["Country"], subset["Month"])

In this extract that shows 1,249 United Kingdom lines, 73 Norway, 21 EIRE and 20 France for 2010-12.

Which tool should I use?

One summary number per group: groupby(...).agg(...)
Clean column names for several summaries: named aggregation
Group stat on every original row: transform
Spreadsheet-style grid of a value: pivot_table
Counts of category combinations: crosstab

Common mistakes

Forgetting to aggregate. A bare groupby object prints nothing useful.
Reading a mean without its count. Netherlands shows a mean of 96.30, but it's based on only 2 rows. Always look at count next to mean.
Mixing up line items and orders, as above.

Try it yourself

The best way to lock this in is to change one thing at a time. Swap Country for StockCode or CustomerID and rerun the same named aggregation. Add a second metric such as max or std. Then group by two columns and unstack the result. Each small change shows you how the pieces fit together, and that's far more memorable than reading about it. Once groupby feels natural, nearly every later topic, from cohort analysis to feature engineering, becomes a variation on the same split-apply-combine idea.

A final tip: when a result looks surprising, check count and nunique before you believe it. Most "interesting" findings in small groups turn out to be small samples.

Get the full notebook, dataset and quiz

The lesson on the site has the full runnable notebook, the dataset, and a quiz so you can check you've really got it.

Full lesson: Pandas groupby and Aggregation
Watch the video walkthrough: https://www.youtube.com/watch?v=Vp3kePFGE-w
Subscribe for the next lesson in the series: https://www.youtube.com/@KarMat-Analytics?sub_confirmation=1

Everything is free. See you in the next one!

Customer Segmentation with RFM Analysis in Python (pandas)

Mathew Kariuki — Sun, 04 Oct 2026 14:23:35 +0000

Hi, I'm Mathew. If you've ever wondered "who are our best customers?", this one is for you. We'll answer it with RFM analysis, one of the most practical techniques in marketing analytics, using nothing but pandas.

RFM scores every customer on three questions:

Recency: how many days since they last bought?
Frequency: how many orders have they placed?
Monetary: how much have they spent in total?

We'll use a real, cleaned UK online retail dataset: 391,150 transaction rows, with a Revenue column already calculated.

Step 1: Pick a snapshot date

Recency needs a "today". Rather than using the real today (which would make everyone look inactive), we use the day after the last transaction in the data:

import pandas as pd

clean = pd.read_csv("online_retail_clean.csv", parse_dates=["InvoiceDate"])

snapshot_date = clean["InvoiceDate"].max() + pd.Timedelta(days=1)
# Timestamp('2011-12-10 12:50:00')

Step 2: Build R, F and M with groupby

Each metric is one groupby on CustomerID:

recency = clean.groupby("CustomerID")["InvoiceDate"].max()
recency = (snapshot_date - recency).dt.days

frequency = clean.groupby("CustomerID")["InvoiceNo"].nunique()
monetary = clean.groupby("CustomerID")["Revenue"].sum()

rfm = pd.DataFrame(
    {"Recency": recency, "Frequency": frequency, "Monetary": monetary}
).reset_index()

That gives 4,334 customers with no missing values. The distributions already tell a story. The median customer last ordered 51 days ago and has placed 2 orders, while the biggest customer has 206 orders. Monetary value ranges from 3.75 to 279,138.02, with a median of 662.57 but a mean of 2,015.97. A few huge customers pull the average way up.

Step 3: Score each metric 1 to 4

We split each metric into quartiles with pd.qcut. Two details matter:

Recency is reversed. Fewer days since the last order is better, so the labels run [4, 3, 2, 1].
Frequency has lots of ties (plenty of customers have exactly 1 order), so qcut can't find clean edges. Ranking first with method="first" breaks the ties.

rfm["R_score"] = pd.qcut(rfm["Recency"], 4, labels=[4, 3, 2, 1]).astype(int)
rfm["F_score"] = pd.qcut(rfm["Frequency"].rank(method="first"), 4,
                         labels=[1, 2, 3, 4]).astype(int)
rfm["M_score"] = pd.qcut(rfm["Monetary"], 4, labels=[1, 2, 3, 4]).astype(int)

Each score now holds roughly 1,080 customers, as quartiles should. The recency cut points were 1, 18, 51, 143 and 374 days, so "R = 4" means "ordered within the last 18 days".

Step 4: Turn scores into named segments

You could just add the three scores together, but a total hides the shape. A customer scoring 4-1-1 and one scoring 2-2-2 both total 6, and you'd treat them very differently. Rules on the individual scores are more useful:

def segment_customer(row):
    if row["R_score"] >= 3 and row["F_score"] >= 3 and row["M_score"] >= 3:
        return "Champions"
    elif row["R_score"] >= 3 and row["F_score"] >= 2:
        return "Loyal Customers"
    elif row["R_score"] >= 3:
        return "New Customers"
    elif row["R_score"] == 2:
        return "At Risk"
    else:
        return "Lost"

rfm["Segment"] = rfm.apply(segment_customer, axis=1)

The order of the if branches matters: the first match wins, so the strictest rule goes first.

Step 5: The interesting part, customers vs revenue

segment_revenue = rfm.groupby("Segment")["Monetary"].sum()
customer_share = (rfm["Segment"].value_counts(normalize=True) * 100).round(1)
revenue_share = (segment_revenue / segment_revenue.sum() * 100).round(1)

pd.DataFrame({"Customer %": customer_share, "Revenue %": revenue_share})

Segment	Customer %	Revenue %
Champions	30.3	73.0
At Risk	24.8	12.3
Lost	24.9	8.0
Loyal Customers	14.0	5.7
New Customers	6.0	1.0

Champions are about 30% of customers but 73% of revenue. Their average profile is 17 days since last order, 9.3 orders and 4,847.80 total spend. Compare that with the Lost group: 247.8 days of silence, 1.6 orders and 645.50 spend.

What to do with this

Champions: protect them. Early access, thank-you notes, anything that keeps them from drifting.
At Risk: this is the group I'd look at first. They're about 12% of revenue, and the top of this list contains customers who spent more than 11,000 but haven't ordered for 56 to 114 days. A win-back campaign here is cheap and targeted.
New Customers: average 1.0 orders. The goal is a second purchase.
Lost: lowest priority, but worth a cheap automated email.

A few sanity checks worth running

Good analysts check their own work. The notebook confirms that the RFM table has one row per customer, that every customer has a segment, and that no Champion has only a single order (0 of them). It also checks that the score means something: average spend rises steadily from 162.40 at a combined score of 3 to 8,826.60 at 12. Frequency and Monetary are correlated at 0.55, which makes sense: people who order more spend more.

One caveat: quartile scoring is relative. These segments describe this dataset's customers compared with each other, not an absolute standard of "good".

Why this beats guessing

Before RFM, a lot of teams segment by gut feel: "big spenders" or "recent buyers". The trouble is that each of those looks at one dimension. RFM forces three views at once, which is why a customer who spent a lot two years ago (high M, terrible R) doesn't get mistaken for an active one. Because it's built from just three groupby calls and qcut, you can rebuild it in minutes on your own order data, as long as you have a customer ID, an order ID, a date and an amount. Try swapping in your own thresholds in segment_customer and see how the segment sizes move. That experiment alone teaches you a lot about your customers.

Get the full notebook, dataset and quiz

This article is the condensed version. The full lesson also has the charts (segment sizes and a recency vs spend scatter), the top Champions and At Risk lists, the country breakdown, and a quiz to test yourself.

Full lesson: Customer Segmentation with RFM Analysis
Watch the video walkthrough: https://www.youtube.com/watch?v=AvXkk9ltEsU
Subscribe so you don't miss the next one: https://www.youtube.com/@KarMat-Analytics?sub_confirmation=1

Everything on the site is free. Happy analysing!