The cookie jar behind A/B testing: Fisher’s test to churn

How a neighbourhood shop connects Fisher’s exact test, Simpson’s paradox, survival analysis and interference.

AI assistance: An AI agent checked the calculations, ran the illustrative simulation and helped edit this article.

Last time, a jar held seven chocolate cookies and three plain ones, and we packed six into a box without putting any back. The interesting part was not choosing a formula. It was figuring out which boxes could exist at all.

This article extends that cookie-jar experiment into A/B testing. The same hypergeometric counting appears in several familiar tests, with different conditions attached. It is a useful thread to follow, though it does not cover every A/B-test metric or design.

The shop, households and outcomes below are invented examples. They are not customer data or evidence that a reminder, price or recommendation improves a real business.

The jar, in one breath

Two checks came out of the previous example:

• Support. With only three plain cookies, any box of six has at least three chocolate ones. The possible chocolate counts are {3, 4, 5, 6}.
• Mechanism. Drawing without replacement gives those counts probabilities 1/6, 1/2, 3/10 and 1/30. A binomial model of independent draws with replacement also sums to one, yet puts 7.047% on counts that cannot occur in the original experiment.

The habit was to picture how the selection happens before choosing a formula. In a randomized A/B test, we control one selection ourselves: who gets which treatment.

Sharma ji’s chits

Imagine Sharma ji’s kirana store at the corner of your lane. Twenty households buy their milk, bread and occasional packet of Maggi there. Lately a few have drifted to a delivery app.

His nephew suggests a WhatsApp reminder with free home delivery: “Your usual order is ready.” To test this bundle, Sharma ji writes the 20 household names on chits, drops them in a steel dabba, and pulls out 10 without looking. Those 10 get reminders and the delivery offer; the other 10 do not.

A month later, seven households have stopped coming:

Group	Stopped coming	Still coming
No reminder	6	4
Reminder and delivery offer	1	9
Total	7	13

Six versus one looks promising. Could the dabba produce that split even if the offer changed nobody’s outcome?

The dabba is the cookie jar

Suppose the offer changes nothing for any household. The seven households who left would have left whichever pile their chit landed in. This is a sharp no-effect null: it specifies each household’s outcome under either assignment.

The only randomness left is which ten chits Sharma ji pulled. Now the jar has 20 households, seven labelled “would leave,” and a box containing ten reminder assignments. Under this null, the number of leavers in the reminder group is hypergeometric.

Using SciPy’s parameter order, that is hypergeom(M=20, n=7, N=10). Its support is {0, …, 7}. Every one of those counts is feasible here.

The probability of exactly k reminder-group leavers is:

P(X = k) = C(7, k) × C(13, 10 − k) / C(20, 10)

There are 184,756 equally likely assignments. Counts zero and one occur in 286 and 5,005 assignments, respectively. So:

P(X ≤ 1) = (286 + 5,005) / 184,756 = 0.02864

For the probability-ordering two-sided Fisher test, we add all tables no more probable than the observed one. This balanced example is symmetric, so the other tail contains six and seven leavers. The total is p = 0.05728. Two-sided conventions need more care in asymmetric examples. SciPy’s Fisher documentation describes the counting rule.

Original figure from exact integer counts. The highlighted tails define this example’s two-sided Fisher p-value.

That is Fisher’s exact test, viewed through randomized assignment. It also has a conditional sampling interpretation; it is not restricted to randomized trials or small samples. Wikipedia’s overview is a useful entry point.

The p-value is a probability calculated under the null. It is not the probability that the offer does nothing. At a preselected two-sided 5% threshold, this result does not reject the null. It also does not establish that the offer has no effect.

There is a subtlety here. Exact tests have discrete p-values, so their rejection rate can fall below the nominal threshold. Under a different model—two independent binomial groups with ten households each—Boschloo’s and Barnard’s tests both give about 0.02278 with SciPy’s default two-sided settings. Those tests use a different reference set, rather than merely finding a better approximation to Fisher. See the Boschloo and Barnard documentation.

The symmetric two-sided mid-p adjustment gives 0.03019, but it gives up the usual guarantee of controlling the rejection rate at the nominal level for every null parameter value. It should not be presented as an interchangeable exact test. This research paper on mid-p values explains that distinction.

Okay, now we know why “which test?” needs an answer about the design and null hypothesis. We should choose the analysis and stopping rule before seeing the result, rather than shop among p-values or keep collecting until one crosses 0.05.

Why the shortcut crosses the line

An uncorrected Pearson chi-square test on the same table gives p = 0.01908. With Yates’ continuity correction, it gives 0.06076.

The uncorrected test uses a continuous approximation. Here the expected number of leavers in each arm is only 3.5, making that approximation questionable. Expected counts around five are a common diagnostic, not a universal boundary between valid and invalid analysis. SciPy’s chi-square documentation describes the approximation and correction.

The following tables were deliberately chosen near the 5% line. They are separate illustrations, not one experiment observed repeatedly:

Households per arm	Leavers: no reminder / reminder	Fisher p	Uncorrected chi-square p
10	6 / 1	0.05728	0.01908
100	40 / 27	0.07182	0.05147
1,000	400 / 358	0.05876	0.05289
10,000	4,000 / 3,864	0.05066	0.04898

The numerical gap shrinks in these examples. Even at 10,000 per arm, a rigid 0.05 decision still flips. Large total sample size alone does not guarantee good approximation: rare outcomes, dependence and the assignment design still matter.

One detail from the original jar survives the scale-up. For a fixed population of size M, marked fraction p, and N draws without replacement:

Var(X) = N × p × (1 − p) × (M − N) / (M − 1)

Here the finite-population factor is 10/19. The reminder-leaver count has variance about 1.20, compared with 2.275 for independent binomial draws with p = 7/20. This is the hypergeometric variance.

That comparison concerns one count with the total number of leavers fixed. It is not an instruction to multiply an ordinary two-proportion z-test variance by another correction factor. The two arms are complementary under the conditional jar model; their covariance matters too.

A pricing test with two kinds of shopper

Next Sharma ji tests a combo price on Maggi, bread and milk. He still draws chits uniformly within each time slot, but changes the allocation. In the morning, 80 of 100 shoppers see the combo price. In the evening, only 20 of 100 do.

The results look like this:

Slot	Combo buyers / offered	Regular-price buyers / offered
Morning	20 / 80 = 25%	4 / 20 = 20%
Evening	12 / 20 = 60%	44 / 80 = 55%
Pooled	32 / 100 = 32%	48 / 100 = 48%

Original figure from the illustrative counts above. The denominators explain the reversal.

The combo does slightly better in both slots, yet seems to lose when pooled. Fisher’s test on the pooled table even gives p = 0.03009.

This is Simpson’s paradox. Evening shoppers buy more, and most saw the regular price. Pooling gives the regular-price group a much larger share of that high-purchase crowd.

The pooled randomization story is wrong: Sharma ji drew from two dabbas at different ratios.

The Cochran–Mantel–Haenszel test keeps a jar per slot. Under the no-association null, the expected combo-buyer counts are 19.2 in the morning and 11.2 in the evening, with hypergeometric variances 2.94788 and 3.98222. The observed counts are 20 and 12.

Without a continuity correction:

CMH statistic = [(20 − 19.2) + (12 − 11.2)]² / (2.94788 + 3.98222) = 0.36940

Comparing that statistic with a chi-square distribution with one degree of freedom gives p = 0.54333. The Mantel–Haenszel common odds-ratio estimate is 1.27027, favouring the combo. The data provide weak evidence about an effect; they do not prove a benefit. The per-slot odds ratios are 1.33 and 1.23. Statsmodels’ stratified-table documentation and the CMH overview describe this approach.

On an app, assignment ratios can differ across days, cities or devices. Ramping traffic can create this trap when outcome rates also change over time. It does not automatically do so. Adjustment should follow the design and use suitable pre-treatment strata, rather than blindly condition on anything measured after treatment.

Churn is about when: a stack of jars

“Did they leave by week 12?” treats departures in week one and week eleven alike. Survival analysis keeps the timing. A household still shopping at the end contributes information up to that point; its later departure time is right-censored, rather than entirely unknown.

Suppose a new test follows ten households per arm for twelve weeks. Reminder households leave in weeks 4, 9 and 11. Control households leave in weeks 1, 2, 2, 3, 5, 6 and 9. All others are observed still shopping and censored at week 12. In this simplified story, leaving is a single recorded event; a real churn definition needs to handle inactivity and returning customers explicitly.

Original Kaplan–Meier figure, ten households per arm. Plus signs mark administrative censoring at week 12. No confidence bands are shown; this small example is for explaining the calculation.

The log-rank test has a familiar piece inside it. At each departure time, freeze everyone still at risk. Conditional on that risk set and the number leaving, the null allocates departures between groups hypergeometrically.

At week one, ten of twenty households are in the reminder group, so one departure gives an expected reminder count of 0.5. At week two, ten of nineteen remaining households have reminders; two departures give an expected count of 20/19, or about 1.053.

Across the eight distinct departure times, the reminder group has 3 observed departures, 6.12061 expected, and a summed variance of 2.25598:

χ² = (3 − 6.12061)² / 2.25598 = 4.31663

The usual log-rank approximation gives p = 0.03774. The risk sets overlap and evolve; these are not eight independent experiments. Their conditional hypergeometric contributions produce the standard log-rank statistic. Its reported p-value is asymptotic, so this tiny example is not an exact small-sample guarantee. SciPy documents the statistic and approximation.

The jar connection is real, with conditions attached. Censoring must be suitably non-informative, and log-rank is particularly sensitive to proportional-hazard alternatives. Crossing survival curves can make a single log-rank result unhelpful. Kaplan–Meier’s overview explains the curve; the survival package documentation connects log-rank and Cox regression. Cox models also use risk sets and can include covariates, but are not simply the same test with a new name.

When the boxes touch: the last packet of milk

So far, we have treated each household’s outcome as depending on its own assignment. Shared stock can break that assumption.

Picture a dark store with 30 packets of milk and 60 customers. Half get a nudge. Each customer wants at most one packet, independently: with probability 0.7 if nudged and 0.4 otherwise. There is no restocking.

For a deliberately strong illustrative scenario, all nudged customers arrive before all control customers. This is an explicit modelling choice, not a measured description of an app.

An independent simulation of 20,000 mornings gives:

• Mixed A/B comparison: nudged customers are 41.5 percentage points more likely to receive milk than controls.
• Whole-store launch comparison: nudging everyone instead of nobody increases the share receiving milk by 10.1 percentage points.

Original figure from the explicitly specified simulation, seed 20261018. These are distributions across simulated mornings, not confidence intervals for real customers. The comparisons answer different questions.

Why the gap? In the mixed test, nudged customers can take milk that controls would otherwise receive. In the full launch, there is still only enough stock for half the customers. The mean quantities served are about 23.93 packets with nobody nudged and 30.00 with everyone nudged.

This is interference: one customer’s assignment can change another customer’s outcome. It violates the no-interference part of SUTVA, which also includes a requirement about well-defined treatment versions. A more exact p-value cannot turn this mixed-arm contrast into the effect of a whole-store launch.

The 41.5-point result depends heavily on the chosen arrival rule. Different stock, demand or arrival assumptions can change the magnitude and even the pattern of bias. It is a counterexample to a naive interpretation, not a forecast.

Possible designs include randomizing whole stores or geographic clusters, provided spillovers between them are limited, or using randomized switchback time blocks. Switchbacks need to account for time trends, dependence and carryover: simply alternating every hour does not guarantee a clean comparison. DoorDash’s engineering account explains why the unit of analysis matters; research on switchbacks examines carryover bias.

For search and recommendation models, interleaving is another useful evaluation idea: blend two rankers’ results and attribute clicks to their sources. It estimates a relative ranking preference under its own assumptions. It is not a general cure for shared-stock interference or a substitute for measuring a full rollout’s impact. Microsoft’s original interleaving research gives the distinction.

The electrician’s habit

When a room’s light flickers, a good electrician traces the wiring: which switch feeds which point, and what is shared with the next room.

A statistical test deserves the same habit. Trace the assignment and outcome process:

• Fixed group sizes and a binary outcome lead us to the conditional jar behind Fisher’s test.
• Different assignment ratios across strata require keeping those strata in view.
• Event times and censoring lead to evolving risk sets.
• Shared stock or other spillovers require asking what effect the randomization actually identifies.

That is the useful bridge from the cookie jar to evaluating models and product changes. First picture how the selection happens, write down what it permits, then choose the formula.

What was checked

The illustrative calculations were independently checked with Python 3.13.9, SciPy 1.18.1 and NumPy 2.5.4. No live A/B test or trained ML model is reported here.

• Fisher: exact integer counts were 286, 5,005, 27,027, 60,060, 60,060, 27,027, 5,005 and 286 out of 184,756; these matched scipy.stats.fisher_exact within absolute tolerance 1e-12.
• Alternatives: boschloo_exact([[6, 1], [4, 9]]) and barnard_exact([[6, 1], [4, 9]]) gave 0.02278. Here the columns are the ten-household arms. Fisher is unchanged by transposition; these unconditional tests generally are not.
• Chi-square: chi2_contingency gave 0.01908 without the correction and 0.06076 with it. All four scale illustrations were recomputed.
• CMH: per-slot hypergeometric means and variances gave statistic 0.36940, approximate p = 0.54333 and common odds ratio 1.27027, without a continuity correction.
• Log-rank: summing the stated risk-set contributions matched scipy.stats.logrank at p = 0.03774 within absolute tolerance 1e-12. SciPy returns the signed square root of the chi-square statistic above.
• Milk: a fresh simulation used seed 20261018 and 20,000 independent mornings. Shared uniform demand draws coupled the all-nudged and nobody-nudged scenarios. The middle 95% of morning differences spanned 13.3 to 73.3 points for mixed A/B and 0 to 21.7 points for launch. Exact binomial-demand summation gave mean differences 41.420 and 10.157 points, consistent with Monte Carlo error. These are properties of the specified toy model.