The quarterly retention review had a good slide. The churn model's top decile had been sent a 20% discount offer, and only 18% of those customers left, against the 31% the model had predicted for them. Thirteen points, at a few thousand customers, came to a satisfying number in saved revenue.
Then someone asked the question that ends most of these meetings: "Compared to what?"
The 31% was the model's forecast for those customers if nothing changed. But the model was trained on last year's customers, the offer went out in a quarter with a price change and a new competitor, and nobody had held back a group of top-decile customers who got no offer. So nobody could say whether the discount saved those customers, whether they would have stayed anyway, or whether it pushed some of them out the door. The model was fine. The program had no way to show it had done anything.
This example is invented, but the design is the standard one: score everyone for churn risk, treat the riskiest, report the result against the model's own forecast. It has a gap in it that a better model can't close.
Two questions that sound like one
A churn model answers: if we do nothing, how likely is this customer to leave? That's a prediction about one possible future.
A retention program needs a different answer: how much does contacting this customer change whether they leave? That's a comparison between two futures, one where you act and one where you don't, and for any single customer you only ever see one of them. It's a causal question, and it has its own name in marketing analytics: uplift.
The two answers don't have to line up. A customer can be at high risk and impossible to keep, because they've already signed with a competitor. A customer can be at low risk and very easy to influence. And some customers are at low risk until you contact them.
Eva Ascarza (2018) tested this directly. Combining two field experiments, at a wireless provider and a membership organisation, with machine learning, she showed that "customers identified as having the highest risk of churning are not necessarily the best targets for proactive churn programs." Her conclusion is blunt: retention programs "are sometimes futile not because firms offer the wrong incentives but because they do not apply the right targeting rules." Targeting on sensitivity to the offer, regardless of risk, was "significantly more effective than the standard practice of targeting customers with the highest risk of churning."
Some customers leave because you called
The uncomfortable part is that a retention contact can cause churn. A call about a contract renewal reminds a customer that they're out of contract. A discount tells a customer who hadn't been thinking about price that they may be paying too much.
Radcliffe and Surry's review of real-world uplift modelling (2011) includes a case from a mobile operator: "A mobile churn reduction initiative actually increased churn from 9% to 10%." With an uplift model, the operator found a 30% subsegment where the campaign helped. Targeting only that group "reduced overall churn from 9% to under 8%, while reducing spend by 70%." The same paper says this kind of backfire is common in retention: it is "not uncommon, particularly in the area of retention, for uplift modelling to deliver more value by identifying populations where negative effects are prevalent" than by ranking the rest. They list when it tends to happen: interventions that are intrusive, and customers who "are already unhappy with the supplier."
Look at that list again with a churn model in mind. Unhappy customers are exactly who a churn model puts at the top.
What a randomised holdout shows that the model can't
Public retention experiments with a random control group are rare, but there's a well-known randomised marketing test with the same structure. Kevin Hillstrom's MineThatData e-mail dataset has 64,000 customers. A third "were randomly chosen to receive an e-mail campaign featuring Womens merchandise," a third got a men's e-mail and a third "were randomly chosen to not receive an e-mail campaign." Visits were tracked for two weeks.
I treated it as a retention problem: a customer who doesn't visit has lapsed, the women's e-mail is the retention action, and the no-e-mail group is the holdout. Then I built the two kinds of model on half the data and compared them on the other half, ten times with different random splits:
- A risk model, trained only on holdout customers, that predicts the chance of lapsing if nobody e-mails them. That's what a churn model trained on history estimates.
- A simple uplift model: one model for the e-mailed customers, one for the holdout, and the difference between their predicted visit rates.
Because the e-mail was assigned at random, the effect inside any group of test customers can be measured directly: their visit rate if e-mailed, minus their visit rate if not. No model is involved in that number.
First, the risk model works. It ranks lapse risk better than chance (AUC 0.63 on holdout customers it hadn't seen), and the gap between its deciles is real: 97.7% of the top decile lapsed without an e-mail, against 81.4% of the bottom decile. Here is what the e-mail did in each decile, averaged over the ten splits:
| Risk decile | Lapsed, no e-mail | Lapsed, e-mailed | Extra visits per 1,000 e-mailed |
|---|---|---|---|
| 1 (highest risk) | 97.7% | 92.7% | 49 |
| 2 | 93.9% | 89.2% | 47 |
| 3 | 92.5% | 87.8% | 46 |
| 4 | 91.0% | 86.8% | 43 |
| 5 | 90.0% | 85.3% | 47 |
| 6 | 88.2% | 84.3% | 39 |
| 7 | 88.1% | 83.4% | 47 |
| 8 | 86.6% | 82.4% | 42 |
| 9 | 84.8% | 79.9% | 48 |
| 10 (lowest risk) | 81.4% | 76.5% | 49 |
The risk column falls steadily. The effect column doesn't move. Across the whole test half the e-mail added 45 visits per 1,000 customers, and every decile sits close to that. The model knows who is likely to lapse. It knows nothing about who the e-mail reaches.
That shows up as soon as you have a budget and can only contact some customers. Say the budget covers 10%. Each model scores every test customer and the top 10% are the ones it would e-mail. Because the real e-mail went out at random, about half of that group got it and half didn't. The gap in visit rate between those two halves is what the e-mail did for exactly the customers that model chooses.
Here is that gap, as extra visits per 1,000 customers e-mailed:
| Budget (share of customers e-mailed) | Chosen at random | Chosen by highest churn risk | Chosen by highest uplift |
|---|---|---|---|
| 10% | 45 | 49 (range 37–57) | 73 (range 54–93) |
| 20% | 45 | 48 (range 33–59) | 70 (range 55–82) |
| 30% | 45 | 47 (range 41–53) | 67 (range 59–77) |
| 50% | 45 | 46 (range 41–51) | 61 (range 56–66) |
Mean over ten random splits, with the lowest and highest split in brackets. "Chosen at random" is the effect across the whole test half, so it's the same at every budget.
Targeting by risk was about as good as picking customers at random. Targeting by uplift got 40–50% more out of every e-mail, and it beat risk targeting in all ten splits at 20% and 30%. The ranges are wide because each test half holds only about 2,000 customers per arm in a 20% slice, which is a point we'll come back to.
Two cautions. This is an e-mail campaign measured on visits, not a retention offer measured on cancellations, and this data shows no group that the e-mail drove away. The mobile operator above shows that group does exist in retention. The data is a free download and the whole comparison is under 100 lines of pandas and scikit-learn, with the raw results for all ten splits, so you can swap in a better model, or the men's e-mail as the action, and check.
The model you need can only be built from an experiment
Notice what made the uplift model possible: a group of comparable customers who randomly got no e-mail. Without that, the uplift model has nothing to learn the difference from, and the evaluation has nothing to measure it against.
Radcliffe and Surry put this first on their checklist for when uplift modelling is worth doing: "if no valid control group exists in the historical data, uplift modelling cannot normally be employed. (But one can be created for the next iteration of the campaign.)"
It's tempting to skip the experiment and fit an uplift model to past campaigns, where account managers chose who to call. A 2026 study suggests that's a poor bet. Pacella, Papadia and Giliberti compared uplift models with conventional baselines on the public Telco Customer Churn dataset in two setups: "a causally controlled semisynthetic scenario and an observational proxy scenario," where who got treated was confounded with who was likely to churn. Uplift models won in the controlled setup. In the second, "traditional baselines remain economically superior." They also found that "the highest-yielding causal policy frequently suffers from severe targeting instability": which customers it picked changed a lot between runs. That's the same instability visible in the ranges above.
The evidence that uplift modelling pays off at all comes from settings with randomised data. Devriendt, Berrevoets and Verbeke (2021) start from the observation that "the literature, however, does not provide conclusive empirical evidence that uplift modeling outperforms predictive modeling," then compare the two head to head on a financial-services retention campaign and find that "uplift models outperform predictive models and lead to improved profitability of retention campaigns." Companies running these programs at scale build on experiments too. An AI Council 2026 talk by Intuit's Avik Basu opens with "accurate predictions do not imply better decisions" and describes Intuit using uplift modelling "to enhance targeting and experimentation."
So the missing piece in most churn programs isn't a better model. It's an experiment.
The one question to ask about your retention program
When the last retention campaign went out, was a random group of eligible customers held back and given nothing?
If the answer is no, nobody can say what the campaign did, and the slide comparing actual churn to predicted churn doesn't change that. The prediction is a forecast for a world where nothing happened, made by a model trained on a different period. It isn't a measurement.
If the answer is yes, you can already do the analysis above with no new model: split the campaign by the churn model's deciles and compare treated with held-out customers in each one. A table like this is enough:
import pandas as pd
def effect_by_decile(df):
"""df: one row per eligible customer, with churn_score, held_out (bool), churned (0/1)."""
df = df.assign(decile=pd.qcut(df.churn_score.rank(method="first", ascending=False), 10,
labels=range(1, 11)))
rates = df.pivot_table(index="decile", columns="held_out", values="churned",
aggfunc=["mean", "size"], observed=True)
rates.columns = ["churn_contacted", "churn_held_out", "n_contacted", "n_held_out"]
rates["points_saved"] = 100 * (rates.churn_held_out - rates.churn_contacted)
return rates
A negative points_saved in the top deciles is the sign to look for. That's where your offer may be losing customers it was meant to keep. With a few hundred held-out customers per decile, a single decile can swing by several points on noise alone, so look for a pattern across neighbouring deciles, or bootstrap the table, before acting on one number.
Setting up the holdout for next quarter
A few things decide whether the holdout is useful:
- Hold out across every decile, not just the bottom ones. If you only hold back low-risk customers, you can't measure the effect where you spend the most.
- Randomise, and don't let anyone override it. An account manager who "just calls this one customer" from the holdout has turned it back into an observational comparison.
- Size it for the effect you expect, which is usually small. Detecting a drop from 10% to 9% churn with a standard two-sided test at 5% significance and 80% power takes about 13,500 customers in each group. Radcliffe and Surry's rule of thumb for modelling uplift, rather than just measuring it, is stricter: the overall uplift times the size of each group should be at least 500. A one-point uplift means at least 50,000 customers per group. If you're well below that, measure the average effect first and target later.
- Count the cost of the holdout honestly. It's the customers you might have saved by contacting them. If the campaign has never been measured, that cost buys you the only number that says whether the campaign works.
None of this replaces the churn model. Risk is still useful for sizing the problem, for forecasting revenue and for deciding when to act. What it can't do is pick who to act on. That's the same distinction SophiArch's Applied Machine Learning course draws in its interpretability lesson, where SHAP values on a churn model come with the warning that they "are not causal explanations. They describe the model's reasoning, not the world." A feature that pushes a customer's churn score up isn't a lever you can pull, and a high score isn't a reason to call.
What this doesn't show
The experiment above is one public dataset with a cheap action (an e-mail) and a soft outcome (a visit). The uplift model is the simplest kind, and better ones exist. The evidence that retention offers drive some customers away comes from published case studies and field experiments, not from my numbers. And uplift models are noisy: the ranges above, and the instability Pacella and colleagues report, are a reason to measure a targeting rule on a fresh holdout before trusting it, not a reason to skip the model.
What it does show is the gap. A model can rank risk well and still be no better than random at choosing who to contact. The only way to find out which situation you're in is to hold some customers back.
References
- Ascarza, E. (2018). Retention Futility: Targeting High-Risk Customers Might Be Ineffective. Journal of Marketing Research, 55(1), 80–98. Two field experiments showing that the highest-risk customers are not necessarily the best targets, and that targeting by sensitivity to the intervention reduces churn more.
- Radcliffe, N. J., & Surry, P. D. (2011). Real-World Uplift Modelling with Significance-Based Uplift Trees. Stochastic Solutions white paper / Portrait Technical Report TR-2011-1. Source of the mobile churn case (9% to 10%, then under 8%), negative effects in retention, and the control-group and volume checklist.
- Pacella, M., Papadia, G., & Giliberti, V. (2026). Stability-Aware Uplift Policy Selection for Customer Retention: From Predictive Scores to Actionable Segments. Applied Sciences, 16(10). Uplift models win under controlled treatment assignment, baselines win on confounded data, and the best-value policy is often unstable.
- Devriendt, F., Berrevoets, J., & Verbeke, W. (2021). Why you should stop predicting customer churn and start using uplift models. Information Sciences, 548, 497–515. Head-to-head comparison of churn prediction and uplift models on a financial-services retention campaign.
- Hillstrom, K. (2008). The MineThatData E-Mail Analytics and Data Mining Challenge. The randomised 64,000-customer e-mail test used for the experiment in this article.
- Basu, A. (2026). From Prediction to Uplift: Causal Modeling for Better Decisions. AI Council 2026. Practitioner talk on uplift modelling at Intuit.
- SophiArch. Applied Machine Learning. Course covering evaluation with statistical rigour, imbalanced data, calibration, SHAP interpretability and what it can't tell you, and production pipelines.
Top comments (0)