This was my final project for IBM's Supervised Machine Learning: Classification course: predict which telecom customers will leave, using the IBM Telco Customer Churn dataset (7,043 customers, 26.5% of whom churned). The modelling was the easy part. The interesting decision came at the end, when the best three models turned out to be indistinguishable.
A fair comparison first
Every model family was grid-searched on the same five stratified folds of the same 80/20 split, scored by ROC AUC, and then evaluated once on the held-out test set with a 95% confidence interval.
| Model | CV AUC | Test AUC | 95% CI |
|---|---|---|---|
| Gradient boosting | 0.849 | 0.846 | 0.825–0.868 |
| Random forest | 0.849 | 0.843 | 0.821–0.865 |
| Logistic regression (L2, C = 0.1) | 0.844 | 0.843 | 0.821–0.865 |
| SVM, RBF kernel | 0.844 | 0.843 | 0.819–0.865 |
| k-nearest neighbours | 0.838 | 0.832 | 0.808–0.854 |
| Decision tree (pruned) | 0.839 | 0.831 | 0.807–0.854 |
Gradient boosting has the highest number, but its lead over logistic regression is 0.003 AUC, well inside overlapping confidence intervals. When the accuracy is a tie, I pick on the things that matter after deployment: can the business read the model, and is its ranking stable?
Why logistic regression got the job
Logistic regression gives odds ratios that a retention team can act on directly:
| Feature | Odds ratio | What it means |
|---|---|---|
| Two-year contract | 0.33 | a third of the churn odds of month-to-month |
| Fibre-optic internet | 2.43 | 2.4 times the odds of DSL |
| Tenure (per ~24 months) | 0.45 | churn risk halves with every two years |
| Tech support add-on | 0.68 | support customers stay more often |
I checked whether the simple model was leaving signal on the table: permutation importance on the gradient-boosting model ranks the same drivers (tenure, contract, fibre) at the top. The model is also stable: eight variations (L1 or L2, C from 0.1 to 10, with and without class weights) all land within 0.001 AUC of each other.
Two details that are easy to miss
Collinearity. MonthlyCharges is almost entirely determined by the service flags (R² = 0.999), and TotalCharges is essentially tenure times monthly charges (r = 0.9996). Keeping them makes coefficients unreadable without adding information, so both were dropped, leaving 22 model features.
The threshold. The default 0.5 threshold is rarely the right one for an imbalanced problem. I set it at the point on the precision–recall curve that maximises F1 (0.35 for logistic regression), which gives precision 0.58 and recall 0.70. Class weighting does something similar: class_weight="balanced" lifts recall from 0.52 to 0.79 at the default threshold, at a cost in precision.
What it means for a retention campaign
A churn model is only useful if it changes who gets a call. Ranking customers by predicted probability:
- contacting the top 20% reaches 50% of the churners and 56% of the revenue at risk;
- the top 30% reaches 66% of the churners and 71% of that revenue.
Limits
The dataset is a single snapshot with no time dimension, so the model predicts who has churned rather than who will churn next quarter. The threshold ignores the cost of contacting a customer, which a real campaign would put in.
- Code, notebook and the 10-page report: github.com/engomaressam/telco-churn-classification
- Kaggle notebook: Telco churn: 6 models compared
- Model: omaressamrme/telco-churn-logistic-regression
Omar Essam El Sherbiny. Head of Data at WalaPlus. Data platforms on Google Cloud, data governance, applied ML. More at omaressam.web.app.
Top comments (0)