My Master's capstone was a churn prediction model for retail banking customers 10,000 records, the goal being to flag customers likely to leave before they actually do. At the time I reported it as a success: 86.5% accuracy from a Random Forest, best of the three models I tried. Going back to it recently, I realized that number was quietly hiding a model that was catching less than half the customers it was actually supposed to catch. This is what I found re-examining it, and what changed once I fixed it.
The original setup
The dataset: 10,000 bank customers, with features like credit score, geography, age, account balance, number of products, and whether they're an active member, and a binary target did they leave (Exited) or not.
My original capstone code (still in the repo) trained three models Logistic Regression, Decision Tree, and Random Forest with standard scaling and label encoding, and compared them on accuracy and ROC AUC. Random Forest won on both: 86.5% accuracy, 0.847 AUC.
What I missed the first time
The dataset is imbalanced about 80% of customers stayed, 20% left. I didn't account for that in the original analysis, and accuracy on an imbalanced dataset is a genuinely misleading number: a model that just predicted "stays" for everyone would already score close to 80%.
Going back and breaking the original Random Forest's performance down by class tells the real story: it only caught 45.9% of customers who actually churned (recall), while correctly flagging most of the stayers. Precision on the customers it did flag as leaving was decent (78.6%) but that's cold comfort for a churn-prevention model whose entire point is catching people before they leave. Missing more than half of them defeats the purpose. The original Logistic Regression was far worse on this front: 14.3% recall it was essentially only ever predicting "stays."
Fixing it
I re-ran the same three models with class_weight='balanced', which reweights the loss function so the minority class (churners) isn't drowned out during training, and added XGBoost with scale_pos_weight set to the actual class ratio for the same reason.
Model Accuracy Precision Recall F1 AUC Logistic Regression (original) 80.5% 58.6% 14.3% 0.229 0.771 Random Forest (original) 86.5% 78.6% 45.9% 0.580 0.847 Logistic Regression (balanced) 70.8% 38.3% 71.7% 0.500 0.774 Random Forest (balanced) 85.8% 77.3% 42.8% 0.551 0.848 XGBoost (default) 85.4% 70.6% 48.4% 0.574 0.838 XGBoost (weighted) 82.1% 55.2% 62.2% 0.585 0.831
A few things stand out:
- Accuracy went down for almost every "improved" version. That's expected and, in this context, the right trade a model that's slightly less accurate overall but catches noticeably more actual churners is doing its actual job better, even though the headline number looks worse.
- Balanced Logistic Regression's recall jumped from 14.3% to 71.7%, a huge swing from essentially useless to genuinely useful at catching churners, at the real cost of many more false alarms (precision dropped to 38.3%).
- Weighted XGBoost gave the best overall balance (highest F1, 0.585), catching 62.2% of churners while keeping precision at a still-usable 55.2%.
Which of these is "best" isn't a purely statistical question it depends on the actual cost of a false alarm (an unnecessary retention offer to someone who wasn't leaving) versus a missed churner (losing a customer entirely). I don't have real cost figures for this bank, so I'm not going to invent a false-precision "optimal" answer but the tradeoff itself, and having several real options along that curve, is the actually useful output of this analysis, more useful than my original single accuracy number ever was.
What actually drives churn in this data
Feature importance from the balanced Random Forest:
- Age (25.3%) by far the strongest single predictor.
- Balance (14.3%), Estimated Salary (14.0%), Number of Products (13.6%), and Credit Score (13.4%) form a fairly even second tier.
- Tenure (7.9%), Geography (3.9%), Active Member status (3.7%), Gender (2.0%), and having a credit card (1.9%) mattered comparatively little.
Age dominating this strongly, with tenure barely registering, wasn't what I expected going in I'd assumed how long someone had been a customer would matter more than it did.
Where this still falls short
- I didn't have real business cost figures (cost of a retention incentive vs. value of a retained customer), so the "which model to deploy" decision here is illustrative, not a final recommendation.
- The dataset is a single static snapshot rather than a time series, so this predicts churn risk at one point rather than tracking how risk changes over a customer's lifetime.
- I used simple class-weighting rather than more involved resampling techniques (like SMOTE) worth comparing directly in a future pass.
Why I'm publishing the "I got it wrong the first time" version
It would have been easier to just write up the original 86.5%-accuracy result and stop there. I think the more honest and more useful piece is this one going back, checking the number I trusted at the time, and showing exactly where it broke down and what fixing it actually looked like. That's closer to how real model evaluation works than any single clean result.
Code
Original capstone notebook and the updated comparison: github.com/DostonUr/bank_churn
If you've handled class imbalance differently, or have a view on how the precision/recall trade-off should be set for a churn use case like this, I'd like to hear it reply here or find me on LinkedIn.
Top comments (1)
great