Most machine learning tutorials treat classification like a neat academic contest. You download a dataset, split it eighty-twenty, run a gradient booster, look at an ROC-AUC curve, and call model.predict().
If you do that inside a retail bank, you quietly burn money.
Here is why. Standard predict() calls in Scikit-learn and XGBoost apply a hardcoded zero point five probability cutoff. That cutoff assumes an unstated premise: that a false positive costs the exact same as a false negative.
In banking, that premise is completely false.
The Mathematics of the Asymmetric Penalty
Retail banks run on deposits. When an account holder leaves, you do not just lose a record in a PostgreSQL table. You lose net interest margin, card swipe fees, and the chance to underwrite a future mortgage. Replacing that client through paid search and branch promotions costs five to seven times more than keeping them.
Let us put conservative numbers on the board:
- Customer Lifetime Value preserved if saved ($V$): $500
- Direct cost of a retention offer ($C$): $50
- Acceptance rate of the offer ($r$): 30%
If your retention team calls a customer with churn probability $p$ and offers a retention package, the expected profit is:
$$\mathbb{E}[\text{Profit}] = (p \times r \times V) - C$$
When does outreach make economic sense? Exactly when expected profit is positive:
$$(p \times r \times V) - C \ge 0 \implies p \ge \frac{C}{r \times V}$$
Substitute the banking unit economics:
$$t^* = \frac{50}{0.30 \times 500} = \frac{50}{150} = 0.3333$$
The profit-maximizing boundary is not fifty percent. It is thirty-three point three percent.
Traditional Approach (0.50 Cutoff):
[ 0.00 .......................... 0.49 ] [ 0.50 ................... 1.00 ]
❌ Misses 45%+ of genuine churners ⚠️ Only catches obvious flight risks
Leaves $100k+ in deposit runoff Wasted budget on uncalibrated guesses
Prescriptive Decision Boundary (t* = 0.333):
[ 0.00 ... 0.16 ] [ 0.17 ...... 0.33 ] [ 0.333 ................... 1.00 ]
🟢 SAFE ZONE 🟡 WATCHLIST 🔴 ACTIVE RETENTION ZONE
Preserve Automated Direct Banker Outreach
Budget Digital Nudges Expected Net ROI is Positive
If you wait until your classifier is fifty percent certain before contacting an account, you miss over forty-five percent of genuine churners in your portfolio. You save fifty dollars on an outreach call only to lose five hundred dollars in customer lifetime value.
Why Raw Probabilities Lie (The Calibration Problem)
Setting a cutoff at thirty-three percent only works if thirty-three percent actually means thirty-three percent.
Tree-based ensembles like XGBoost and LightGBM are brilliant at ranking. Their split criteria push samples to extreme leaves, which produces distorted, sigmoidal probability outputs. A raw model score of zero point four might correspond to a true real-world churn rate of only twenty percent or sixty percent.
If your probabilities are uncalibrated, your cost equation breaks down immediately.
from sklearn.calibration import CalibratedClassifierCV
from xgboost import XGBClassifier
# Base estimator optimized via Optuna TPE (50 trials)
xgb_champion = XGBClassifier(
n_estimators=320,
max_depth=5,
learning_rate=0.035,
subsample=0.82,
colsample_bytree=0.78,
scale_pos_weight=1.0,
random_state=42
)
# 5-fold cross-validation isotonic probability mapping
calibrated_model = CalibratedClassifierCV(
estimator=xgb_champion,
method="isotonic",
cv=5
)
calibrated_model.fit(X_train, y_train)
We benchmarked Logistic Regression, Random Forest, LightGBM, and XGBoost across stratified five-fold splits. While uncalibrated XGBoost had strong discrimination, wrapping it in five-fold Isotonic Regression drove the Brier score down to zero point zero nine nine four.
On our strict two thousand customer holdout set, the results were:
- ROC-AUC: 0.8682
- PR-AUC: 0.7174 (three point five times over the twenty point four percent baseline)
- Top 10% Decile Lift: 4.20x (capturing 42.0% of all churners in the first decile)
- Holdout Campaign Net Profit: +$17,850
Moving from "Who Leaves" to "What Should We Do?"
A high churn probability tells a branch manager that an account is bleeding. It does not tell them how to fix it.
Telling an account manager that customer twelve-thousand-eight has a sixty-eight percent departure probability without context leads to generic calls and awkward conversations.
We solved this with two layers:
1. Instant SHAP Waterfall Drivers
Every prediction passes through a TreeExplainer instance to compute feature attributions in log-odds space:
import shap
explainer = shap.TreeExplainer(base_pipeline.named_steps["classifier"])
shap_values = explainer(preprocessed_features)
This immediately exposes the root cause:
- Older age pushing risk up right before retirement
- An account holding three or four products (which our data showed has an eighty-two to one hundred percent churn rate due to fee fatigue)
- Inactive digital membership status
2. Counterfactual Prescriptive Recourse
Instead of stopping at explainability, the engine evaluates actionable operational levers:
- Can we get the customer to download the app and set up bill pay (
IsActiveMember: 0 -> 1)? Cost: $15. - Can we consolidate three fragmented fee-bearing accounts into two clean relationship products (
NumOfProducts: 3 -> 2)? Cost: $30. - Can we issue a rewards card with a first-year annual fee waiver (
HasCrCard: 0 -> 1)? Cost: $35. - Or should we deploy a combined retention bundle? Cost: $50.
The recourse engine recalculates the calibrated probability under each counterfactual scenario and computes the expected net dollar return:
$$\text{Expected Net ROI} = (p_{\text{current}} - p_{\text{new}}) \times \text{CLV} \times \text{Save Rate} - \text{Intervention Cost}$$
If the bundle drops the churn probability from sixty-four percent down to twelve percent, that generates an expected net return of fifty-eight dollars and forty cents. The system flags that strategy as optimal and writes a personalized phone script for the relationship manager:
"Hello, thank you for your four years with our bank. We noticed you currently hold three separate product plans with an account balance of $92,000. We can consolidate those accounts into our premier fee-free tier and enable instant mobile security alerts today."
The 2x2 Strategic Action Matrix
When managing a whole portfolio with fifty thousand accounts, individual lookups are not enough.
We built an automated Pandera sanitization engine that handles incoming batch CSV files, maps column aliases, clamps anomalous inputs, and maps every account onto a two-by-two value-risk grid:
The quadrants assign clear playbooks:
- VIP Save Priority (High Risk / High Balance): Red zone. Dedicated relationship manager outreach and custom rate concessions.
- Automated Digital Save (High Risk / Modest Balance): Amber zone. Push notifications, email surveys, and targeted fee waivers.
- Relationship Deepening (Low Risk / High Balance): Blue zone. Loyal high-balance accounts ready for private banking cross-sell.
- Standard Service (Low Risk / Modest Balance): Green zone. Stable accounts requiring zero retention spend.
The interface tracks Expected Balance Runoff across the entire book:
$$\text{Expected Runoff} = \sum_{i=1}^{N} \text{Balance}_i \times p_i$$
This gives treasury and asset-liability teams a clear forecast of potential deposit flight in dollars.
Architecture and Serving
The project ships as two complementary layers:
- The Streamlit Application: Hosted on Streamlit Cloud for interactive decision support, scenario simulation, and campaign CSV exports.
- The FastAPI Microservice: Packaged in a multi-stage Docker container for low-latency batch jobs and core banking integrations.
Where to Explore
The entire platform is open source:
- Live Web Application: bankcustomerchurning.streamlit.app
- Full Documentation & Metric Dictionary: Streamlit_app.md
- GitHub Repository: github.com/ronaldgosso/bank-churn
Machine learning in finance should not end with an AUC score. When you connect model predictions to unit economics, calibration, and prescriptive actions, machine learning stops being an academic experiment and starts defending the balance sheet.




Top comments (0)