Building a churn model that actually gets used

Jul 2026

Most ML models die in a Jupyter notebook. Here’s how I turned a prototype into a production system that the sales team checks every morning.

I’ve built more churn models than I care to admit. The pattern was always the same: spend two weeks in a notebook, get a respectable AUC, show a slide deck, get polite nods… and then the model quietly disappears. Sales keeps using their gut and the CRM filters they already know.

Last year I decided to break the cycle. The result is a system the sales team opens every morning before their stand-up. They don’t call it “the model.” They call it “the risk list.” That linguistic shift told me we had finally crossed the line from data science project to working tool.

Here’s exactly how we did it.

The starting point (and why most models fail)

We had the usual ingredients:

• ~180k active customers
• 24 months of transaction, support, and product-usage data
• A clear business definition of churn (no activity + cancelled subscription within 60 days)

The first notebook version used a gradient-boosted tree, engineered ~40 features, and hit 0.84 AUC on a time-based holdout. Classic. And completely unused.

The failure modes were predictable:

1. No clear owner – Data science “owned” the model; sales owned the customers.
2. No daily rhythm – Predictions lived in a static CSV or a dashboard no one bookmarked.
3. Too much friction – Sales had to leave Salesforce, open another tool, interpret probabilities, and decide what to do.
4. No feedback loop – We never measured whether the model actually reduced churn.

We treated those four problems as requirements, not afterthoughts.

Step 1: Define success the way the business does

We stopped talking about AUC in the first meeting with sales leadership. Instead we asked:

“If this worked perfectly, what would change in your day?”

Their answer was concrete:

“I want a ranked list of customers who are likely to leave in the next 30–60 days, with a short reason, and I want it inside Salesforce every morning by 7 a.m.”

That became the product spec. Everything else (model choice, infrastructure, monitoring) was downstream of that sentence.

Step 2: Keep the model simple enough to explain

We deliberately chose a model that sales could interrogate. After testing several options we landed on a LightGBM model with strong regularization and a small set of high-signal features. The final feature set was only 18 columns:

• Recency and frequency of product usage
• Support ticket volume and sentiment trend
• Contract value and recent expansion/contraction
• Login and feature-adoption decay
• A handful of interaction terms

We avoided deep learning and complex ensembles. Explainability mattered more than an extra 0.02 AUC.

Feature importance (averaged across folds) looked like this:

(The top drivers were almost always some form of declining engagement + rising support friction. Sales already knew this intuitively; the model just quantified and ranked it daily.)

Step 3: Productionize the boring parts ruthlessly

The notebook was rewritten into a clean pipeline:

1. Daily batch job (Airflow / Prefect) that pulls the previous day’s data from the warehouse.
2. Feature store (we used a simple dbt + Parquet layer) so training and inference use identical logic.
3. Model training runs weekly; inference runs nightly.
4. Predictions + SHAP values are written back to a table.
5. Salesforce sync pushes the top-risk customers into a custom object and a list view that sales already uses.

Architecture (simplified):

Warehouse → dbt features → nightly inference → predictions table
                                              ↓
                                    Salesforce (list view + email digest)
                                              ↓
                                    Sales actions → logged back to warehouse

No real-time scoring was needed. Churn risk doesn’t change hour-by-hour for our business.

Step 4: Design for the morning ritual

The user experience was non-negotiable:

• Every sales rep sees a ranked “At-Risk” list in Salesforce filtered to their book.
• Each row shows: customer name, risk score (High/Medium), top 2–3 reasons in plain English (“Usage down 40% in last 30 days”, “3 unresolved support tickets”), and a one-click “Log outreach” button.
• A short daily Slack digest goes to team leads with the highest-risk accounts.

We A/B tested the language. “Probability 0.73” was ignored. “High risk – usage collapsing” got action.

Step 5: Close the loop and measure what matters

We instrumented everything:

• Did the rep open the list?
• Did they take an action within 7 days?
• Did the customer ultimately churn?

After three months the numbers looked like this:

Metric	Before	After (3 months)
% of high-risk accounts contacted within 7 days	18%	67%
Churn rate in the high-risk segment	31%	19%
Overall logo churn	8.4%	6.9%

The model didn’t magically retain customers. The sales team did — because the system made the right customers visible at the right time with just enough context to act.

What actually made the difference

Looking back, five decisions mattered more than any modeling trick:

1. Started with the workflow, not the algorithm.
2. Optimized for adoption metrics (actions taken) instead of offline metrics.
3. Kept the model and the explanation simple.
4. Put the output inside the tool sales already lived in.
5. Built a feedback loop from day one so we could prove (or disprove) value quickly.

Most models die in notebooks because we treat the notebook as the finish line. The real product is the daily habit it creates for the people who talk to customers.

If your churn model (or any model) is still sitting in a repo or a dashboard that only data people open, ask the uncomfortable question: What would have to be true for the sales team to check this every morning?

Then build that.

The algorithm is the easy part.