How we built a customer health score you can argue with

The problem: nobody trusts a number they can't interrogate

Every customer success tool ships a "health score." Open one up and you get something like:

62/100 🟡

And then nothing. Why 62? Why yellow? What moved it since last week? The person staring at that number has no idea, so they do the rational thing: they ignore it and go back to their gut and their inbox.

That is the dirty secret of health scores. Most of them are black boxes, and a black box you can't question is a black box you won't act on. We learned that the expensive way, so when we rebuilt ours, we gave ourselves one rule:

Every score has to be one you can argue with.

If someone can look at a score, see exactly why it is what it is, and push back ("no, that usage dip is seasonal"), then the score is doing its job. The argument is the feature, not the bug.

Here is how we built it.

Design principle: explainable by default, not explainable on demand
The tempting path is to train a model first, get a nice AUC, and bolt on SHAP values later to explain it. We flipped that. Explainability was the constraint we designed around from line one, not a layer we added after.

The reason is the consumer. The person reading this score has to decide whether to spend an hour on an account today. They do not need three more decimal places of accuracy. They need to know what to do and why. So we made a conscious tradeoff:

A slightly less "accurate" score that a human trusts and acts on beats a more accurate one they ignore. An unused prediction has an effective accuracy of zero.

The architecture: signals in, contributions out

The score reads from four kinds of source: product usage, billing, support, and CRM. The trick is that we never collapse them into one opaque number. Each source emits signals, and every signal carries four things:

a direction (is this good or bad for retention)
a weight (how much it moves the score)
a confidence (how sure we are)
a reason, in plain English

{
"signal": "core_feature_usage_drop",
"source": "product",
"direction": "negative",
"weight": -14,
"confidence": 0.9,
"reason": "Core feature usage down 41% over the last 30 days"
}
The final score is an aggregation of these contributions. But, and this is the whole point, we keep the contributions. The number is just the top of the stack. Underneath it is the full list of what pushed it up and what pulled it down, ranked by impact. That list is the thing you argue with.

Deterministic facts don't belong in a fuzzy score

Some signals are not probabilistic. They are facts. A card that expires in 12 days will fail. Auto-renew switched off is a decision, not a hint. Dunning already running means a payment has already bounced.

We treat these as hard flags, not soft weights, because smoothing a fact into a probability throws away the most reliable signal you have. This is where a lot of health scores quietly lose money: the account looks healthy on engagement, the score stays green, and then the renewal fails on a dead card nobody was watching. Billing facts get surfaced separately and are allowed to override the fuzzy score.

{
"signal": "auto_renew_disabled",
"source": "billing",
"type": "deterministic",
"reason": "Auto-renew switched off 6 days ago",
"override": true
}

The hard parts (where the naive version breaks)

1. Missing data. Not every account has every source connected. The naive version penalizes an account for a missing integration, which is nonsense. Score on the sources you have, and lower the confidence rather than inventing a red flag out of silence.

2. The everything-is-yellow problem. Average enough signals together and every account converges to a mushy 70. So do not average. Weight by impact and recency and let strong signals dominate, so a healthy account actually looks healthy and a sliding one actually moves.

3. Recency and decay. A support blow-up six months ago is history. The same blow-up last week is a signal. Time-decay weights so old events fade instead of anchoring the score forever.

4. Reasons have to be human. "feature_7 contributed -0.12" helps nobody. Every signal maps to a sentence a person would actually say out loud. That translation layer was more work than the scoring math, and it is the part that makes people trust the output.

What the output actually looks like

Instead of 62/100 🟡, the person sees:

This account - 62, and sliding
Why:
↓ Core feature usage down 41% in 30 days (biggest factor)
↓ Champion replies slowed from ~4h to 3 days
↓ Card on file expires in 12 days
↑ Two new seats added last week
Renewal in 38 days.
Now they can do something. Act on the usage drop, re-engage the champion, get the card updated before it silently fails, or look at it and say "that dip is their December freeze, ignore it." All four are wins, because all four are informed decisions instead of a shrug at a yellow dot.

Why "argue with" is the whole game

The moment someone can overrule the score with a reason, two things happen. They start trusting it, because it is not pretending to be an oracle. And they start using it, because it points them somewhere specific.

An oracle you cannot question is just a number on a dashboard. A colleague who shows their reasoning is something you will actually listen to, and correct, and come to rely on. The corrections are trust and training data at the same time.

If you are building your own

• Design for explainability from line one. Do not train a black box and bolt on explanations later.
• Keep the per-signal contributions, not just the final number.
• Treat deterministic facts (especially billing) as facts, not soft weights.
• Translate every signal into a sentence a human would say.
• Optimize for "acted on," not for AUC.

We built all of this into GainTrace: it reads product, billing, support, and CRM, and turns them into one explainable score per account, with the reason attached, so risk shows up about 45 days before a renewal. If you want the product-side version of this rather than the engineering one, we go deeper in how to build a customer health score that predicts churn.

Curious how other people handle the missing-data and everything-is-yellow problems. Tell me in the comments.