A model can tell you it approved a loan at 59%. What it won't tell you is why: which facts about the applicant earned that number, and how much each one mattered. That gap is what explainability tries to close, and SHAP is the sharpest tool we have for it. I built a small interactive version from scratch to see exactly how it works, and a few things finally clicked.
The setup is a tiny loan-approval model over five features: income, credit score, debt, age, and years employed. Feed it an applicant and it returns an approval probability. The real question is how to split that probability across the five inputs in a way that's actually fair.
SHAP's answer comes from an old idea in cooperative game theory. Lloyd Shapley asked, back in 1953: if a group of players cooperate and produce some payout, how do you divide it fairly among them? He proved there's exactly one split that satisfies a handful of common-sense rules, and it's now called the Shapley value. SHAP reframes a prediction as that game. The players are the features. The payout is how far the prediction sits from the model's average output. Each feature's Shapley value is its fair cut of that gap.
The subtle part is defining what a feature "contributes," because it depends on who else is already in the room. Credit score might swing things a lot on its own but add almost nothing once income is known. So SHAP measures a feature's marginal contribution across every possible coalition of the other features: how much does the prediction move when you add this feature to a group that already has these others? Average that over all groups, equivalently over every order the features could be revealed in, and you get the Shapley value.
With five features this is completely doable by brute force. For each feature I enumerate all 2^4 = 16 subsets of the other four, compute the model output with and without the feature, and take a weighted average of the differences. No sampling, no approximation, the exact number. A "missing" feature just gets replaced with its background value, here the population mean, which is how you get the model to produce an output for a partial set of inputs.
What sold me was the additivity check. The five SHAP values sum, exactly, to prediction minus base value. Base is what the model predicts for an average applicant, 0.500 in my setup; prediction is this applicant's number. Every point of the difference gets handed to some feature, nothing invented, nothing lost. I verified it in Node and the error was about 1e-17, pure floating-point noise. That's the efficiency axiom, and it's what makes the waterfall plot a real accounting rather than a vibe: start at the base, let each feature push the running total up (red) or down (blue), and you land precisely on the prediction. The force plot shows the same numbers as opposing forces meeting at the output.
A single explanation is local, it's about one applicant. To understand the model overall you average the absolute SHAP value of each feature across many rows. That gives global importance: how much the feature moves predictions in general. The absolute value matters here, because a feature that pushes up for some people and down for others would cancel to zero otherwise.
The one thing that doesn't scale is the brute force. Enumerating 2^n coalitions is fine at 5 features (32), painful at 20 (a million), and hopeless at 100. So real libraries approximate. KernelSHAP samples coalitions and fits a specially weighted regression whose coefficients are the Shapley values. TreeSHAP exploits the structure of decision trees to get the exact values in polynomial time, which is why SHAP and gradient-boosted trees pair so well. In practice you write three lines: build a shap.TreeExplainer, call it on your data, and plot the waterfall, force, or beeswarm.
Two caveats I keep in mind. SHAP explains the model, not reality, so if the model is biased the explanation faithfully describes the bias. And it's attribution, not causation: a large SHAP value doesn't mean changing that feature would change the outcome. Read it as "what drove the model here," never as "what causes this."
One more thing worth stating plainly: this is not the same as a tree's built-in feature importance. That number is global, always positive, and unsigned. SHAP is per-prediction and signed, so it tells you not just that income mattered but that here it pushed approval up by 0.14 while debt pulled it down by 0.17.
You can drag the feature sliders and watch the waterfall recompute here: https://dev48v.infy.uk/ml/day29-shap.html
Next up is learning curves: plotting train versus validation error against dataset size to tell whether more data, or a different model, will actually help.
Top comments (0)