You have a model and one prediction. The question everyone asks is: how much of this number is down to each input? Shapley values answer it by asking a more careful question — how much does this feature add, on average, over every possible order in which the features could have arrived?
👉 Live, runs in your browser: https://dev48.in/ml/day81-shapley-values.html
The obvious alternative, and both ways it breaks
Leave-one-out: drop a feature, see how much the prediction moves, call that its importance. One line of code.
Case 1 — interaction. A model that fires only when both features are present, worth 10.
| feature | Shapley | leave-one-out |
|---|---|---|
| has_card | 5 | 10 |
| is_verified | 5 | 10 |
| total | 10 ✓ | 20 ✗ |
Drop either one and the prediction falls to the floor, so leave-one-out scores them both as fully responsible. It attributes twice the prediction.
Case 2 — redundancy. Two features carrying the same information.
| feature | Shapley | leave-one-out |
|---|---|---|
| signal | 4 | 0 |
| signal_copy | 4 | 0 |
Drop either copy and nothing moves, so it scores both at exactly zero. A feature-importance plot built this way shows real signal as irrelevant, and somebody deletes it.
🔴 It errs in both directions — over-attributing interaction, under-attributing redundancy — and it agrees with Shapley exactly on additive models. Which is precisely why an additive example proves nothing about an attribution method.
Three derivations that agree to the last bit
phi_i = sum over S not containing i of
|S|! (n-|S|-1)! / n! * ( v(S + i) - v(S) )
That factorial weight is not decoration. It is exactly the fraction of the n! orderings in which S arrives before i and everything else after — a probability distribution over the position at which feature i arrives, summing to 1.
The page computes the same values a second way, as an average over every permutation. The verifier computes them a third way that touches neither factorials nor permutations: Möbius inversion of the set function into Harsanyi dividends, then split each dividend equally among the coalition that earned it.
m(S) = sum over T subset of S of (-1)^(|S|-|T|) v(T)
phi_i = sum over S containing i of m(S) / |S|
For the AND model the whole dividend sits on the pair — m({a,b}) = 10, m({a}) = m({b}) = 0 — which is the statement that neither feature does anything alone, in one line.
All four axioms, checked
Shapley values are not a heuristic that happens to work. They are the unique attribution satisfying four requirements at once:
- Efficiency — the values sum to the prediction minus the baseline. This is the one leave-one-out cannot keep.
- Dummy — a feature that never changes anything scores exactly 0.
- Symmetry — interchangeable features get identical values.
- Linearity — attributing a sum of models is the sum of the attributions.
Checked on every model, with symmetry re-derived by swapping columns of v rather than trusting the page's detector.
The sampled estimator keeps one property for free
Exact Shapley costs 2ⁿ model evaluations. Past about twenty features that is hopeless, so practice samples permutations.
Efficiency stays exact at every sample count — even at 1. Each permutation's marginal contributions telescope to v(full) − v(empty) whatever order you drew. What sampling costs you is per-feature accuracy, and that is the number you should report.
And attribution has a sign
A feature that lowers the prediction gets a negative value: +6 for the helpful one, −4 for the harmful one. A method that reports magnitudes only cannot say that.
470 verifier asserts, 23 in-page checks, 0 failures.
Top comments (0)