Originally published at nbaproplab.com/learn/model-postmortem.
A model that looks excellent offline and random in production is usually blamed on the serving code. Ours was not: the two computations matched to 2e-16. The cause was a calibration snapshot whose export query nobody wrote down.
What we built
We score NBA and WNBA player props through seven independent blocks: player form, matchup, game context, market line, teammate synergy, analysis quality and external signals. Each returns a 0-100 score with a confidence index, and a confidence-weighted mean produces the final score that decides whether a pick is published.
On top of that engine we fitted a logistic meta-learner. It takes the seven block scores plus the pick direction and outputs a single calibrated probability that the pick hits. The idea was sound: a weighted mean treats every block as positively predictive, while signed coefficients let an anti-predictive block subtract instead.
Offline it looked excellent. The top tier came out at a 63.7% hit rate with an AUC of 0.56. We deployed it in shadow mode, writing its probability next to every saved pick without letting it influence anything, and planned the switch for the following release.
What went wrong
The switch never happened, because the backtest never reproduced.
Measured against clean live-only data, the deployed model scored an AUC of about 0.51. That is a coin flip. Worse than the flat result, the ordering was inverted: the band the model was most confident about hit 53%, while the band immediately below it hit 67.6%. A model whose top tier underperforms its second tier is not a weak model, it is a broken one.
The obvious suspect is the serving path, so we checked it first. We recomputed the probability for saved picks using the production code and compared it against the training-time computation. The difference was 2e-16, which is floating point noise. The code was correct.
The actual cause: a CSV nobody documented
The training set had been exported once, into a file, from a database snapshot. The export query was never recorded. Neither was the date window, which turned out to be a narrow slice of spring. Nobody wrote down how the file had been produced, so nobody could check whether it matched what production actually served.
It did not match. The training population and the serving population were different, which is the textbook train-serve mismatch. What made it survive review for weeks was not the mismatch itself but its invisibility: with no provenance recorded, there was nothing to compare against, and the impressive offline numbers had no way of being challenged.
What we changed
- The model is parked. It is still computed and stored next to every pick, and it still drives nothing user-facing.
- Provenance is mandatory. Any dataset that informs a decision carries its exact query, its date window and its reason in the changelog, in the same commit that uses it. If it is not written down, it did not happen.
- The replacement only graduates on forward data. A gradient boosting combiner runs as a frozen forward shadow, writing predictions to its own table. It replaces the live engine only if its volume-matched top tier stays ahead over several hundred forward-settled picks. No backtest can promote it.
- Calibration is monitored daily. Brier score, log loss, AUC and per-tier hit rates over a trailing window, with the window floored at the model finalize date so training data cannot leak into the metric that is supposed to police it.
The numbers that replaced it
| Population | Settled picks | Hit rate | Note |
|---|---|---|---|
| Everything the engine graded | 58,459 | 54.3% | Includes internal tiers that are never published |
| Top tier, live only, since April 2026 | 1,357 | 55.6% | Break-even at -110 odds is 52.4% |
| Top tier, 2026 WNBA season | 546 | 62.3% | Fully live: the season began after the backtest cutoff |
| Top tier, full public record | 3,340 | 62.5% | Includes the backtested span up to 2026-04-02 |
Every settled pick is public and filterable by league, tier and date at nbaproplab.com/track-record, and the raw JSON is open at /api/v1/track-record with no key and no signup. Figures cover 2026-01-01 to 2026-08-30.
If you build these things
A public record that shows only the good parts is marketing, not evidence. We publish the misses next to the hits for the same reason we published this postmortem: a claim nobody can falsify is worth nothing.
If you are fitting a meta-learner on top of your own scoring stack, the cheapest insurance is not a better model. It is writing down, in the commit that uses it, exactly which rows your training file contains and how you got them.
Top comments (0)