You tuned a decision threshold on validation to hit "precision at least 90%". You shipped it. The population's prevalence drifted from 10% to 2%.
The model did not change. The scores did not change. TPR and FPR at that threshold are bit-identical — the same reduced fractions, 22663/29409 and 21043/499953.
Precision is now 63.46%.
👉 Live, all 201 thresholds enumerated exactly in your browser: https://dev48v.infy.uk/ml/day79-threshold-transfer.html
Nothing is sampled and no rate is a float
The score axis is 201 bins of declared integer weights, all 201 thresholds are enumerated, and every rate is an exact rational compared by BigInt cross-multiplication — so an argmax is never decided by rounding, and there are no confidence intervals anywhere because there is nothing to be uncertain about.
The identity the whole thing rests on
Over a virtual population of q·WP·WN, every confusion cell is an exact integer:
TP = p * Sp * WN FP = (q - p) * Sn * WP
FN = p * (WP - Sp) * WN TN = (q - p) * (WN - Sn) * WP
So:
TPR = TP / (TP + FN)
= (p·Sp·WN) / (p·Sp·WN + p·(WP−Sp)·WN)
= Sp / WP
p and q cancel. TPR does not know what the prevalence is. Neither does FPR. Both are conditioned on the class.
Precision is not. Over six prevalences from 0.1% to 90% at one fixed threshold:
| prevalence | TPR (reduced) | FPR (reduced) | precision |
|---|---|---|---|
| 0.1% | 22663/29409 |
21043/499953 |
0.0180 |
| 2% | 22663/29409 |
21043/499953 |
0.2720 |
| 10% | 22663/29409 |
21043/499953 |
0.6704 |
| 30% | 22663/29409 |
21043/499953 |
0.8870 |
| 50% | 22663/29409 |
21043/499953 |
0.9482 |
| 90% | 22663/29409 |
21043/499953 |
0.9940 |
Two identical columns and a fifty-fold swing. That is an algebraic identity on one side and the entire problem on the other.
Two of five tuning rules pick a portable threshold
Re-running each rule on the shifted population and reading off where it lands:
| tuning rule | source | after 10%→2% | portable? |
|---|---|---|---|
| score ≥ 0.50 | bin 100 | bin 100 | n/a — it cannot move |
| maximise F1 | bin 127 | bin 146 | no |
| maximise TPR − FPR (Youden) | bin 100 | bin 100 | yes |
| precision ≥ 0.90 | bin 148 | bin 169 | no |
| recall ≥ 0.90 | bin 98 | bin 98 | yes |
The two that survive are exactly the two whose definitions mention only class-conditional rates. That is not an empirical coincidence to be spot-checked; it follows from the identity above. Youden picks the same bin at five different prevalences; max-F1 picks a different bin at nearly every one.
Scope: recall≥0.90 is also unmoved by the score shift I used here, but only because that shift moves the negatives. Recall does not look at negatives. Move the positives and it moves like everything else.
Re-tuning meets the target and hollows out the model
Re-running precision≥0.90 on each world does restore the number:
| world | re-tuned threshold | precision | recall |
|---|---|---|---|
| source | bin 148 | 0.9044 | 0.4955 |
| prevalence shift | bin 169 | 0.9088 | 0.2412 |
| score shift | bin 166 | 0.9041 | 0.2773 |
| both | bin 193 | 0.9022 | 0.0190 |
In the last world the threshold that satisfies the contract fires on 1.9% of the positives. A precision target is a promise about the answers you give, and it can always be kept by giving fewer. A monitor watching only precision reports a healthy system right up to the point where the system is not answering.
The part with no good answer
In production you usually have no labels. The one thing you can watch is how often the model says yes.
I constructed two worlds with the exactly equal positive rate — not close, the identical rational 780643/49995300 ≈ 1.5614%. The prevalence for the second was solved from rate = π·TPR + (1−π)·FPR, which has an exact rational answer.
| world | positive rate | precision | recall |
|---|---|---|---|
| prevalence shift, 10% → 2% | 780643/49995300 |
0.6346 | 0.4955 |
| score shift, prevalence 0.0702% | 780643/49995300 |
0.0223 | 0.4955 |
Same firing rate. Same recall. A 28.5× difference in precision. Every dashboard that watches the positive rate shows a flat line across both.
What this does not cover
One score axis in 201 bins, one shift direction (negatives up 12 bins, positives never move), one pair of class-conditional shapes, five tuning rules. There is no cost matrix, so "which threshold is right" is never answered here — only which one is stable. No calibration model and no recalibration method: the point is what breaks, not how to fix it.
21 in-page checks, 72 verifier assertions, 0 failures.
Top comments (0)