Four days of this series asked what a number somebody reports is worth. This one is about the number somebody sorts on: the leaderboard column that picks which model ships.
Run it: https://dev48.infy.uk/ml/day77-proper-scoring-rules.html
Nothing is sampled, and nothing has a tolerance
A world is 9 score bins with rational masses on one common denominator, so every confusion matrix is a pair of BigInt sums and every comparison is a cross-multiplication. "Brier is proper" is settled by the sign of an exact rational derivative, not by a search. Absolute error is settled the same way, and its optimum is a corner — never the truth.
The two metrics everybody actually reports are not scoring rules at all, so the question has to be replaced with: which reports can this number not tell apart? On a 241-value grid:
| metric | size of its indistinguishable set |
|---|---|
| Brier | exactly one point |
| AUC | C(N+1, m) = 6.50×10¹⁵ |
| F1@0.5 | 5.29×10¹⁸ |
The fixed point
Search all 512 decision rules — not the m+1 thresholds — and the F1-optimal rule is a threshold on η whose cut is exactly half the F1 it achieves: 0.2696, never 0.5.
So the model that tops an F1@0.5 leaderboard is the one that over-states every probability by exactly 0.4607 = 1 − F1*. What that buys (0.2253 of F1) and what it costs (0.055005 of Brier, an ECE of 0.2283) are closed forms, confirmed against the enumeration.
The leaderboard itself
The exactly-correct model is first on Brier, first on log loss, first on AUC, first on accuracy — and 5th of 10 on F1@0.5. It is beaten by the lie, by "flag every row" (0.4327), and by a model that is exactly backwards — AUC 0.2629 — scoring 0.3688 against the truth's 0.3140.
Over 246 enumerated worlds the truth is first on Brier 246 times and first on F1@0.5 once. Flagging everything beats it 193 times. The backwards model beats it 144 times.
The repair works, and takes something with it
Tune the cut and the lie is worth exactly zero — and five models that differ by 0.055 in Brier become bit-identical. You bought honesty by deleting the ability to rank.
And the column I built to rank the metrics ranks them backwards
AUC's indistinguishable set is 814× smaller than F1@0.5's — and yet AUC ties 5 of the models on the board where F1@0.5 ties 1, in 245 of 246 worlds. Resolution in principle and resolution on your actual board are different quantities, and I had assumed they were the same one.
Verifier 78 asserts, page self-check 38, 0 failures.
Top comments (0)