The calibration error conditions on the confidence: among the inputs where a model says 0.83, it should be right 83% of the time. A real network emits a different confidence for every input, so that event has probability zero and the number is not estimable until you choose a grouping - which can only ever cancel.
groupCE(conf, acc, w, gid) // sum_g | sum_g w*acc - sum_g w*conf |
// gid = () => 0 -> conf - acc, one bin
// gid = i => binIndex(conf[i], B) -> ECE-B
// gid = i => conf[i] -> one group per distinct confidence
One argument, and "the calibration error" becomes a ladder. The page prints the whole ladder because the input space is finite and p*(y|x) is known, so every rung is an exact weighted sum rather than an estimate. The student is a real 163-parameter net, backward pass hand-written and checked against independently written central differences: 73.5% accurate, 83.1% confident. https://dev48.infy.uk/dl/day70-calibration-temperature-scaling.html
| grouping | as trained | T = 1.5969 | reported drop |
|---|---|---|---|
| 1 bin | 0.0961 | 0.0033 | 96.6% |
| 15 bins | 0.0961 | 0.0576 | 40.1% |
| 200 bins | 0.1194 | 0.0877 | 26.6% |
| per distinct confidence | 0.1380 | 0.1161 | 15.9% |
Fifteen bins agrees with one bin to 1.9e-16 - summation-order rounding - because all nine occupied bins lean the same way and nothing cancels. Before recalibration the standard apparatus adds no information to conf minus acc.
Fit one temperature and the number you report falls 40.1% while the strictest reading falls 15.9%. The net signed error is bin-independent, so everything a finer grouping buys is cancellation: after the fit, 6 bins are overconfident and 4 under. The metric goes blind when you fix what it could see. Over a 27-configuration grid the reported drop flattered the real one 27 times out of 27, and in 3 the finest rung got worse while the report improved.
Two rows score a perfect zero at every rung: p* itself, and an input-ignoring predictor emitting the class marginal - a perfect score, available to a model that never looks at its data.
And the floor nobody quotes: hand the estimator p*, whose error is exactly 0, and 15 bins reports 0.0929 at n=100 and 0.0451 at n=500 - so the 0.026 gain this page's held-out set would have claimed sits below its own instrument's floor.
What the measurement took back
I set out to write "never fit T on the training set" as advice. It is an identity. Scaling the logits by beta is scaling the last layer by beta, so dNLL/dbeta at beta=1 is the directional derivative of the loss along the output layer - two unrelated code paths agreeing to 6.2e-17 - and training has spent every step flattening it. Measured: T_train = 0.9125 against 1.5969 held out - the training set wants to sharpen an already overconfident model.
The method survives: one parameter halves the excess NLL, 0.1905 to 0.0955 nats above the irreducible floor, at bit-identical accuracy. So the page had to call the instrument unreliable rather than the fix. Scored as a ranker over 108 trained models, ECE-15 orders probability quality at AUC 0.712 - below the plain error rate at 0.791, far below held-out NLL at 0.951 - and recovers to 0.875 on the one comparison it was designed for. A scope failure, not incompetence.
134 verifier assertions, 65 running in the page, not one asset fetched off it. One deep-learning idea a day, computed rather than quoted: https://dev48.infy.uk/deeplearningfromzero.php
Top comments (0)