Your model is 95% accurate and still lying: calibration, the reliability diagram, ECE, and how to fix it

A classifier can be accurate and still lie about its confidence. When a well-trained model says "90% sure this is spam," you'd like it to be right about 90 times out of 100 — but modern models (boosted trees, deep nets) are famously over*confident: among the cases it stamped 0.9, the truth might be positive only 70% of the time. Accuracy asks "is the label right?"; **calibration* asks the harder question "is the probability right?" I built a demo that bins real samples, computes the calibration error live, and fits two fixes from scratch — no library. Here's what it shows.

The reliability diagram exposes the lie

The tool that reveals miscalibration is the reliability diagram: sort predictions into bins over [0,1], and for each bin plot the mean predicted probability (conf) against the fraction that were actually positive (acc):

def reliability(p, y, n_bins=12):
    edges = np.linspace(0, 1, n_bins + 1)
    idx = np.clip(np.digitize(p, edges) - 1, 0, n_bins - 1)
    rows = []
    for b in range(n_bins):
        m = idx == b
        if m.sum():
            rows.append(dict(conf=p[m].mean(), acc=y[m].mean(), n=int(m.sum())))
    return rows

Perfect calibration rides the 45° diagonal. An overconfident model sags below it (its 0.9s under-deliver); an underconfident one climbs above it. The demo's sharpness knob lets you sweep between the two and watch the curve bow either way — with the accuracy completely unchanged, because only the numbers on the confidence move.

ECE: one number for the whole curve

Expected Calibration Error compresses that picture into a single value — the size-weighted average vertical gap between each bin and the diagonal:

def ece(p, y, n_bins=12):
    edges = np.linspace(0, 1, n_bins + 1)
    idx = np.clip(np.digitize(p, edges) - 1, 0, n_bins - 1)
    N, e = len(p), 0.0
    for b in range(n_bins):
        m = idx == b
        if m.sum():
            e += (m.sum() / N) * abs(p[m].mean() - y[m].mean())   # weight x gap
    return e

The crucial point the demo hammers home: a 95%-accurate model can still have a huge ECE if its confidences lie. Accuracy and calibration are different axes.

Two fixes, both monotone

Calibration is post-processing — leave the classifier alone and learn a monotone map from its scores to honest probabilities on a holdout. Two classic choices:

Platt scaling fits a 2-parameter sigmoid σ(A·logit(p) + B) by minimizing log-loss. Smooth, low-variance, ideal when the distortion really is sigmoid-shaped and when data is scarce — two knobs can't overfit much.

Isotonic regression fits any non-decreasing step function via Pool-Adjacent-Violators: sort by score, then repeatedly merge any adjacent block whose value exceeds its right neighbour into a pooled weighted-mean block. The result is a free-form monotone staircase — it corrects odd distortions Platt can't, but it's hungrier for data and goes jagged on a small fit set. The demo draws both maps side by side and races their ECE; shrink the fit-set slider and you watch isotonic get spiky while Platt stays steady.

Because both are monotone, neither changes the ranking — AUC and (near enough) accuracy are untouched. Only the probabilities become trustworthy.

The honest protocol

The mistake that quietly ruins calibration results: never fit the calibrator on data the classifier already saw, and never report ECE on the same data you calibrated on — both leak and understate the error. Split off a calibration holdout, fit there, measure ECE on a separate test set:

cal = fit_platt(p_cal, y_cal)              # fit on the holdout
print("raw ECE:", ece(p_test, y_test))
print("cal ECE:", ece(cal(p_test), y_test))   # should drop sharply

In practice, reach for scikit-learn's CalibratedClassifierCV (method="sigmoid" for Platt, "isotonic" for isotonic, holdouts built in). For deep nets the standard near-free fix is temperature scaling — divide the logits by one learned scalar T, a special case of Platt with B=0 that softens overconfident logits and keeps the argmax (so accuracy is fixed).

Whenever a probability drives a decision — a medical risk, a fraud threshold, an expected-value bet — calibration is what makes the number mean what it says.

Move the sliders and watch the curve straighten and the ECE drop:
https://dev48v.infy.uk/ml/day44-calibration.html