Conformal prediction: turn any model into an interval with a coverage guarantee using one calibration quantile

A model that only ever answers "the value is 3.2" is hiding how sure it is. Conformal prediction wraps any trained model in an interval — "the value is in [2.4, 4.0]" — that comes with a genuine mathematical promise: the true value lands inside at least 90% of the time (or 95%, or whatever 1−α you ask for), and the promise holds for any data distribution, with finite data, no Gaussian assumption, no faith in the model being right. The recipe is almost embarrassingly simple, and I built a demo that computes the quantile and measures the resulting coverage live, from scratch, no library. Here's the whole thing.

Split, then score how strange each point is

The "split" in split (inductive) conformal: carve a calibration set out of your data that the model never trains on. Fit the model on the training part; the calibration part is a clean, exchangeable sample used only to measure how big the errors really are. Then compute a nonconformity score on every calibration point — for regression, the natural choice is the absolute residual:

yhat_cal = model.predict(X[cal])
scores   = np.abs(y[cal] - yhat_cal)     # nonconformity = |y - yhat|

That distribution of "typical strangeness" is all conformal needs. (Any score works — |y−ŷ|, |y−ŷ|/σ(x), 1 − softmax prob for classification — the machinery doesn't care.)

The one quantile that carries the guarantee

Here's the entire trick. Sort the calibration scores and take the ⌈(n+1)(1−α)⌉-th smallest:

def conformal_quantile(scores, alpha):
    n = len(scores)
    k = int(np.ceil((n + 1) * (1 - alpha)))    # the magic rank
    if k > n:                                   # too little data for this alpha
        return np.inf
    return np.sort(scores)[k - 1]

The +1 is a "phantom" for the unseen test point, and it's what upgrades an ordinary empirical quantile into a finite-sample guarantee. If the rank exceeds n, you honestly don't have enough calibration data for that α, and q is +∞ — an infinite, useless-but-honest interval. Then you just pad every prediction: the interval is [ŷ − q, ŷ + q], and the theorem says the true y lands inside with probability at least 1 − α.

The guarantee is real, not a fluke

The claim is easy to distrust, so the demo checks it empirically. On a held-out test set the covered fraction should sit at or just above the target, whatever the data looks like:

covered = np.mean((y[te] >= lo) & (y[te] <= hi))
print(f"target {1-alpha:.2f}  ->  empirical coverage {covered:.3f}")

Sweep α across its whole range, recompute the quantile each time, and plot achieved-vs-target coverage: the points ride the diagonal and sit just above it — conformal is guaranteed to be at least 1−α, slightly conservative because of that +1 in the rank. α is the single knob: shrink it for tighter guarantees and wider bands, grow it for narrower bands and weaker guarantees. The width 2q is the price — honest uncertainty in the model's own units.

Make the band breathe with the data

A constant q is one-size-fits-all, so it wastes width where the model is confident and undershoots where it isn't. The fix is a normalized score: fit a second model σ̂(x) to the size of the residuals, score |y−ŷ|/σ̂(x), and the interval becomes ŷ ± q·σ̂(x) — same marginal coverage, but it widens where the model is unsure and pinches where it's confident. Under heteroskedastic noise that grows with x, the adaptive band spends its width far more honestly than the constant one while keeping the exact same coverage.

Why reach for it over the alternatives

Compared with Bayesian credible intervals or a naive Gaussian ±1.96σ, conformal is the only option whose coverage guarantee is finite-sample and distribution-free — its only assumption is exchangeability, not a correct prior or truly Gaussian errors. It's a pure wrapper that works on any model and costs one calibration split plus a quantile. The main caveat: coverage is marginal (averaged over the population), not per-x, which is exactly what the normalized/CQR variants improve. In practice reach for MAPIE or crepes — they handle the split, the quantile, cross-conformal, and for classification produce a guaranteed-coverage set of labels that naturally grows on ambiguous inputs. Whenever a prediction feeds a real decision — a dosage, a price, a risk threshold — conformal turns "my best guess is 3.2" into "it's in [2.4, 4.0], and I'm 90% sure," with the math to back the claim.

Move the sliders and watch the quantile, the band and the coverage recompute from the actual data:
https://dev48v.infy.uk/ml/day45-conformal-prediction.html