A classifier ends in a softmax and is trained by cross-entropy against a one-hot target: 1 on the right class, 0 everywhere else. That target is quietly impossible to satisfy — and chasing it is what makes models say "99.9% cat" on images they should be unsure about. Label smoothing is a one-line fix, and I built a demo with a single ε slider so you can watch the target melt and the confidence follow. Here is what is actually going on.
The disease: a target the softmax can never reach
To make softmax output exactly 1.0 for the true class, its logit has to run off to +∞ relative to the rest. The loss −log(p_true) only hits zero in that limit, so its gradient is non-zero for any finite logits — there is always more reward for being more confident. The model gets the class right but the probability wildly wrong, and gradient descent never stops inflating it. That miscalibration bites whenever something downstream trusts the number: triage, ensembling, selective prediction, risk scoring.
The fix, in two lines
Blend the one-hot target with a uniform distribution: y = (1−ε)·onehot + ε/K. The true class becomes 1−ε+ε/K — a reachable, finite goal — and every wrong class gets a small floor of ε/K the model is no longer allowed to crush to zero. ε = 0 recovers the hard target.
function smoothTarget(K, i, eps) {
return Array.from({ length: K },
(_, j) => (j === i ? 1 - eps : 0) + eps / K);
}
const soft = smoothTarget(6, 0, 0.1);
// [0.9167, 0.0167, 0.0167, 0.0167, 0.0167, 0.0167] (sums to 1)
// ^ true = 1 - 0.1 + 0.1/6 ^ others = 0.1/6
What it does to the loss
Cross-entropy against the smoothed target splits into two terms: the usual "be right" pull, scaled down slightly, plus a new term minimised when the model spreads some probability to all classes — a built-in "don't be too peaky" penalty. Equivalently: CE(soft, p) = (1−ε)·CE(onehot, p) + ε·CE(uniform, p) + const. Its minimum sits at a plateau, not a spike.
The gradient sign flip — the whole point
The gradient of softmax + cross-entropy with respect to the logits is beautifully simple: p − t. With a hard target the true-class gradient is p_true − 1, negative until confidence hits 1 — so the logit is pushed up forever. With smoothing the target is only 1−ε+ε/K, so the moment the model's confidence passes that ceiling the gradient goes positive and actively pulls the over-confident logit back down. For the first time there is a finite fixed point.
const grad = p.map((pj, j) => pj - target[j]); // dL/dz = p - t
// TRUE class, hard: p_true - 1 = 0.908 - 1 = -0.092 (push UP, forever)
// TRUE class, eps=0.18: p_true - ceiling = 0.908 - 0.85 = +0.058 (push DOWN!)
// each WRONG class: target = eps/K > 0, so its logit can't be driven to -Infinity
Watching that number cross zero in the demo — from "▲ push up" to "▼ push down" as you drag ε — is the clearest way I have seen to feel why smoothing works.
Calibration, and the trade-off
A model is calibrated when its confidence matches how often it is right. Hard targets are systematically over-confident (confidence far above accuracy, high Expected Calibration Error). Label smoothing drags the confidence ceiling 1−ε+ε/K down toward the true accuracy. Pick ε so the ceiling roughly matches accuracy and the model becomes well-calibrated, usually with a small accuracy bump too. But there is a real trade-off: too much ε pushes the ceiling below accuracy and the model turns under-confident, throwing away genuine certainty. ε is a dial from over-confident (0) to under-confident (large); ~0.1 is the standard default. Confirm on a reliability diagram, not by feel.
In practice it is one argument
You never hand-roll this. Frameworks fold it into the loss — pass raw logits and integer targets, set label_smoothing, done. One hyper-parameter, no architecture change, no extra compute: one of the cheapest wins in the training toolbox.
import torch.nn as nn
criterion = nn.CrossEntropyLoss(label_smoothing=0.1) # the whole feature
loss = criterion(model(x), y) # y = integer class indices
loss.backward()
Two caveats worth knowing. A smoothed teacher hurts knowledge distillation — squashing the logits to equal gaps erases the "dark knowledge" about class similarity a student learns from. And it can wash out the embedding geometry that retrieval and metric learning depend on. Rule of thumb: great for a final classifier whose probabilities you will read; think twice when something downstream consumes the logits or features. It belongs to the same family as Mixup and CutMix and distillation's temperature-softened targets — all built on the idea that a hard 0/1 target is an overconfident lie.
Drag the ε slider and watch the target soften, the gradient flip sign, and the confidence ceiling fall toward real accuracy:
https://dev48v.infy.uk/dl/day36-label-smoothing.html
Top comments (0)