Train a one-stage dense detector like RetinaNet and you hit a brutal imbalance: every image floods the classifier with ~10⁴–10⁵ candidate boxes, almost all of them easy background the model already gets right. Under plain cross-entropy each of those easy negatives still contributes a small loss — and there are so many of them that their sum swamps the handful of hard, informative foreground examples. The gradient ends up dominated by examples that teach the model nothing. Focal loss fixes exactly that with one multiplicative factor, and I built a demo that computes the whole thing live. Here's the idea.
Cross-entropy is never zero, and that's the problem
Everything is written in terms of p_t, the probability the model assigned to the correct class. Big p_t → 1 means an easy example (right and confident); small p_t → 0 means a hard one (the model is wrong). Cross-entropy is just −log(p_t) — and even a well-classified example at p_t = 0.9 still contributes −log(0.9) ≈ 0.105.
ce(0.9) # 0.105 <- tiny, but there are ~100k of these
ce(0.25) # 1.386 <- the example that actually matters
# 1000 * 0.105 = 105 >> 20 * 1.386 = 27.7
# easy examples win the gradient by sheer count
The modulating factor down-weights by difficulty
Multiply CE by (1−p_t)^γ. When the example is hard (p_t → 0) the factor → 1 and the loss is untouched. When it's easy (p_t → 1) the factor → 0 and the loss is crushed. The exponent γ ("focusing") sets how sharply.
def focal(pt, gamma):
return -(1.0 - pt) ** gamma * math.log(max(pt, 1e-12))
focal(0.9, 2) # 0.00105 easy -> 100x smaller than CE
focal(0.25, 2) # 0.780 hard -> only ~1.8x smaller
focal(0.9, 0) # 0.105 gamma=0 IS cross-entropy
Note γ=0 recovers plain CE exactly — focal loss is a strict generalisation, not a different loss. RetinaNet uses γ=2, which down-weights a p_t=0.9 example by 100× while barely touching a p_t=0.1 one.
α-balancing is orthogonal — it weights the class
On top of focusing, RetinaNet keeps a per-class weight α_t (e.g. 0.25 for the rare foreground). It's a flat scale that addresses class frequency, not example difficulty — so α slides the totals up or down but never changes the easy/hard split. Only γ does that. The two stack: FL = −α_t (1−p_t)^γ log(p_t).
Watch the gradient flip
The clearest way to see it is to aggregate a population and compare each group's share of total loss (which tracks the gradient):
def share(gamma):
easy = {'n':1000, 'pt':0.90} # model already right
hard = {'n': 20, 'pt':0.25} # model wrong
L = lambda o: o['n'] * (1-o['pt'])**gamma * -math.log(o['pt'])
e, h = L(easy), L(hard)
return 100*e/(e+h), 100*h/(e+h)
share(0) # (79.2, 20.8) CE: the easy crowd wins on volume
share(2) # ( 6.3, 93.7) focal: FLIPPED — hard examples own the gradient
What you'd actually ship
Real code never forms p_t then takes a log — it computes sigmoid/BCE from logits with a stable kernel and multiplies by (1−p_t)^γ. Torchvision ships this as sigmoid_focal_loss, and RetinaNet pairs γ=2, α=0.25 with a bias-init trick: start the classifier's output bias at −log((1−π)/π) with π≈0.01, so training doesn't explode from the massive negative majority in epoch one.
def sigmoid_focal_loss(logits, targets, alpha=0.25, gamma=2.0):
p = torch.sigmoid(logits)
ce = F.binary_cross_entropy_with_logits(logits, targets, reduction="none")
p_t = p * targets + (1 - p) * (1 - targets)
loss = ce * (1 - p_t) ** gamma
a_t = alpha * targets + (1 - alpha) * (1 - targets)
return (a_t * loss).mean()
It's distinct from label smoothing (which softens the targets to curb overconfidence) — focal loss keeps hard targets and reweights the per-example loss by difficulty. One line, and it's what made single-stage detectors match two-stage accuracy at one-stage speed.
Drag γ and the probe, and watch the loss curves peel apart and the easy/hard bars flip:
https://dev48v.infy.uk/dl/day47-focal-loss.html
Top comments (0)