This is what a confident mistake looks like

Cross-entropy, the loss behind almost every classifier, as a landscape and as a bet.

A classifier doesn't just pick an answer. It gives every class a score, and softmax turns those scores into probabilities: cat, dog, bird. To learn, the model needs one number that says how good those probabilities were. For classifiers, that number is almost always the cross-entropy loss.

One number, one log

Cross-entropy looks at a single probability: the one the model gave to the right answer. Call it p. The loss is minus its log:

loss = −log(p)

If the picture is a cat and the model said 67% cat, the loss is −log(0.67) ≈ 0.4. If it said 1% cat, the loss is −log(0.01) ≈ 4.6.

The loss as a landscape

Take a classifier with three classes, and hold the bird score still. Two numbers are left: the cat score and the dog score. For every pair, cross-entropy gives a loss. Make that loss the height, and you get a landscape.

The loss over the cat and dog scores: a landscape. The ball is a confidently wrong model: cat 1%, loss 4.61.

Where the cat score wins, the ground is flat and close to zero. A model that's confident and correct sits there at almost no cost: at 98% cat, the loss is 0.02.

Where the cat score wins, the ground is flat and close to zero.

Where the dog score wins, the ground climbs, and it never levels off. Far enough up that ramp the loss grows by one for every extra point of dog score. There is no flat spot for a confident mistake.

Where the dog score wins, the ground keeps climbing.

Training rolls downhill

Training follows the slope. For softmax with cross-entropy, the slope with respect to each score is just the prediction minus the truth, where the truth is 1 for the right class and 0 for the others. So each training step moves the scores by that amount, times a learning rate:

z ← z − (p − y)

Start at the confident mistake, cat 1% and dog 99%, with a learning rate of 1. Three steps take the cat from 1% to 7%, to 32%, to 64%, and the loss falls from 4.61 to 0.44. The ramp is steep exactly where the model is most wrong, so the push back is strongest there.

Three real gradient-descent steps from the top of the ramp: cat 1% → 64%.

The same loss as a bet

Here's another way to see it. Think of the model's probabilities as a bet: 100 chips spread across the answers, 67 on cat, 24 on dog and 9 on bird. When the answer turns out to be a cat, cross-entropy sends a bill: minus the log of the cat's share, −log(67/100) ≈ 0.4.

The model bets 100 chips. It's a cat, so the bill is −log(0.67) ≈ 0.4.

Halve the chips on the truth, and the bill grows by the same amount every time, log(2) ≈ 0.69:

• 67 chips cost 0.40;
• 34 chips cost 1.09;
• 17 chips cost 1.79;
• one chip costs 4.61;
• zero chips: there is no limit.

Every halving adds 0.69. One chip costs 4.6; at zero, the bill has no limit.

That's why a confident mistake costs so much: being sure of the wrong answer means betting almost nothing on the truth, and every halving of that bet keeps adding to the bill.

Try it in PyTorch

You never write the log yourself. nn.CrossEntropyLoss takes the raw scores and the right class and does the softmax and the log for you:

import torch
from torch import nn
z = torch.tensor([[2., 1, 0]])
y = torch.tensor([0])  # cat
print(nn.CrossEntropyLoss()(z, y))

It prints tensor(0.4076), the 0.4 from above. Pass it probabilities instead of raw scores and it applies softmax a second time, and the loss comes out wrong.

Key takeaways

• Cross-entropy = −log(probability of the right answer).
• Over the scores it is a landscape: flat where the model is right, a ramp with no top where it is confidently wrong.
• Its slope is prediction minus truth, so training pushes hardest where the model is most wrong.
• As a bet: every halving of the chips on the truth adds 0.69, and zero chips has no limit.

---

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.