Softmax, the last step of almost every classifier, explained with three numbers.
When a neural network looks at a photo and decides "cat," it doesn't jump straight to an answer. Its last layer produces a raw score for every possible class. Those scores are useful, but they aren't probabilities yet.
Softmax is the small function that turns them into probabilities. It takes two steps, and you can do both by hand.
Raw scores are not probabilities
Say a network has three classes: cat, dog, and bird. For one image, its last layer gives these raw scores (often called logits):
- Cat: 2
- Dog: 1
- Bird: 0
Cat clearly scores highest. But these numbers have two problems if you want to read them as probabilities. They can be negative, and they don't add up to 1. A score of 2 doesn't tell you "how sure" the network is.
The network's last layer gives one raw score per class: cat 2, dog 1, bird 0.
Step 1: take the exponential
First, softmax raises the number e (about 2.718) to the power of each score. This is called taking the exponential.
- Cat: e² ≈ 7.4
- Dog: e¹ ≈ 2.7
- Bird: e⁰ = 1
The exponential has a useful property: its result is always positive, even when the score is negative. A score of −3 becomes e⁻³ ≈ 0.05, small but still positive. It also stretches the gaps, so higher scores pull further ahead.
Step 1: the exponential turns 2, 1 and 0 into about 7.4, 2.7 and 1. Every result is positive.
Step 2: divide by the total
Next, add the three results: 7.4 + 2.7 + 1 ≈ 11.1. Then divide each one by that total.
- Cat: 7.4 ÷ 11.1 ≈ 67%
- Dog: 2.7 ÷ 11.1 ≈ 24%
- Bird: 1 ÷ 11.1 ≈ 9%
Together, they add up to 100%. Now the numbers are real probabilities: all positive, and they sum to one.
As a formula, for a class with score zᵢ: softmax(zᵢ) = e^(zᵢ) ÷ (the sum of e^(z) over all classes).
Step 2: divide each exponential by the total. Cat gets 67%, dog 24%, and bird 9%, adding up to 100%.
Why "soft" max?
A hard max would simply pick the winner and give it everything: cat 100%, dog 0%, bird 0%. That throws away information about how close the race was.
Softmax is the soft version. The winner gets the biggest share, but the others keep a share too. That matters during training, because the network can learn from how confident it was, not just whether it was right.
A hard max would give cat everything and the others nothing. Softmax keeps a share for every class.
Quick check
Add 10 to every score: cat 12, dog 11, bird 10. Do the probabilities stay the same, or does cat's share grow?
Answer: they stay the same. Adding 10 multiplies every exponential by the same amount, e¹⁰. That common factor cancels when you divide by the total. Only the differences between scores matter.
Key takeaways
- A classifier's last layer outputs raw scores, which can be negative and don't sum to one.
- Softmax takes the exponential of each score, which makes every value positive.
- Then it divides by the total, so the results add up to 100%.
- Scores of 2, 1 and 0 become about 67%, 24% and 9%.
- Shifting every score by the same amount doesn't change the result. Only the gaps matter.
This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.
Top comments (0)