Entropy and Cross-Entropy, Explained

If you've spent any time around machine learning the familiarizing yourself with the concept, you have seen the term "entropy" and "cross entropy" show up everywhere. These two distinct terms often show up in topics regarding: classification models, neural networks, and understanding language models. But if you are like me a few weeks ago, you would not be able to write down the formulas without feeling the meaning of these terms.

This post is my attempt to build this intuition of yours from the ground up. This blog will cover topics ranging from entropy to cross-entropy. As you read this post, these topics will finally connect to why they are the loss functions of choice in machine learning.

PART 1: What Entropy Actually Measures

Forget about the neural networks for a second. Entropy comes from the information theory and, at its core, it answers one main question: How surprised should you be by the outcome of a random event?

Now let's image two coins:

• Coin A is fair with 50% heads and 50% tails.

• Coin B is rigged with 99% head and 1% tails.

If I flip Coin B and tell you the result, you will almost never be surprised if the outcome is heads. However, if I flip Coin A, you will have no idea what is coming. This is because every flip of Coin A is a genuine mystery that is only uncovered once the coin is flipped.

Entropy is a formal way of measuring this same "amount of surprise." A fair coin like Coin A has high entropy (maximum uncertainity) while a rigged coin like Coin B has low entropy (easily predictable).

Mathematically, the entropy for a probabilitiy distribution is: H(P)=−x∑​p(x)log(p(x)).

Now, lets unpack this formula and understand why it makes sense.

• log(p(x)) basically represents the "surprise." Rare events—small p(x)—have a more negative log value, meaning higher surprise rates. Common events—large p(x)—have log values that are closer to 0, meaning lower surprise rates.

• The negative sign flips to a positive sign, knows as a surprise score, since the probabilities are between 0 and 1 with the log being a negative.

• Multiplying by the p(x) value and summing it up gives you the expected value of the surprise. The expected value of surprise is the average surprise you'd experience sampling from this distribution over and over.

So basically: Entropy is the average amount of "surprise" backed into a distribution.

Part 2: From Entropy to Cross-Entropy

Entropy already assumes that you know the true distribution of the outcomes. However, in machine learning, we rarely know the true distribution. We are trying to predict the distribution.

Here is the mathematical definition of cross-entropy: H(P,Q)=−x∑​p(x)log(q(x)).

This is where cross-entropy plays a major role. It measures the surprise you experience when:

• The true outcomes come from distribution P (considered the actual label of an image).

• But you are using a predicted distribution Q (your model's guess) to measure that particular surpise.

Notice the subtle but crucial difference from entropy: we still weight by the true probabilitiy p(x), but we take the log of the predicted probability q(x).

Here is the intuition: if your model's predicted distribution Q is close to the true distribution P, then the cross-entropy will be low while the actual entropy is close to P. However, if the model is confidently wrong, the log term completely explodes and the cross-entropy shoots up.

This is exactly something what we want in a loss function:

• Confident and correct predictions are low loss.

• Confident and high predictions are high loss.

• Uncertain predictions are the moderate losses.

Part 3: Why Does This Matter For Neural Networks?

When a classifier is trained, the model puts out a probability distribution over classes. Cross-entropy loss compares that predicted distribution against the true label, usually a one-hot vector.

Since the true label is one-hot, the formula is properely simplified as almost every term in the sum becomes zero except for the one corresponding to the correct class. The equation goes like: Loss = -log(q(correct class)).

During the training, the model is just trying to push the predicted probability of the correct class as close to the value 1 as possible. This is because -log(1)=0 and -log(x) has x approaching 0.

Conclusion

I am a high school senior that is interested in learning ML from the ground up and this is one of the first concepts that I have thoroughly researched in detail. I hope this blog was helpful to other students and adults like me that have a fond interest in ML.