Ask someone how "good" a language model is and you get vibes. Ask a researcher and you often get one number: perplexity. It is the oldest, cheapest, most-used way to score a language model, and once you see how it is built you never forget it.
Start with what the model actually does
A language model reads the tokens so far and outputs a probability for every possible next token. Give it "the cat sat on the" and it might say mat 40%, floor 12%, rug 8%, and so on, summing to 1 across the whole vocabulary. That is the entire machine. Evaluating it comes down to a single question: how much probability did it put on the token that actually came next?
If the real next token was mat and the model gave it 0.40, that is a decent guess. If it gave mat only 0.02, the model was surprised. Perplexity is just the average surprise, packaged nicely.
Turn a guess into surprise
We measure surprise as -log P. When the model was confident and right (P near 1), the log is near 0, so surprise is near 0. When it assigned a tiny probability to what actually happened, -log P is large. Using log base 2 gives you "bits of surprise"; a token the model rated at 1-in-8 costs 3 bits.
Why the log at all? Two reasons. The probability of a whole sentence is the product of the per-token probabilities (chain rule). Multiply a few hundred numbers below 1 and you underflow to zero fast. Take logs and the product becomes a sum, which is stable for any length and easy to differentiate during training. Logs are monotonic, so nothing is distorted: a more likely sentence still scores higher.
The average is the training loss
Average that per-token surprise over the sentence and you get the cross-entropy. This is not a coincidence or a separate metric bolted on afterward. Cross-entropy, the average negative log-likelihood per token, is the exact loss function LLMs are trained to minimize. The number you optimize during pretraining and the number you evaluate with are the same thing.
Exponentiate to get perplexity
Cross-entropy is an abstract loss sitting in log space. Perplexity undoes the log to put it back on a human scale:
perplexity = exp(average negative log-likelihood) # natural log
= 2 ^ (average bits per token) # base 2
Both give the same value. Three bits of average surprise means a perplexity of 2^3 = 8.
What the number means
Here is the intuition that makes it click. A perplexity of 8 means the model was, on average, about as unsure as if it were choosing uniformly among 8 equally likely tokens at each step. A fair six-sided die has perplexity 6. Perfect prediction, where the model always knew the next token, is a perplexity of 1. And if your vocabulary has V tokens, blind random guessing gives perplexity V, the ceiling any real model has to beat. That is why people call it an effective branching factor.
So lower is better, unambiguously. Watching validation perplexity fall over training is the classic sign pretraining is working. A sudden rise flags a data bug, a distribution shift, or overfitting.
Two traps
Perplexity is measured per token, so it depends entirely on how the text was split. A model with a bigger vocabulary chops text into fewer, longer, easier-to-predict tokens and can post a lower perplexity without being any smarter. Never compare perplexities across different tokenizers. When you need a fair cross-model comparison, normalize to the raw text instead: bits-per-byte (total bits divided by UTF-8 byte count) does not move with the tokenizer, which is why compression-style leaderboards use it.
The bigger trap is what perplexity does not measure. It scores how well the model fits the distribution of the text, its fluency. It says nothing about whether the output is true, useful, on-task, or safe. A model can be confidently, fluently wrong and still post a great perplexity. Use it as an intrinsic training signal, then gate anything user-facing on downstream benchmarks and human or LLM judgement.
See it move
I built an interactive version. Type a fluent sentence and watch the perplexity sit low. Type a garbled one and watch it spike. Flip between a confident model and an uncertain one and see how the same sentence scores differently. Every probability, bit, and perplexity on the page is real code running the actual exp(mean -ln p) math, token by token.
https://dev48v.infy.uk/ai/days/day32-perplexity.html
Part of AIFromZero. One AI idea a day, built from scratch. https://dev48v.infy.uk
Top comments (0)