Every Transformer you have heard of normalizes its activations somewhere. For years that meant LayerNorm. Since 2019, and pretty much everywhere by now, it means RMSNorm instead. LLaMA, T5, Mistral, Gemma, PaLM all use it. It is the kind of change that sounds like a rounding error and turns out to be in every model.
Here is what LayerNorm does to a single token's feature vector. It computes the mean and the variance across the features, subtracts the mean (re-centering), divides by the standard deviation (re-scaling), then multiplies by a learned gain gamma and adds a learned bias beta:
y = gamma * (x - mean) / sqrt(var + eps) + beta
Two statistics, two learned vectors. RMSNorm looks at that and asks a blunt question: do we actually need the mean subtraction and the bias?
The paper that introduced it (Zhang and Sennrich, 2019) took LayerNorm apart and tested the pieces. Their finding: almost all of the benefit comes from the re-scaling, pinning the magnitude of the activations, and almost none from the re-centering. Subtracting the mean makes a vector invariant to a constant shift, but in a deep network with learned weights that invariance does not buy much. So drop it.
What is left is RMSNorm:
y = gamma * x / sqrt(mean(x^2) + eps)
No mean. No beta. Instead of variance, which is spread around the mean, it uses the root-mean-square, the raw magnitude of the vector: RMS(x) is the square root of the mean of the squared features. That is one reduction over the features instead of two, and one learned vector instead of two.
The neat part is how it relates to LayerNorm. When a vector already has zero mean, its variance equals the mean of its squares, so the standard deviation equals the RMS, and the two normalizers compute exactly the same thing (with beta = 0). A few layers into a network, and after any learned linear projection, activations tend to sit near zero mean anyway. So in the regime that dominates, RMSNorm hands you LayerNorm's answer for less work.
When the input is not zero-mean, the two diverge, and the divergence tells you exactly what each one does. Take a vector, add a constant offset to every element, and normalize. LayerNorm erases the offset: its output is the same as before you added it, and the features sum to zero. RMSNorm does not subtract anything, so the offset survives; it folds into the denominator and the output keeps its shifted direction. I checked this numerically while building the demo, and it is clean: LayerNorm is invariant to the offset, RMSNorm is not. That is the whole difference in one experiment.
So why switch? Two reasons, both about cost. First, operations: no mean pass and no per-feature subtraction. Second, parameters: gain only, no bias, so the normalizer carries half the learned parameters. Per layer that is nothing. But an LLM runs these norms on huge activation tensors, twice per layer, across dozens of layers, for every token, so it adds up. The original paper reports up to a 64% run-time cut on some RNN tasks and single-digit percentages on Transformers. The clincher is that quality does not drop; trained models match LayerNorm, so you take the cheaper option and never look back.
A few practical notes. The eps inside the square root is not optional. Leave it out and a near-zero vector makes you divide by zero and training goes to NaN. LLaMA uses 1e-6. Also compute the mean-of-squares in float32 even in a bf16 model, then cast back, because the squared sum is where precision dies. And RMSNorm lives in the pre-norm slot: you normalize the input of each sub-layer inside the residual branch, as in x + Attn(Norm(x)), never the residual stream itself, which is what keeps very deep stacks trainable. There is a final RMSNorm before the output head too.
RMSNorm is not always the right call. If your input carries a large, meaningful offset that the layer should not be forced to keep, LayerNorm's subtraction is the tool that removes it. But for the centered activations inside a modern Transformer, RMSNorm is cheaper, simpler, and just as good, which is why it won.
I built an interactive version where you drag the six features and watch both normalizers do their arithmetic side by side, with a 10-step walkthrough and the from-scratch code:
Top comments (0)