Batch normalisation is one of the most widely used tricks in deep learning, and one of the most misunderstood. It was introduced in 2015 as a fix for "internal covariate shift", and that explanation was repeated in tutorials for years. The trouble is that the explanation is probably wrong, and the real reason batch normalisation works is more interesting. This article walks through the algorithm, the maths, what the evidence actually says, and the traps that still catch experienced practitioners.
The algorithm
Take one layer of a network and look at the activations it receives over a mini-batch of size m. Batch normalisation standardises those activations before passing them on, then re-scales and re-shifts them with two learnable parameters.
First, compute the batch mean and variance:
Then normalise each activation:
Finally, apply a learnable affine transform:
The small constant exists only for numerical stability. The parameters and are learned alongside the weights. They matter: without them, normalisation would force every layer's output to have zero mean and unit variance, which would destroy representational power. With them, the network can recover the identity transform by learning and if that is what the loss wants. Normalisation constrains the optimisation, it does not constrain the function class.
During training, the mean and variance come from the current mini-batch, and the gradients flow through them. During inference there is no batch, so the layer uses running averages of the batch statistics accumulated during training. That train-test difference is the source of most batch normalisation bugs, and we return to it below.
The old story: internal covariate shift
The original paper argued that as the weights of early layers change during training, the distribution of inputs to later layers keeps shifting, and later layers must constantly re-adapt. Normalising each layer's inputs was supposed to reduce this "internal covariate shift" and let each layer learn against a stable input distribution.
It is a tidy story, and it has a problem: it does not survive experiment. Santurkar and colleagues (2018) tested it directly. They injected extra covariate shift into a network trained with batch normalisation, by adding noise to the activations after normalisation, and the network still trained well. Conversely, they built a variant that reduced covariate shift without normalising, and it did not get the training benefit. If the benefit came from stabilising the input distribution, neither result should have happened.
The lesson is not that batch normalisation fails. It is that the mechanism behind its success is different from the one in its own paper, and understanding the real mechanism changes how you use it.
What the evidence actually says: a smoother loss landscape
The Santurkar paper's central finding is geometric. Batch normalisation makes the loss landscape smoother in two precise senses: the loss becomes more Lipschitz, and its gradients become more Lipschitz, meaning the gradient itself changes more slowly as you move through parameter space. A function is L-smooth when its gradient satisfies:
A smaller L means the gradient is more predictable, which means a gradient step taken from the current point is more likely to still point downhill after you take it. That is exactly what lets you use larger learning rates with batch normalisation: the step size that would overshoot on a jagged landscape is safe on a smooth one.
There is a clean way to see why. Normalising the activations makes the loss invariant, to first order, to rescaling the weights of the preceding layer. If you multiply a weight matrix by a constant, the normalisation divides it back out. Directions in parameter space that would previously have changed the loss sharply now change it hardly at all, which flattens the landscape along those directions. The gradient also becomes better behaved: its magnitude is bounded in terms of the normalised activations rather than the raw ones, so large activations no longer produce exploding gradients through this layer.
This also explains a second, often overlooked benefit. Batch normalisation reduces the dependence of training on the initialisation. Poorly scaled initial weights still produce normalised activations, so the first few steps of training are far less fragile. That is why it is so effective in very deep networks, where activation magnitudes would otherwise drift layer by layer.
The regularisation side effect
Batch normalisation adds noise. Each example's normalised value depends on the other examples in its mini-batch, so the same input produces slightly different activations depending on its batch-mates. That noise acts as a mild regulariser, similar in spirit to dropout, and it is one reason networks with batch normalisation sometimes need less explicit regularisation.
But it is an unreliable regulariser. Its strength depends on batch size: large batches make the statistics nearly deterministic, and the noise disappears. Small batches make the statistics noisy, which regularises more but also makes the gradient estimates noisier. Treating batch normalisation as your regularisation strategy is a mistake. Use it for optimisation, and regularise deliberately.
Train versus test: the hidden trap
The running statistics are the part most people get wrong. During training, the layer maintains exponential moving averages of the batch mean and variance. At test time it uses those averages instead of the batch statistics. Two failure modes follow.
First, if you forget to switch the model to evaluation mode, the layer keeps using batch statistics at test time. A single test example gets normalised by statistics computed from itself alone, which collapses its activations to zero mean. Predictions become garbage, and the bug is silent: no error is raised, the loss just looks wrong. The classic symptom is a model that trains well and tests terribly.
Second, the running statistics are only good if the training batches are representative. If the data distribution drifts between training and deployment, or if training uses a different batch composition than the deployment data, the frozen running statistics are wrong and every prediction is systematically shifted. This is one more reason to monitor deployed models rather than assuming training metrics transfer.
A minimal PyTorch example shows both the usage and the trap:
import torch
import torch.nn as nn
model = nn.Sequential(
nn.Linear(128, 256),
nn.BatchNorm1d(256),
nn.ReLU(),
nn.Linear(256, 10),
)
model.train() # batch statistics; running stats update
for x, y in train_loader:
loss = loss_fn(model(x), y)
loss.backward()
optimiser.step()
model.eval() # running statistics; no updates
with torch.no_grad():
preds = model(x_test)
Forgetting model.eval() before inference is the single most common batch normalisation bug in production code.
When batch normalisation breaks
Batch normalisation assumes the batch is a reasonable sample of the data distribution, and that assumption fails in several common settings.
Small batches are the first. With a batch of 4 or 8, the mean and variance estimates are noisy, the gradient noise grows, and training can destabilise. This is why batch normalisation is rarely used in object detection or segmentation, where memory limits force small batches, and why alternatives were developed.
Non-independent batches are the second. In contrastive learning or recurrent models, examples within a batch are often correlated, for example multiple crops of the same image or consecutive frames. The batch statistics then reflect the correlation structure rather than the data distribution, and the normalisation leaks information between examples. In contrastive learning this leakage can even let the model cheat, since each example's representation contains information about the others.
Variable-length sequences are the third. Normalising across a batch of sequences with padding mixes real tokens with padding tokens, unless the padding is handled explicitly. This is a large part of why transformers use LayerNorm instead: normalising across the feature dimension of a single example avoids all batch-level coupling.
LayerNorm, RMSNorm and GroupNorm
The alternatives all move the normalisation axis. LayerNorm normalises across the features of a single example, so it is independent of batch size and works identically at train and test time. That is why it dominates in transformers and recurrent models. RMSNorm drops the mean-centring and normalises by the root mean square alone, which is cheaper and works just as well in practice for large language models. GroupNorm splits the channels into groups and normalises within each group, giving a middle ground that works well for vision models with small batches.
None of these is universally better. BatchNorm still wins when batches are large, independent and identically distributed, which covers much of image classification. The right choice follows from the structure of your data and your batch, not from fashion.
Practical checklist
- Use batch normalisation when your batches are large (32 or more) and your examples are independent.
- Always call
model.eval()before inference, and double check it in any serving code path. - Do not rely on it as your only regulariser. Its noise vanishes at large batch sizes.
- Watch the running statistics when the data distribution can drift. Stale statistics silently shift every prediction.
- If your batches are small, correlated or padded, prefer GroupNorm or LayerNorm.
- Keep the learning rate high enough to use the benefit. The smoother landscape is what lets you train faster; a tiny learning rate wastes it.
Takeaway
Batch normalisation does not work because it stabilises the distribution of layer inputs. It works because it smooths the loss landscape, bounds the gradients, and makes the optimisation far less sensitive to scale and initialisation. The algorithm is three lines of maths. The understanding that matters is the geometry behind it, the train-test split in its statistics, and the batch assumptions it quietly depends on. Get those right and it is one of the most reliable tools in deep learning. Get them wrong and it fails silently, in exactly the way that looks like a data problem.
Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial
Top comments (0)