Normalization layers are usually taught with a picture of four cubes with different faces shaded. The picture is fine. It is also unnecessary, because the whole family is one function with a different axis list.
BatchNorm normalize over (N, H, W) per channel
LayerNorm normalize over (C, H, W) per sample
InstanceNorm normalize over (H, W) per sample, per channel
GroupNorm normalize over (C/G, H, W) per sample, per group
Everything else follows from that line. Group Normalization (Wu & He, 2018) sits between LayerNorm and InstanceNorm, and the two extremes are literally special cases:
G = 1 -> LayerNorm
G = C -> InstanceNorm
Not "similar to". Identical, and the page checks it numerically across five shapes: max difference 0.0e+0.
Every number here is computed live: https://dev48.infy.uk/dl/day64-group-normalization.html
The property that made GroupNorm matter
BatchNorm's statistics come from the batch. So the output for one image depends on which other images happened to be next to it.
Push the same sample through batches of 1, 2, 4, 8, 16, 32 and measure the drift:
| worst drift | |
|---|---|
| GroupNorm | 0.0e+0 |
| LayerNorm | 0.0e+0 |
| InstanceNorm | 0.0e+0 |
| BatchNorm | 6.714 |
And the leak, directly: change one image in the batch, measure how much the others move. GroupNorm 0.0e+0. BatchNorm 1.477.
This is why detection and segmentation moved off BatchNorm. Not fashion — those models run at batch size 2 per GPU because the inputs are enormous, and at batch size 2 the statistics are noise.
The thing I had wrong
My page originally repeated the folklore: at batch size 1, BatchNorm output is identically zero — one sample, so the mean is the sample, so everything cancels.
That is false whenever you have spatial dimensions.
At N=1 with H×W > 1, BatchNorm still averages over (N, H, W) — which is now just (H, W). It has silently become InstanceNorm, verified to 0.0e+0. Your model still trains. It is simply not running the layer you think it is.
The collapse everyone quotes needs N=1 and H·W=1 — a fully-connected layer at batch size 1. Only then is there one number per channel, and only then is the input erased (max |v| = 0.0e+0, also measured).
Two different claims, one of them true, and the false one is the one repeated. The measurement is what separated them.
The implementation is nine lines
def group_norm(x, gamma, beta, G, eps=1e-5):
N, C, H, W = x.shape
x = x.reshape(N, G, C // G, H, W)
mean = x.mean(axis=(2, 3, 4), keepdims=True)
var = x.var(axis=(2, 3, 4), keepdims=True)
x = (x - mean) / np.sqrt(var + eps)
x = x.reshape(N, C, H, W)
return x * gamma.reshape(1, C, 1, 1) + beta.reshape(1, C, 1, 1)
Note what is not there: no running statistics, no train/eval divergence, no batch axis anywhere in the reduction. GroupNorm behaves identically in training and inference, which removes a whole category of "works locally, fails in prod" bug.
Checks worth keeping: per-group mean 4.9e-17 and variance within 1.1e-5 of 1 (ε accounts for the rest); rescaling one group leaves every other group bit-identical; every divisor of C is a legal G and all of them produce well-formed output.
Choosing G
G=32 is the paper's default and a reasonable one. The real constraint is that channels inside a group should be comparable in scale — grouping a high-variance channel with a near-dead one lets the loud one dominate the statistics and squash the quiet one.
What I would keep
When a family of layers differs only by an axis list, write the identities as assertions. G=1 == LayerNorm and G=C == InstanceNorm are two lines of test that pin the whole design, and they would have caught the N=1 claim if I had written them first.
Part of a from-scratch series — one deep-learning idea a day, computed in-browser: https://dev48.infy.uk/deeplearningfromzero.php
Top comments (0)