In training, batch norm normalises each row with the statistics of the batch it happens to be in. At inference there is no batch, so it uses a running average collected during training.
Those are two different functions, and the layer swaps one for the other silently.
See it: https://dev48.infy.uk/dl/day78-batch-norm-at-inference.html
Every score is a real forward pass over an enumerated 512-row population, so the errors are exact means rather than estimates.
Same input, same weights, two answers
| train mode vs eval mode | RMSE 0.3219 |
| worst running-variance error | 5.36 on a single feature |
Bigger batches make inference worse
The intuition is that a larger batch gives a better estimate, so the running average should improve. The first half is true. The second does not follow — the running average is an exponential moving average over updates, and a bigger batch means fewer of them.
| batch size | updates | running stats vs truth |
|---|---|---|
| 8 | 64 | 0.1036 |
| 32 | 16 | 0.1754 |
| 64 | 8 | 0.4506 |
| 256 | 2 | 1.3771 |
13× worse at 256 than at 8. With momentum 0.9 and two updates, the initialisation still carries weight 0.81 — the running variance never leaves its starting value of 1, on features whose real variance is nowhere near 1.
Two things move in opposite directions as the batch grows, and the second wins. The page checks the first separately, so the mechanism is not assumed: a bigger batch really does estimate the mean better. The loss is entirely in the number of updates.
The knob that makes training more stable makes the thing training hands to production less accurate.
And the tests cannot see any of it
Here is everything a batch-norm implementation can assert about itself without a second implementation to compare against:
- the training output has mean ≈ β per feature
- and variance ≈ γ² per feature
- eval mode gives the same answer twice
- the output has the input's shape
Checked on shuffled data and on data sorted by its first feature — which is what you get whenever rows arrive grouped by class, by user, or by time.
All four pass in both. Meanwhile the train/eval divergence goes from 0.3219 to 0.6648 — it doubles — because sorted batches are not exchangeable, so each batch's statistics describe its own slice and not the data. Nothing in the four properties mentions the order of the rows.
60 verifier asserts, 14 in-page checks, 0 failures.
Top comments (0)