Part 12 of my "revisiting my AI/ML notes" series. Every post so far in this series has assumed weights already have reasonable values. This one covers the step before any of that: how weights get their starting values in the first place, and why that choice matters more than it sounds like it should.
Why initialization isn't a minor detail
There are a few hard constraints on a good initialization scheme:
- Weights should be small, but not vanishingly small — too small, and the earlier posts in this series on vanishing gradients explain exactly what goes wrong.
- Weights should not all be the same value. If every neuron in a layer starts identical, they all compute the same gradient during backpropagation and stay identical forever — the layer never actually differentiates into useful, distinct feature detectors.
- Weights need good variance — enough spread that different neurons can learn different things, without being so spread out that outputs blow up as they pass through many layers (the exploding gradient problem, also covered earlier).
Too-small variance is functionally the same starting point as the vanishing gradient problem — the signal barely moves through the network. Too-large variance heads straight toward the exploding gradient problem. The right initialization scheme depends on which activation function the layer uses.
Xavier / Glorot initialization
Designed for sigmoid and tanh activations. Two common variants:
Xavier Normal: W ~ N(0, sigma^2), where sigma^2 = 2 / (fan_in + fan_out)
Xavier Uniform: W ~ Uniform[-limit, limit], where limit = sqrt(6 / (fan_in + fan_out))
Here fan_in is the number of input connections to a neuron, and fan_out is the number of output connections. Balancing the initialization around both keeps the variance of activations (and gradients) roughly consistent as they flow forward and backward through the network — which is specifically what sigmoid and tanh, with their easily-saturating curves, need to avoid vanishing gradients from the very first forward pass.
He initialization
Designed for ReLU and its variants (Leaky ReLU, ELU, PReLU — all covered in the previous post):
He Normal: W ~ N(0, sigma^2), where sigma^2 = 2 / fan_in
He Uniform: W ~ Uniform[-limit, limit], where limit = sqrt(6 / fan_in)
He initialization only accounts for fan_in, using a larger variance than Xavier. This compensates for the fact that ReLU zeroes out roughly half its inputs (everything negative) — a larger starting variance keeps the surviving signal at a healthy scale.
Why this matters
Picking Xavier for a ReLU network, or vice versa, isn't a fatal mistake, but it does make training measurably harder — slower convergence, or a higher chance of hitting vanishing/exploding gradients early, before the optimizer has had a chance to correct course. Matching the initialization scheme to the activation function is one of the cheapest, most reliable wins available before training even starts.
Original Notes:
Top comments (0)