Part 8 of my "revisiting my AI/ML notes" series. The last post ended on a warning: multiplying several small numbers together, as the chain rule does, shrinks the result fast. This post is about what happens when that shrinking gets out of hand — the vanishing gradient problem.
Where the shrinking comes from
Sigmoid is the culprit here. Its derivative — the local gradient the chain rule multiplies at every layer it passes through — has a maximum value of exactly 0.25, and drops toward zero for large positive or negative inputs.
Now apply the chain rule across a deep network. If every layer contributes a local derivative that's at best 0.25 (and realistically often smaller, once you multiply in typically-small weight values too), then the gradient reaching a weight near the input is the product of many of these small numbers, layer after layer:
gradient_at_layer_1 ~= 0.25 x 0.25 x 0.25 x ... (once per layer)
That product shrinks toward zero shockingly fast as depth increases.
[PASTE DIAGRAM IMAGE HERE - SECOND HALF]
The practical consequence
Recall the weight update rule: w_new = w_old - lr * gradient. If the gradient is effectively zero by the time it reaches an early layer, the update is effectively zero too — that weight barely changes, no matter how many training iterations run. The layers closest to the input stop learning, while the layers closest to the output keep updating more or less normally. The network ends up with a large chunk of its capacity permanently under-trained.
This is a genuinely serious problem for deep networks — it's part of why very deep sigmoid-based networks were historically hard to train, and it's a big part of why ReLU (covered earlier in this series) became the default for hidden layers: ReLU's derivative is either 0 or exactly 1, so it doesn't shrink the gradient the way sigmoid does for positive inputs.
Why this matters
Vanishing gradients are a direct, mechanical consequence of the chain rule and the shape of sigmoid — not a mysterious training instability. Once you see it as "multiplying many numbers less than 1 together," several standard deep learning fixes suddenly make sense: switching activation functions, using residual/skip connections, and normalization techniques all exist specifically to keep this product from collapsing to zero.
Next up: the opposite failure mode — what happens when that same chain of multiplications grows instead of shrinks.
Original Notes:
Top comments (0)