Part 9 of my "revisiting my AI/ML notes" series, and the last one in the backpropagation arc. The previous post covered gradients shrinking to nothing across many layers. This one covers the mirror-image problem: gradients that grow instead, until training becomes unstable.
The same chain rule, the opposite direction
Vanishing gradients happen when the per-layer terms in the chain rule product are consistently less than 1. Exploding gradients happen for the exact opposite reason: when those per-layer terms — usually driven by weight values — are consistently greater than 1.
gradient_at_early_layer ~= 2.5 x 2.5 x 2.5 x ... (once per layer)
Instead of shrinking toward zero, this product grows exponentially with depth. A gradient that should be a small, well-behaved correction instead becomes enormous by the time it reaches the earlier layers.
Why this makes training unstable
Recall the update rule again: w_new = w_old - lr * gradient. If the gradient is huge, the weight update is huge too — even a modest learning rate can't fully compensate for a gradient that's grown by several orders of magnitude. In practice this looks like:
- Loss swinging wildly between iterations instead of decreasing smoothly
- Weights growing to extremely large values, sometimes overflowing into
NaN - Training that was progressing fine suddenly diverging with no clear warning
Common fixes
A few standard techniques exist specifically to keep this under control:
- Gradient clipping — cap the gradient's magnitude at a fixed threshold before applying the update, regardless of how large the raw computed value is.
- Careful weight initialization — techniques like Xavier/Glorot or He initialization choose starting weight scales specifically to avoid values that compound into explosion (or vanishing) across layers.
- Batch normalization — normalizing activations between layers keeps values in a consistent, well-behaved range, which indirectly keeps gradients from spiraling in either direction.
Wrapping up the backpropagation arc
Across these last few posts: forward propagation produces a prediction, the loss measures how wrong it is, the chain rule propagates that error backward through every layer, and gradient descent uses it to update every weight. Vanishing and exploding gradients are what happens when that backward chain of multiplications isn't kept in a healthy range — too small and early layers stop learning, too large and the whole network becomes unstable. Every architecture that comes after this in a deep learning curriculum — CNNs, RNNs, transformers — is still built on this exact same core loop underneath.
Original Notes:
Top comments (0)