You're training a deep or recurrent net, everything looks fine, and then the loss prints nan and never recovers. Lowering the learning rate delays it but doesn't cure it. That's almost always an exploding gradient, and it has a one-line fix. I built a demo with a real 4-unit linear RNN doing genuine back-prop-through-time on a deliberately explosive setup — with clipping off the gradient norm and loss rocket to NaN in a few steps; switch clip-by-norm on and the same net stays finite and converges. Every norm, clip, and direction angle is computed live. Here's what's going on.
Why gradients explode: repeated multiplication
Back-prop computes the gradient by the chain rule, which multiplies the local Jacobian of every layer — or every time-step, in an RNN. Multiplying many numbers is a runaway process: if the factors average above 1, the product grows geometrically. For a net unrolled over T steps the gradient carries a factor like Wᵀ; if W's spectral radius ρ (its largest eigenvalue) exceeds 1, that factor grows like ρᵀ. With ρ=1.3 and T=15 that's already ~50×.
Geometrically, that's a steep cliff in the loss surface. On the flat plateau before the wall the gradient is gentle; at the wall it's enormous. Gradient descent multiplies that huge gradient by the learning rate and takes a proportionally huge step — so instead of climbing carefully down the wall, the weights get catapulted to a random terrible point, the next forward pass overflows to inf, and inf − inf = NaN, which then spreads to every parameter.
The fix: cap the magnitude, keep the direction
Clipping accepts that the direction of the gradient is still useful — it's the magnitude that's pathological. It runs after backward() fills the gradients and before opt.step() uses them. There are two flavours.
Clip by value clamps every element into [−c, c]:
torch.nn.utils.clip_grad_value_(params, clip_value=1.0)
# g_i <- max(-c, min(c, g_i)) for every element
It never NaNs, but clamping components by different amounts changes their ratios, so the update no longer points along true steepest descent. A blunt but effective seatbelt.
Clip by norm looks at the whole vector's length. If ‖g‖ is under the threshold τ, leave it alone; if it's over, rescale the entire vector by τ/‖g‖:
torch.nn.utils.clip_grad_norm_(params, max_norm=1.0)
# if ‖g‖ > τ: g <- g * (τ / ‖g‖) # same direction, length = τ
# else: g unchanged
Why norm-clipping doesn't bend the step
This is the crux. Multiplying a vector by a positive scalar α = τ/‖g‖ cannot change where it points — only its length. The angle between the clipped gradient and the original is exactly 0°, so clip-by-norm still steps in the true steepest-descent direction, just with a bounded stride. Clip-by-value, clamping components independently, generally rotates the update by a real non-zero angle. In the demo you can watch this directly: the direction-change readout shows 0.00° for norm and tens of degrees for value. That's why clip-by-norm is the default for RNNs and transformers.
One more subtlety the library handles: clip_grad_norm_ doesn't clip each tensor separately — it concatenates every parameter's gradient into one giant vector, takes that single global norm, and rescales everything by one shared factor, preserving the direction of the whole update. Handily it returns the pre-clip total norm — the perfect thing to log.
Choosing τ — measure, don't guess
τ is a real hyperparameter. Log the total grad-norm over a few hundred steps of a stable run, look at the distribution, and set τ a bit above the typical value so it only bites the rare spikes:
tau = np.quantile(norms, 0.90) # clip the top ~10%
Too low and you throttle every honest update (slow, biased training); too high and the cliffs sail through. Good defaults: ~1.0 for transformers/LLMs, 5–10 for RNNs/LSTMs. (With mixed precision, remember to unscale_ before you clip, or you'll clip the wrong magnitude.)
And keep it in perspective: clipping treats the symptom. The real cures for exploding gradients are good initialisation, normalisation (Batch/Layer/RMSNorm), residual connections, and gated units (LSTM/GRU). Use those to reduce the spikes — and keep the clip as a cheap, always-on seatbelt for the ones that still get through.
Train the real RNN, drag τ, and watch it NaN then recover:
https://dev48v.infy.uk/dl/day44-gradient-clipping.html
Top comments (0)