Under Adam, Your L2 Penalty Is Divided by the Gradient. The Biggest Gradients Get the Least Decay

Two things everyone treats as synonyms:

L2 regularisation adds a penalty to the loss.

L'(w) = L(w) + (λ/2)‖w‖²     ⟹     ∇L'(w) = ∇L(w) + λw

Weight decay does not touch the loss. It shrinks the weight as part of the update.

w ← (1 − lr·λ)·w  −  lr·∇L(w)

Expand both under plain SGD and they are the same update. That is why the terms became synonyms, why frameworks named the L2 knob weight_decay, and why nobody looked again for twenty years.

Everything below computed live: https://dev48.infy.uk/dl/day65-adamw.html

The identity, asserted to the bit

Before claiming they diverge under Adam, the page pins the part that must be true:

a = a - lr * (grad(a) + lambda * a);        // L2 into the gradient
b = b * (1 - lr * lambda) - lr * grad(b);   // decay onto the weight
assert(Math.abs(a - b) < 1e-12);            // every step, 400 steps

400 steps × 4 lambdas × 3 learning rates, bit-identical. If that fails, the whole article is half wrong.

Adam breaks it

Adam does not apply the gradient. It applies the gradient divided by a running estimate of its own magnitude:

w ← w − lr · m̂ / (√v̂ + ε)

Put the L2 term in the gradient and it goes through that division too:

effective decay ≈ lr · λ·w / (√v̂ + ε)

The decay is now divided by the gradient magnitude. A parameter with large, consistent gradients has a large √v̂ and gets less regularisation. A parameter that barely moves gets more.

Measured on a problem with a 30× gradient-scale spread:

• proportional decay under AdamW: spread 1.000000× across parameters
• proportional decay under Adam + L2: spread 5,368×
• small-gradient parameters decayed 115× more than large-gradient ones

That is exactly backwards from what regularisation is for.

The fix is one line moved

function step(p, g, s, cfg){
  if (cfg.decoupled === false) g = g + cfg.lambda * p;    // L2: into the GRADIENT
  // ... Adam's moments, bias correction, adaptive step ...
  let next = p - cfg.lr * mHat / (Math.sqrt(vHat) + cfg.eps);
  if (cfg.decoupled === true) next -= cfg.lr * cfg.lambda * p;   // decay: onto the WEIGHT
  return next;
}

Two ifs around identical arithmetic. That is genuinely the whole difference between Adam(weight_decay=) and AdamW, and writing them as one function is the clearest way to see it — there is nowhere else for a difference to hide.

The cleanest demonstration: with a zero gradient, only the decay acts. AdamW is exactly w·(1 − lr·λ)ⁿ to 1e-12. Adam+L2 is not.

Why it took until 2017

Adam's adaptivity is sold as its feature: every parameter gets its own effective learning rate. True and useful for the gradient. Nobody noticed it was also being applied to the penalty, because L2 and weight decay had been interchangeable for so long that the distinction had stopped being one.

And the difference is nearly invisible on a well-conditioned problem — 2.9% apart at a 1× gradient spread versus 6.7% at 100×. Real networks have parameters whose gradients differ by orders of magnitude; toy problems do not.

My own test was wrong before the code was

The SGD identity check failed at lr = 1e-2 with NaN. Not because the identity is false — because on a 30×-spread problem plain SGD overflows at that rate. Both variants hit Infinity, and |Inf − Inf| is NaN.

Retested at rates SGD survives, with the instability asserted separately: plain SGD diverges at 1e-2 where Adam does not. Which is a fine advertisement for the normaliser this whole article is complaining about.

What changes in practice

• λ needs retuning. AdamW's λ is not comparable to Adam's weight_decay. Typical AdamW values are much larger — 0.01–0.1 where Adam+L2 used 1e-4 — because the decay is no longer divided by anything.
• λ couples to the learning rate. AdamW shrinks by exactly lr·λ per step, so a decaying schedule decays your regularisation too.
• Do not decay everything. Biases and normalisation scales are conventionally excluded — shrinking a LayerNorm scale toward zero fights the layer's purpose.
• Check which one you are running. In PyTorch, Adam(weight_decay=λ) is coupled; AdamW is decoupled. Same argument name, different algorithm.

And the honest caveat: decoupling is a genuinely different regulariser, so a λ tuned for one is wrong for the other. A comparison that reuses the same λ is measuring the retuning, not the method.

Part of a from-scratch series — one deep-learning idea a day, computed in-browser: https://dev48.infy.uk/deeplearningfromzero.php