The 0.2% Accuracy Drop That Cost Us 3 Days
Swapping Adam for AdamW in a ResNet-50 training script boosted validation accuracy from 76.1% to 76.3%. Not earth-shattering, but enough to beat the baseline we'd been stuck at for a week.
The weird part? The loss curves looked almost identical. Training ran at the same speed. Memory usage didn't budge. The only difference was whether weight decay happened before or after the gradient update step.
Most tutorials treat Adam and AdamW as interchangeable. They're not. Understanding why requires looking at what weight decay actually does to the optimizer's update rule — and why the "obvious" implementation in Adam turns out to be wrong.
Weight Decay vs L2 Regularization (They're Not the Same)
Here's the confusion that trips up everyone initially: weight decay and L2 regularization produce identical results in SGD, but diverge completely in adaptive optimizers like Adam.
L2 regularization adds a penalty term to the loss function:
Continue reading the full article on TildAlice
Top comments (0)