Everyone tunes the learning rate. Fewer people tune how it changes over training — and that schedule often matters as much as the starting value. I built a visualiser that plots the classic schedules and shows a tiny optimiser descending under each. Here's the tour.
Why a fixed LR is rarely best
Too high and training diverges or oscillates; too low and it crawls, maybe stalling in a bad spot. But the right rate isn't the same at every stage: early on you want to move fast across a rough landscape, later you want to settle gently into a minimum. One constant number can't do both. A schedule can.
The schedules
Warmup — start near zero and ramp up over the first few hundred/thousand steps. Early gradients (especially with Adam, large batches, or transformers) are noisy and the parameter scale is unsettled; blasting a big LR immediately can wreck the run. Warmup buys stability, then hands off to the real rate.
Step decay — hold the LR, then cut it by a factor (say ×0.1) at fixed milestones. Simple, effective, the old ResNet default.
Exponential decay — multiply by a constant each step for a smooth downward glide.
Cosine annealing — follow a half-cosine from the base rate down to ~0 over training: lr = base · 0.5·(1 + cos(π·t/T)). Smooth, no cliffs, and empirically excellent — it's the modern default for a reason. In the demo it's the graceful curve that eases into zero.
Cosine with warm restarts (SGDR) — cosine down, then jump back up and repeat over shrinking cycles. Each restart can kick the optimiser out of a mediocre minimum toward a better one.
1-cycle — ramp LR up to a peak then back down (with momentum moving inversely). Super-fast convergence when tuned; popularised by fast.ai.
The pattern that runs the frontier
For transformer training the near-universal recipe is linear (or cosine) warmup, then cosine decay — warm up for stability, anneal for a clean landing. If you've ever called get_cosine_schedule_with_warmup, that's exactly this.
What the demo shows
Pick a schedule and watch two things: the LR-versus-step curve (the signature shape), and a little gradient descent on a bowl driven by that LR each step. Crank the base rate and constant-LR overshoots and bounces; switch to warmup+cosine and it settles smoothly into the minimum. Same optimiser, same loss surface — only the schedule changed.
The takeaway: the learning rate isn't one hyperparameter, it's a function of time. Choosing its shape — warm up, then anneal — is one of the cheapest wins in training deep nets.
See every schedule plotted and driving a live descent:
https://dev48v.infy.uk/dl/day32-lr-schedules.html
Top comments (0)