Early stopping: train loss lies, validation loss U-turns — keep the best epoch, restore its weights, and it's free L2

Give a big enough network enough epochs and it will drive its training loss to zero — by memorising the noise in your data. But the thing you actually care about, error on data it hasn't seen, does something different: it falls while the net learns the real signal, reaches a minimum, then climbs back up as the net starts fitting quirks that don't generalise. Plot both and you get deep learning's most recognisable diagram — a training curve sliding toward zero and a validation curve shaped like a U, the gap between them widening. That gap is overfitting. Early stopping is the simplest, cheapest fix in the whole field, and I built a demo that fits a real 34-basis model by genuine gradient descent to 16 noisy points so you can watch the U-turn happen live.

Training loss is a liar about generalisation

Real data is signal plus noise. A model with more capacity than the signal needs will first fit the broad signal — which lowers both train and val loss — and then start bending itself to pass through the exact noisy training points, which lowers train loss but raises val loss, because the noise in the validation set is different. That crossover is the bottom of the val U: the moment the model switches from learning generalities to memorising specifics. So you hold out a validation set the optimiser never sees a gradient from, and after every epoch you measure loss on it — a live estimate of generalisation. The test set stays sealed until the very end; the moment you tune to val, val stops being unseen.

Watch val, and remember the best

The core idea is one loop. After each epoch, evaluate val loss; if it's the best you've seen, snapshot the weights. That snapshot is the model you'll actually keep — everything else is just deciding when to quit:

best_val = float("inf"); best_state = None
for epoch in range(max_epochs):
    train_one_epoch(model, train_loader)
    v = evaluate(model, val_loader)
    if v < best_val:
        best_val = v
        best_state = copy.deepcopy(model.state_dict())   # remember it

Patience and min-delta keep noise from fooling you

Val loss is measured on a finite, noisy set, so it jitters — it can tick up for an epoch or two and then keep falling to a new minimum. If you stopped at the first uptick you'd quit far too early. Patience is the fix: count how many epochs in a row have failed to beat the best, and only stop when that counter hits patience. Reset it to zero every time you find a new best. And without a threshold, a microscopic drop of 0.00001 still counts as "improvement," resets patience, and grinds forever on noise — so min-delta sets the smallest change that qualifies:

if v < best_val - min_delta:        # must beat it by a margin
    best_val = v; best_state = deepcopy(model.state_dict()); wait = 0
else:
    wait += 1
    if wait >= patience:            # the upturn is real -> stop
        break

Restore best weights — the step everyone forgets

Here's the one that bites people. When the loop breaks, the model is not at its best epoch — it's patience epochs later, already sliding into overfitting. Ship it as-is and you've thrown away the entire benefit. Load the snapshot back:

model.load_state_dict(best_state)   # rewind to the best-val epoch
# THIS is the model you deploy. Skip it and early stopping is half-done.

In the demo, flipping "restore best weights" off makes the deployed test error jump visibly — that jump is exactly the patience epochs of overfitting you accidentally kept. Keras even defaults restore_best_weights to False, which quietly bites a lot of people; always set it True.

Why it's implicit regularisation

This isn't just a stopping heuristic — it's regularisation hiding in the training schedule. Gradient descent starts from small weights and grows them a little each step, so the longer you train, the farther the weights travel and the wigglier a function they can express. Stopping early caps how far they get to move, which keeps the effective model simpler. For a linear least-squares fit you can prove that stopping after t steps gives almost exactly the same solution as an L2 (ridge) penalty of a particular strength — fewer steps ⇔ stronger shrinkage. That's why it fights overfitting without adding a penalty term, composes with weight decay and dropout, and is nearly free: its only real cost is the slice of data you hold out, and it even saves you the wasted epochs. Which is why it's on by default in essentially every serious training run.

Press Train and watch train MSE fall while val MSE U-turns, then drag patience to move the stop:
https://dev48v.infy.uk/dl/day45-early-stopping.html