Your model scores 0.64 on held-out data and the boss wants better. You have two obvious moves: go label another ten thousand rows, or throw out the model and try something bigger. Pick wrong and you burn a week. A learning curve tells you which one is worth doing, and it takes about ten lines to draw.
The idea is simple. Instead of training once on all your data, you train the same model on growing slices of it: 10% of the rows, then 20%, and so on up to the full set. At every size you record two scores. One is the training score, measured on the very rows the model just learned from. The other is the validation score, measured on a held-out set it never saw. Then you plot both against the number of rows. The shapes of those two curves, and the gap between them, are a diagnosis you can act on.
I built a live version from scratch to watch it happen. The hidden target is sin(1.5x) with noise added, and the model is a polynomial fit by least squares. Low degree can only draw gentle shapes; high degree can wiggle through every point. That one knob lets you produce every regime on demand.
Start with a degree-1 line trying to trace a sine wave. Both curves shoot toward each other and flatten out at a low score, almost touching. That is high bias, or underfitting. The model is too simple to represent the pattern, so it does about equally badly on data it saw and data it didn't. The tell is the combination: low level, small gap, and already converged. And here is the expensive insight: once those two curves have met and gone flat, more data does nothing. Every extra row lands on a line that has already plateaued. This is the case where teams spend a month collecting data that mathematically could not have helped.
Now crank the degree up to 14. The training score stays high, because a flexible model nails the handful of rows it saw, but the validation score sits well below it, leaving a wide gap. That is high variance, or overfitting. The model memorized the noise in its particular sample instead of the signal. But look at the right edge of the validation curve: it is still climbing, still chasing the training curve without catching it. That rising, un-closed gap is the signature that more data would keep pushing validation up. This is the one case where "get more rows" is the right instinct.
In between, around degree 6, both curves converge to a high score with only a small gap. Good fit. Complexity matches the signal, and you should ship it rather than chase marginal gains.
Two quantities are worth reading separately. The gap between the curves is variance, how much the model's performance depends on which particular rows it trained on. The level where the validation curve plateaus is the bias floor, the best this model family can do, set by the irreducible label noise plus whatever the model's simplicity forces it to miss. Closing the gap and lowering the floor are different jobs with different levers. More data or regularization closes a gap. Only a better model lowers the floor.
Which is exactly why the two fixes point in opposite directions. High bias wants a bigger model: add features, raise the complexity, reduce regularization, and do not bother collecting data. High variance wants the reverse: more data, or if that is impossible, more regularization, a simpler model, fewer features. Notice the same regularization knob fixes variance when you turn it up and fixes bias when you turn it down. Guessing which way to turn it is what the curve saves you from.
One thing not to confuse: a learning curve varies the dataset size, but a validation curve fixes the data and varies a single hyperparameter like the polynomial degree. On that plot you literally watch bias turn into variance as the knob increases, and you pick the degree where the validation score peaks.
In practice you never write the loop. scikit-learn's learning_curve takes an estimator, X, y, a list of train_sizes and a cv scheme, and hands back train and validation scores at each size, averaged over folds so the curve is smooth. Its sibling validation_curve does the same for a swept hyperparameter. I checked my from-scratch version against the same logic in Node: the degree-14 model shows a train-validation gap over 0.2, the degree-1 model plateaus low with the two curves touching, and the training score sits at or above validation everywhere, exactly as the theory says.
You can drag the complexity, noise, and regularization sliders and watch the diagnosis flip live here: https://dev48v.infy.uk/ml/day30-learning-curves.html
Next up is the precision-recall curve: how to score a classifier honestly when the positive class is rare.
Top comments (0)