Classical learning theory tells a tidy story. As a model gets more complex, training error falls, test error first falls and then rises, and the best model sits at the bottom of a U-shaped curve. That story is not wrong. It is incomplete.
In modern machine learning, test error often falls, rises sharply to a peak, and then falls again as capacity keeps growing. This second descent is called double descent. It helps explain why very large models, which can fit their training data exactly, sometimes generalise surprisingly well, while a model just large enough to interpolate the data generalises terribly.
This article explains what double descent is, where the peak comes from, what mathematics describes it, and when you should - and should not - expect to see it.
The classical picture: bias, variance, and the U curve
Start with the familiar decomposition. For a prediction problem with input x, true function f, noise, and a learned predictor, expected test error can be split into three parts: squared bias, variance, and irreducible noise.
Bias measures how far the average model, over different training sets, is from the truth. Variance measures how much the fitted model moves when the training set changes. Noise is what no model can remove.
In the classical regime, increasing complexity reduces bias but increases variance. At first the bias reduction wins, so test error falls. Later variance dominates, so test error rises. The sweet spot is the trade-off point.
That picture assumes we stay in the underparameterised regime, where the number of parameters p is smaller than the number of training examples n. The model cannot fit every training point exactly. Once p passes n, the picture changes.
Crossing the interpolation threshold
Define the interpolation threshold as the point where p is roughly equal to n. At this point the model has just enough capacity to fit the training data almost exactly, including its noise.
Near this threshold, the fitted model is extremely sensitive to the training data. A small change in one example can swing the fit a lot, because the model has almost no freedom left: it must thread through nearly every point. Variance explodes, and test error peaks.
Past the threshold, in the overparameterised regime where p is much larger than n, there are many parameter settings that all fit the training data exactly. The learning algorithm does not pick one at random. It picks a particular one, usually the one closest to its initialisation or the one with smallest norm under the optimisation dynamics. That implicit choice acts as a form of regularisation, and it can reduce variance again. Test error descends a second time.
So the full curve is: descent, ascent to a peak at interpolation, then a second descent. That is double descent.
The maths: least squares and the minimum-norm interpolator
The cleanest setting for double descent is linear regression with more features than examples.
Suppose we have n training examples and p features, with p greater than n. Let X be the n by p design matrix and y the vector of targets. There are infinitely many weight vectors w that satisfy Xw = y exactly. Gradient descent starting from zero, run to convergence on the squared loss, converges to the minimum-norm interpolator:
This is the interpolating solution with the smallest Euclidean norm. The norm matters because, for linear models, a smaller norm typically means a smoother, less wild function, and therefore often better generalisation.
For ridge regression, the solution is:
where is the regularisation strength and I is the identity matrix. As goes to zero in the overparameterised regime, ridge regression tends to the minimum-norm interpolator above. Positive shrinks the solution further and can soften or remove the peak.
A useful way to think about the peak is through the singular values of X. Near p = n, the smallest singular values of the data matrix can become very small. Inverting them, as the minimum-norm formula does, amplifies noise enormously. Far past the threshold, the spectrum spreads out, the effective amplification falls, and the interpolator becomes more stable. Random matrix theory makes this precise for idealised Gaussian features: test risk can be written as a function of the ratio p/n, and it diverges at p/n = 1 in the unregularised limit, then decreases as p/n grows.
The same intuition extends, with more work, to random feature models and to wide neural networks in the kernel regime, where the model behaves like kernel regression with the neural tangent kernel. The details differ, but the mechanism is similar: a peak at interpolation, followed by a second descent driven by the implicit regularisation of the fitting procedure.
A practical example: watching the peak appear
You can see double descent without a large neural network. Take a synthetic regression problem, expand the input with random features, and increase the number of features past the number of training examples. Use ridge regression with a tiny regularisation strength.
import numpy as np
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error
rng = np.random.default_rng(0)
n_train, n_test = 200, 1000
X_train_raw = rng.normal(size=(n_train, 20))
X_test_raw = rng.normal(size=(n_test, 20))
true_w = rng.normal(size=20)
y_train = X_train_raw @ true_w + 0.5 * rng.normal(size=n_train)
y_test = X_test_raw @ true_w + 0.5 * rng.normal(size=n_test)
def random_features(X, p):
# Random ReLU features: a simple, explicit expansion.
W = rng.normal(size=(X.shape[1], p)) / np.sqrt(X.shape[1])
return np.maximum(X @ W, 0)
for p in [10, 50, 100, 190, 200, 210, 300, 500, 1000, 2000]:
Phi_train = random_features(X_train_raw, p)
Phi_test = random_features(X_test_raw, p)
model = Ridge(alpha=1e-6)
model.fit(Phi_train, y_train)
err = mean_squared_error(y_test, model.predict(Phi_test))
print(f"features={p:5d} ratio p/n={p / n_train:.2f} test MSE={err:.3f}")
In a typical run, test error falls as p grows toward n, spikes near p = n, and then falls again as p grows well past n. The exact heights depend on the noise level, the feature distribution, and the regularisation strength, which is exactly the point: double descent is a phenomenon of the whole setup, not a property of model size alone.
Try two variations. First, increase the label noise: the peak gets taller. Second, increase the ridge penalty: the peak shrinks and may disappear. Both effects follow from the maths above. Noise is what gets amplified near interpolation, and explicit regularisation competes with the implicit regularisation of the minimum-norm solution.
Why the second descent does not mean "bigger is always better"
Double descent is often misread as a licence to ignore model size. That reading is wrong for several reasons.
First, the second descent is not guaranteed. It depends on the data distribution, the feature map, the noise level, the loss, and the optimiser. With heavy label noise, poor features, or a badly chosen architecture, test error past the threshold can stay high or even rise again.
Second, the peak is real and dangerous. A model near the interpolation threshold can be the worst model in the whole family, even though its training error is near zero. If you tune model size by training error alone, you will walk straight into the peak.
Third, test error is not the only cost. Larger models cost more to train, serve, and maintain, and they can be harder to interpret and debug. A second descent in test error of a few percent may not justify a tenfold increase in compute.
Fourth, double descent in test error does not imply good behaviour elsewhere. Calibration, robustness to distribution shift, fairness across subgroups, and performance on rare cases can all behave differently from average test error. A model past the peak can still fail badly on the examples that matter most.
Finally, modern practice rarely varies only p with everything else fixed. Architecture, data scale, training time, and regularisation all change together. Real scaling curves blend many effects, and attributing all of an improvement to double descent would be a mistake.
Common pitfalls when testing for double descent
- Tuning on test data to find the peak. Use a validation set, and fix the evaluation protocol before sweeping model size.
- Forgetting regularisation. Even a tiny ridge penalty, weight decay, or early stopping changes the shape of the curve. Report it.
- Confusing interpolation with memorisation in a harmful sense. Interpolation means fitting training points exactly. Whether that harms generalisation depends on how the interpolator is chosen.
- Sweeping too coarsely. The peak is narrow. If your grid jumps from p well below n to p well above n, you will miss it and conclude the curve is monotone.
- Ignoring the optimiser. Different optimisers, initialisations, and learning rates pick different interpolators. The minimum-norm story is specific to gradient descent from zero on squared loss; other settings need their own analysis.
What to take away
The classical U curve describes the underparameterised regime well. Double descent extends it: test error can peak at the interpolation threshold, where the model just barely fits the data, and then fall again as overparameterisation lets the optimiser's implicit bias select a smoother interpolator.
For practice, the lesson is concrete. Do not judge a model by training error, do not assume the worst model is the largest one, and do not assume the largest one is the best. Sweep capacity carefully, keep regularisation explicit, watch the region around p = n, and validate on data the model has never seen.
Bigger is not always better, and smaller is not always safer. The shape of the curve depends on the problem, and the only way to know it is to measure it.
Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial
Top comments (0)