Underfitting and overfitting aren't two separate problems — they're two ends of one dial. That dial is the bias-variance tradeoff, and once you see it, model selection finally makes sense. Here it is, live.
⚖️ Slide model complexity and watch: https://dev48v.infy.uk/ml/day19-bias-variance.html
Error has three parts
Total expected error = bias² + variance + irreducible noise.
- Bias = error from wrong/too-simple assumptions → the model can't capture the pattern → underfitting.
- Variance = error from being too sensitive to the specific training data → it memorizes noise → overfitting.
- Noise = irreducible; no model beats it.
The U-curve
Crank model complexity (the demo uses polynomial degree): training error always drops — which is exactly why training error alone fools you. But test error is U-shaped: high at low complexity (bias), dips at the sweet spot, rises at high complexity (variance). The best model is the bottom of the U.
The levers
- More data → lowers variance.
- Regularization → lowers variance (Day 17).
- Better features → lowers bias.
- Bagging cuts variance; boosting cuts bias (Days 11 & 15).
- Find the sweet spot with cross-validation (Day 18).
🔨 Built from scratch (fit increasing degrees → measure train vs test → find the dip) on the page: https://dev48v.infy.uk/ml/day19-bias-variance.html
Part of MachineLearningFromZero. 🌐 https://dev48v.infy.uk
Top comments (0)