You trained a model. It scored 99% on your training data. You celebrate. Then it sees real-world data and falls flat on its face. What happened?
You just met overfitting — one of the two most common ways a machine learning model can go wrong. Its evil twin is underfitting. Together, they're responsible for more failed ML projects than bad data and wrong algorithms combined.
In this post, you'll learn what overfitting and underfitting actually are (through an analogy that sticks), how to spot each one, and — most importantly — how to fix them.
The Exam Analogy
Imagine two students preparing for a history exam.
Student A memorizes the textbook word for word. Every date, every footnote, every example question and its exact answer. On a practice test using those same questions, Student A scores 100%. But when the real exam asks a question phrased slightly differently — even about the same event — Student A freezes. They memorized the answers, not the concepts.
Student B glances at the chapter titles the night before. They get the general vibe: "something about a war, some treaties." On the practice test, they score 40%. On the real exam, also 40%. They didn't learn enough to answer anything well.
Now imagine Student C — they read the textbook, understood the key themes, practiced with different types of questions, and can apply what they learned to questions they've never seen before. Practice test: 88%. Real exam: 85%. Not perfect, but consistently solid.
In machine learning: Student A is overfitting. Student B is underfitting. Student C is balanced — and that's what we're aiming for.
What Is Overfitting?
Overfitting happens when your model learns the training data too well. It doesn't just learn the patterns — it memorizes the noise, the outliers, the random quirks that are specific to your training set but don't exist in the real world.
The telltale sign: high performance on training data, poor performance on test/unseen data.
If you plot an overfitting model's predictions, it looks like a line that zigzags wildly through every single data point — including the ones that are just random noise. It's too complex. It's trying too hard.
Why does it happen?
- Too little training data. With few examples, the model has no choice but to memorize what's there.
- Too complex a model. A neural network with millions of parameters trained on 500 data points will memorize those 500 points perfectly — and learn nothing generalizable.
- Training for too long. The longer you train, the more the model starts fitting to noise instead of signal.
- Too many features. If you feed the model 200 features for 1,000 samples, it finds spurious correlations that don't hold up in production.
What Is Underfitting?
Underfitting is the opposite problem. Your model is too simple to capture the actual patterns in the data. It hasn't learned enough.
The telltale sign: poor performance on both training AND test data.
An underfitting model draws a straight line through data that clearly curves. It misses the pattern entirely — not because the pattern isn't there, but because the model doesn't have the capacity to see it.
Why does it happen?
- Model is too simple. Trying to fit a linear regression to data that has a non-linear relationship.
- Not enough features. You're trying to predict house prices but only gave the model the number of bedrooms — ignoring square footage, location, age, condition.
- Not trained long enough. You stopped training before the model had a chance to learn.
- Too much noise in data. If the data is so noisy that the signal is buried, even a good model will underfit.
How to Tell Which One You Have
Here's the cheat sheet:
| Symptom | Training Score | Test Score | Diagnosis |
|---|---|---|---|
| Both scores high and close | 92% | 89% | Balanced — you're in good shape |
| Training high, test low | 99% | 62% | Overfitting — model memorized |
| Both scores low | 55% | 52% | Underfitting — model didn't learn |
The gap between training and test performance is your key diagnostic tool. A large gap screams overfitting. Uniformly poor scores scream underfitting.
How to Fix Overfitting
1. Get more training data. More diverse examples make it harder for the model to memorize and force it to learn actual patterns. This is the single most effective cure.
2. Reduce model complexity. Use fewer layers, fewer parameters, a simpler architecture. If a decision tree works, don't use a 50-layer neural network.
3. Apply regularization. Techniques like L1 (Lasso) and L2 (Ridge) regularization penalize the model for becoming too complex, keeping weights small and forcing generalization.
4. Use dropout. In neural networks, dropout randomly "turns off" some neurons during training. This prevents the network from relying too heavily on any single neuron — like forcing Student A to study without their favorite highlighter.
5. Apply early stopping. Monitor the model's performance on validation data during training. When validation performance starts getting worse (even though training performance keeps improving), stop. That inflection point is where memorization begins.
6. Clean and preprocess your data. Remove noise, handle outliers, fix inconsistencies. Cleaner data gives the model less garbage to memorize.
How to Fix Underfitting
1. Increase model complexity. Use a more powerful model — go from linear regression to polynomial, from a shallow network to a deeper one.
2. Better feature engineering. Create new features that capture the patterns the model is missing. Maybe "price per square foot" matters more than price and square footage separately.
3. Clean noise from data. Reduce noise so the actual signal becomes learnable.
4. Train longer. Give the model more epochs to learn. Sometimes it just needs more time.
5. Reduce regularization. If you've applied too much regularization, you might be preventing the model from learning even the real patterns. Dial it back.
The Bias-Variance Tradeoff
What you've just learned has a formal name: the bias-variance tradeoff.
- High bias = underfitting. The model makes too many assumptions and misses the real pattern.
- High variance = overfitting. The model is too sensitive to the specific training data.
You want the sweet spot: low enough bias to capture the pattern, low enough variance to generalize. In practice, you're always balancing these two — pushing one down often pushes the other up.
Think of it like adjusting the focus on a camera. Too blurry (underfitting) and you can't see anything. Too zoomed-in on one pixel (overfitting) and you lose the bigger picture. The sharp image is somewhere in between.
Common Misconceptions
"Higher accuracy is always better." Not if it's only on training data. A model with 99% training accuracy and 60% test accuracy is worse than one with 85% on both. Generalization is what matters.
"More complex models are always better." Complexity without enough data leads straight to overfitting. Start simple, increase complexity only when you have evidence it helps.
"Overfitting only happens with deep learning." Nope. A decision tree with no depth limit will happily overfit a small dataset. Overfitting is a universal ML problem.
What to Try Next
If you're learning ML, here's a quick experiment: take any dataset (Scikit-learn's built-in datasets work great), train a decision tree with no depth limit, and compare training vs test accuracy. Then set max_depth=3 and compare again. You'll see overfitting and the fix in action — in under 10 lines of code.
The best ML models aren't the ones that score highest on training data. They're the ones that perform consistently well on data they've never seen before. Learning to navigate the overfitting-underfitting spectrum is one of the most practical skills in machine learning — and now you know how.
*If this helped you understand the difference, drop a reaction.
Top comments (0)