What's in here
- Splitting your data (and why the old rules don't apply anymore)
- The mismatch trap
- Bias and variance: the only two problems you have
- The recipe
- Regularization: making the model less confident
- Why does this actually stop overfitting?
- Dropout: randomly break your own network
- Two cheaper tricks
- Normalize your inputs (seriously, just do it)
- Vanishing and exploding gradients
- What I'd tell past me
Most people learn neural networks backwards.
You learn how to build one. Then you build one. Then it performs badly and you have no idea why, so you go back and change the number of layers, and it still performs badly, and now you're just guessing.
I've been going through the fundamentals again properly, and the thing that clicked for me is this: almost every "my model doesn't work" problem falls into one of two buckets. Once you know which bucket you're in, the fix is almost mechanical.
Here's what I've learned so far, written the way I wish someone had explained it to me.
Splitting your data (and why the old rules don't apply anymore)
You split your data into three chunks:
Train set - the model learns from this.
Dev set (also called validation) - you use this to pick between different model ideas. Should I use 3 layers or 5? Should the learning rate be 0.01 or 0.001? You try both, check the dev set, keep the winner.
Test set - you touch this once, at the very end, to get an honest number for how good your model actually is.
The classic textbook split is 60/20/20. That was fine when datasets had 10,000 rows.
But if you have a million examples, do you really need 200,000 of them just to compare two learning rates? No. A few thousand is enough to tell two models apart. So modern splits look more like 98/1/1.
The dev and test sets aren't there to be big. They're there to be enough.
And honestly, if you don't need a fully unbiased final number, you can skip the test set entirely. Lots of teams do. They just call their dev set a "test set," which is technically wrong but nobody's stopping them.
The mismatch trap
Here's a mistake that silently destroys models.
Say you're building a cat classifier. You scrape 200,000 clean, well-lit cat photos from the web for training. Then for your dev and test sets, you use 10,000 blurry photos taken by actual users on their phones.
Your model will look great in training and fall apart in the real world. Not because the model is bad, but because you trained it on a different world than the one you're testing it in.
The rule: your dev and test sets must come from the same distribution. And that distribution should be the data you actually care about performing well on.
You can still train on the web photos (data is data). But dev and test both have to look like the real thing. Otherwise you're optimizing for a target that doesn't exist.
Bias and variance: the only two problems you have
Forget the formal statistics definitions for a second. In practice, you diagnose this with two numbers.
Let's say a human can identify cats with ~0% error. That's roughly your ceiling.
| Train error | Dev error | Diagnosis |
|---|---|---|
| 1% | 11% | High variance (overfitting) |
| 15% | 16% | High bias (underfitting) |
| 15% | 30% | High bias AND high variance |
| 0.5% | 1% | You're fine. Go home. |
Read that table again, it's the whole game.
High variance = your model memorized the training data. It's brilliant on data it's seen, useless on data it hasn't. The gap between train and dev is the tell.
High bias = your model is too simple to even learn the training data. It's bad everywhere, equally. There's no gap, just a high floor.
Both at once feels weird when you first see it, but it happens. Imagine a model that's mostly a straight line (too simple for most of the data) but has a few wild spikes where it memorized specific weird examples. Underfitting in general, overfitting in spots. Bad train error and a big gap.
One caveat on that ~0% ceiling. If you're classifying blurry images that even humans get wrong 15% of the time, then 15% train error isn't high bias. It's near-optimal. Always compare against what's actually achievable, not against zero.
The recipe
This is the flowchart I now run in my head every time a model underperforms:
1. Do I have high bias? (check train error)
If yes → bigger network, train longer, try a different architecture. Keep going until train error looks good.
2. Do I have high variance? (check the train-to-dev gap)
If yes → get more data, or regularize. Then go back to step 1 and re-check.
3. Repeat until both are low. Then stop.
What makes this useful is the order. Fix bias first, because a model that can't even fit the training data has nothing to regularize. And notice that the two fixes are different: throwing more data at a high-bias model does nothing at all.
Back in the pre-deep-learning days there was a real tradeoff. Reducing bias meant increasing variance and vice versa. Now? A bigger network reduces bias without much variance cost (as long as you regularize), and more data reduces variance without touching bias. The tradeoff isn't gone but it's much weaker.
Regularization: making the model less confident
When you have high variance and can't get more data (which is most of the time), you regularize.
For logistic regression, you add a penalty term to your cost function:
J(w, b) = (1/m) × Σ Loss(ŷ, y) + (λ / 2m) × ||w||²
That second term is L2 regularization. ||w||² is just the sum of every weight squared. λ (lambda) is how hard you punish big weights. It's a hyperparameter you tune on the dev set.
You usually skip b in the penalty. It's one number out of millions, it doesn't matter.
For a neural network, same idea, but you sum the penalty across every weight matrix in every layer:
J = (1/m) × Σ Loss + (λ / 2m) × Σ_l ||W[l]||²
The matrix version of this norm has a fancy name (Frobenius norm) but it's the same thing: square everything, add it up.
In backprop, this adds a (λ/m) × W term to your gradient. Which means the weight update becomes:
W = W - α × (backprop_gradient + (λ/m) × W)
W = W(1 - αλ/m) - α × backprop_gradient
See that (1 - αλ/m) factor? Every single step, before you even apply the gradient, your weights get shrunk slightly. That's why L2 regularization is also called weight decay. Your weights are constantly leaking toward zero, and only the gradient pushes back.
Why does this actually stop overfitting?
Two ways to see it.
The intuition: crank λ way up. The penalty dominates, so the optimizer drives weights toward zero to minimize cost. Weights near zero means many neurons contribute almost nothing. Your huge network effectively behaves like a much smaller, simpler one. Simpler network, less capacity to memorize.
Obviously you don't set λ that high. But somewhere in the middle you land on a network that's "just simple enough."
The better intuition (my favorite): think about tanh as your activation. When z is small, tanh is basically a straight line. When z is large, it saturates into curves.
Big λ → small W → small z → every neuron is operating in its linear zone. And a network made of linear pieces can only learn roughly linear things. It literally cannot draw the wildly wiggly decision boundary that overfitting requires.
Regularization doesn't just make the model smaller. It makes it smoother.
Dropout: randomly break your own network
This one sounds insane the first time you hear it.
On every training pass, you randomly switch off a chunk of neurons. Different ones each time. Train on the crippled network. Repeat.
Here's how you implement it (this is inverted dropout, the standard version):
keep_prob = 0.8 # keep 80% of neurons in this layer
d3 = np.random.rand(*a3.shape) < keep_prob # boolean mask
a3 = a3 * d3 # zero out the dropped ones
a3 = a3 / keep_prob # ← the important line
That last line is what makes it "inverted," and it's the part people skip and then wonder why their loss is weird.
If you zero out 20% of your activations, the sum feeding into the next layer drops by ~20%. Dividing by keep_prob scales it back up so the expected value stays the same. Without it, every layer's output shrinks and your network quietly falls apart.
At test time: you do not use dropout. No mask, no scaling, no randomness. You want deterministic predictions, and because you already scaled during training, your activations are already at the right magnitude. Nothing to fix.
If you'd used the old non-inverted version, you'd have to manually rescale at test time. Which is exactly why nobody uses it anymore.
Why does dropout work?
Take one neuron in the middle of the network. Its job is to combine inputs from the layer below.
With dropout on, any of those inputs could vanish at any moment. So the neuron can't afford to bet everything on one input. It has to spread its weights out across many inputs, because any single one might be gone next iteration.
Spreading weights out shrinks their squared norm. Which means dropout ends up doing something similar to L2, just through a completely different mechanism.
A few practical notes:
You can set a different keep_prob per layer. Big weight matrices are where overfitting lives, so drop harder there (maybe 0.5). Small layers, leave them alone (1.0). Input layer, keep it near 1.0.
The annoying part: your cost function is no longer well-defined, because you're changing the network every iteration. J doesn't decrease monotonically anymore, so you can't debug with a nice descending loss curve. My workaround is to turn dropout off, confirm the loss decreases properly, then turn it back on.
Computer vision people use dropout almost by default, because they basically never have enough pixels-worth of data. Elsewhere, only reach for it when you actually see overfitting. It's not free.
Two cheaper tricks
Data augmentation. More data fixes variance, but real data is expensive. So fake it. Flip your cat photos horizontally (still a cat). Crop, rotate, zoom, distort. For digits, small rotations and warps. It's not as good as genuinely new data, since a flipped cat carries less new information than a new cat. But it's nearly free.
Early stopping. Plot train error and dev error as training goes on. Train error keeps dropping forever. Dev error drops, bottoms out, then starts climbing. Stop at the bottom.
Why does it work? At the start your weights are tiny (random init). As you train they grow. Stopping mid-training means stopping at a mid-sized W, which is roughly what L2 was trying to give you anyway.
But I'll be honest, I don't love it. Early stopping tangles two jobs together: fitting the training data, and preventing overfitting. Now one knob controls both, and you can't tune them independently. L2 keeps them separate. The only cost is you have to search over λ, which is more compute.
Normalize your inputs (seriously, just do it)
Two steps:
mu = np.mean(X, axis=1, keepdims=True)
X = X - mu # now mean is zero
sigma2 = np.mean(X**2, axis=1, keepdims=True)
X = X / np.sqrt(sigma2) # now variance is one
Use the same mu and sigma from the training set on your dev and test sets. Don't recompute them. Both sets need to be transformed identically or you've broken your own distribution.
Why bother?
Picture feature 1 ranging from 0 to 1, and feature 2 ranging from 1 to 10,000.
Your weights end up wildly different in scale, and the cost surface turns into a long, thin, stretched-out valley. Gradient descent bounces back and forth across the narrow walls, inching forward. You're forced to use a tiny learning rate or it diverges.
Normalize, and that valley becomes a round bowl. Every direction is equally steep. Gradient descent walks straight to the bottom, and you can use a much larger learning rate.
Cheap to do. Never hurts. Just make it a habit.
Vanishing and exploding gradients
This is the problem that kept deep networks from working for years.
Imagine a very deep network. Simplify hard: linear activations, ignore b, and every weight matrix is the same:
W[l] = [[1.5, 0], [0, 1.5]]
Then your output is roughly 1.5^L × x, where L is the number of layers.
With L = 150, that's 1.5^150. An astronomically large number. Your activations explode, your gradients explode, your loss becomes NaN.
Now flip it to 0.5. Output becomes 0.5^150, which is effectively zero. Activations vanish. Gradients vanish. The early layers get updates so small they never learn anything.
The scary part is how sharp this is. Slightly above 1, explode. Slightly below 1, vanish. In a deep network there's almost no middle ground, because you're compounding the error 150 times.
The single neuron fix
Look at one neuron with n inputs:
z = w₁x₁ + w₂x₂ + ... + wₙxₙ
The more inputs (larger n), the bigger z gets, just from adding more terms. So to keep z reasonable, each individual wᵢ should be smaller when n is larger.
That gives you the initialization rule. Set the variance of your weights to 1/n:
W = np.random.randn(shape) * np.sqrt(1 / n_prev)
For ReLU, 2/n works better in practice (this is He initialization):
W = np.random.randn(shape) * np.sqrt(2 / n_prev)
For tanh, 1/n is the common choice (Xavier initialization).
It doesn't fully solve vanishing/exploding gradients. But it makes the compounding start from a much safer place, and in a 150-layer network, where you start matters a lot.
What I'd tell past me
The whole thing collapses into a loop:
Check train error. Too high? Bigger model.
Check the gap. Too big? More data or regularization.
Normalize your inputs. Initialize your weights properly.
Repeat.
That's it. No mysticism. Most of the time when a model isn't learning, it's one of these, and the fix is on this page.
What still bugs me is the λ search. Tuning it properly means training the model over and over just to compare numbers, and there has to be a better way than brute force.
Working on it.
Top comments (0)