Part 6 of my "revisiting my AI/ML notes" series. Everything so far in this series used a single hidden layer as an example. Real networks almost always stack multiple hidden layers — this post covers what changes when you do that, and revisits gradient descent from a more visual angle.
Going from one hidden layer to many
A multilayer network just repeats the same input -> weighted sum -> activation pattern, but chains multiple hidden layers back to back before reaching the output. The structure looks like:
Input layer -> Hidden layer 1 -> Hidden layer 2 -> Output layer
Why the number of weights grows so fast
Every layer is fully connected to the next, so the number of weights is the product of the neuron counts on each side. With 4 input neurons feeding into a hidden layer of 3 neurons, that's 4 x 3 = 12 weights just for that one connection. If that hidden layer of 3 then feeds into a second hidden layer of 2 neurons, that's another 3 x 2 = 6 weights.
This is worth sitting with for a second: weight count isn't linear in the number of neurons, it's closer to combinatorial across layer boundaries. This is exactly why deep networks have millions (or billions) of parameters — it's not that any single layer is huge, it's that the connections between layers multiply fast.
Reading the slope to know which way to move
Gradient descent works by looking at the slope (derivative) of the loss curve at the weight's current position, and using that slope to decide which direction to move the weight:
- If the slope is negative (the curve is heading downward to the right), the weight needs to move up to get closer to the minimum.
- If the slope is positive (the curve is heading upward to the right), the weight needs to move down
This is exactly why the update rule has a minus sign in it — w_new = w_old - lr * gradient — subtracting a positive gradient moves the weight down, and subtracting a negative gradient moves it up. The sign of the gradient is doing the steering; the minus sign is what makes the steering point toward lower loss instead of higher loss.
Why this matters
A single neuron's math (covered earlier in this series) is simple enough to compute by hand. A real network with multiple hidden layers is not — and that's precisely why backpropagation needs the chain rule to work, which is the topic of the next post. Understanding gradient descent visually first, as a ball rolling down a curve based on the local slope, makes the chain-rule math that follows feel like a natural extension rather than an arbitrary formula.
The Original Notes:
Top comments (0)