Part 7 of my "revisiting my AI/ML notes" series. The last post left off with a question: gradient descent needs the slope of the loss with respect to a weight, but in a multilayer network, most weights aren't directly connected to the loss at all — they're several layers removed from it. This is exactly what the chain rule solves.
The problem: weights are buried inside the network
In a single-neuron example, computing dLoss/dw is direct — the weight feeds straight into the output, which feeds straight into the loss. But in a multilayer network, an early weight influences the loss only indirectly: it affects a neuron's output, which affects the next layer's neurons, which eventually affects the final output, which determines the loss.
The chain rule is exactly this: multiply the local effects together
The chain rule from calculus says that if w1 affects a1, a1 affects a2, and a2 affects the loss, then the total effect of w1 on the loss is the product of each individual step's effect:
dLoss/dw1 = (dLoss/da2) x (da2/da1) x (da1/dw1)
Each term on the right is a local derivative — easy to compute on its own, because it only involves two directly-connected quantities. The chain rule is what lets you stitch these small, local calculations together into the one number you actually need: how much this specific weight, buried deep in the network, is responsible for the final error.
Why this is the actual mechanism behind "backpropagation"
This is also where the name comes from. To compute dLoss/dw1 for a weight near the input, you need dLoss/da2 first — which itself needs to be computed from the output side. So the computation naturally flows backward, from the output toward the input, computing each local derivative and multiplying it into the running product as it goes. That backward flow of derivatives, layer by layer, is backpropagation.
Why this matters
This chain-multiplication is elegant, but it has a real consequence worth flagging early: if each individual local derivative is small, multiplying several of them together makes the result very small very fast. If each one is large, the product explodes just as fast. That single observation is the root cause behind two of the most well-known training problems in deep learning — vanishing and exploding gradients — which are exactly what the next two posts in this series cover.
Original Notes:
Top comments (0)