https://www.youtube.com/watch?v=sjYCdifiRNw
Every neural network you've ever used — GPT, Stable Diffusion, AlphaFold — learned through the same algorithm. It wasn't invented at Google or OpenAI. It was formalized in 1986 by Rumelhart, Hinton, and Williams. The algorithm is backpropagation, and once you understand it, deep learning stops being magic.
A Network That Guesses Wrong
A neural network starts as a guess machine. You feed it a picture of a cat, and it says "truck." That's not a bug — it's the starting point. Every network begins with random weights, producing random garbage.
Here's the flow: your input enters the first layer. Each connection has a weight. The input gets multiplied by these weights, passed through an activation function, and flows forward to the next layer. Layer by layer, the signal propagates until it reaches the output. This is the forward pass — the network's best guess with whatever weights it currently has.
The question is: how does the network go from "truck" to "cat"?
Measuring Wrongness
The loss function takes the network's prediction and the correct answer and outputs a single number. The higher the number, the more wrong the prediction. Think of it like a scoreboard: the network predicted 0.1 for "cat" but the correct answer is 1.0. The loss function says you're 0.9 off.
Here's the key insight: this loss isn't just a number. It's a function of every weight in the network. Change any weight, and the loss changes too. That means the loss function creates a landscape — a surface with hills and valleys. The network's current weights place it somewhere on this landscape, and training is just finding a path downhill.
But which direction is downhill?
The Chain Rule Does the Heavy Lifting
The loss depends on the output. The output depends on the last layer's weights. But those activations depend on the layer before that. And that layer depends on the one before it. It's a chain of dependencies.
To figure out how each weight affects the loss, you need the chain rule from calculus. If A affects B, and B affects C, then the effect of A on C is the product of those individual effects.
Backpropagation starts at the output and works backward. How much does the loss change when the output changes? Then: how much does the output change when the last layer's weights change? Multiply those together. Move one layer back, multiply again. Each step through the network adds another link in the chain.
The result is a gradient for every single weight — a number that says "if you increase this weight by a tiny amount, the loss will change by this much."
Nudging Toward Correct
Now you have gradients. What do you do with them? Gradient descent. Take each weight, subtract a small fraction of its gradient. That fraction is called the learning rate.
If the gradient is positive — meaning increasing this weight increases the loss — you decrease it. If negative, you increase it. Every weight gets nudged in the direction that reduces the loss.
One update won't fix everything. But repeat this cycle — forward pass, compute loss, backpropagate gradients, update weights — thousands of times, and something remarkable happens. The loss drops. The predictions get better. The network learns.
| Step | What Happens |
|---|---|
| Forward pass | Input flows through the network → prediction |
| Loss | Compare prediction to truth → single error number |
| Backward pass | Chain rule traces error back through every layer |
| Update | Gradient descent nudges each weight to reduce loss |
This four-step loop is the heartbeat of every neural network. Forward. Loss. Backward. Update. Over and over until the network transforms from random noise into something that recognizes cats, translates languages, or writes code.
Why This Changed Everything
Before 1986, nobody knew how to efficiently train networks with more than one layer. The math was there, but applying the chain rule to computation graphs at scale seemed impractical. Backpropagation made it tractable — computing gradients for millions of weights in a single backward pass.
The algorithm itself is elegant: it's just the chain rule applied systematically to a computation graph. No special math. No secret sauce. The power comes from applying it at scale, with enough data and compute to make the loss converge.
Every breakthrough in modern AI — from image recognition to language models — runs on this exact loop. The architectures change. The data changes. The scale changes. But the learning algorithm at the core is still backpropagation.
Watch the full animated breakdown: Backpropagation: The Algorithm That Taught Machines to Learn
Neural Download — visual mental models for computer science and machine learning.
Top comments (0)