Part 5 of my "revisiting my AI/ML notes" series. So far this series has covered the ANN structure, forward propagation, and activation functions — all the machinery for a network to produce a prediction. This post covers the other half: how the network actually learns from being wrong.
The problem forward propagation doesn't solve
Forward propagation gives you a prediction. On its own, that prediction is basically a random guess — the weights start out essentially arbitrary. Backpropagation is the algorithm that takes the error in that prediction and uses it to adjust every single weight in the network, so the next prediction is a little bit better.
Step 1: measure how wrong the prediction is
This is the loss function. A simple and common one is squared error:
Loss = (y - y_hat)^2
Where y is the actual/true value and y_hat is the network's prediction. We square the difference for two reasons: it makes the loss positive regardless of whether the prediction was too high or too low, and it penalizes bigger errors disproportionately more than smaller ones.
Across a full dataset of n examples, the total loss is just the sum of this across every example:
Loss = sum over i=1 to n of (y_i - y_hat_i)^2
A worked example from my notes
Say a model is predicting whether someone passes an exam based on hours spent on Play (2h), Study (4h), and Sleep (8h), where 1 = pass. The true label is y = 1, but the network's current (untrained) prediction is y_hat = 0.
A worked example from my notes
Say a model is predicting whether someone passes an exam based on hours spent on Play (2h), Study (4h), and Sleep (8h), where 1 = pass. The true label is y = 1, but the network's current (untrained) prediction is y_hat = 0.
Loss = (1 - 0)^2 = 1
A loss of 1 here is about as bad as this particular setup gets — the prediction is completely wrong. That's the signal backpropagation uses to correct the weights.
Step 2: figure out which direction to adjust each weight
This is the actual "backward" part. For every weight in the network, backpropagation computes the partial derivative of the loss with respect to that weight — written dLoss/dw. This derivative tells you two things: which direction increasing the loss lies in, and how sensitive the loss is to that particular weight.
The update rule is the same for every weight in the network:
w_new = w_old - learning_rate * (dLoss / dw)
Notice the minus sign — you always move the weight in the opposite direction of the gradient, because you want to walk downhill toward lower loss, not uphill toward more of it.
Step 3: repeat, many times
One update rarely gets a weight anywhere close to optimal. Backpropagation runs this update for every weight, over and over across many training examples and many passes through the dataset, gradually walking every weight toward the values that minimize total loss — what's usually called the global minimum of the loss function.
Why the learning rate matters so much
The learning rate (lr in the formula above) controls how big each step is. This is a genuine balancing act:
- Too high, and updates overshoot the minimum — the loss can bounce around or even diverge instead of settling down.
- Too low, and training crawls: technically correct, but painfully slow, and it can get stuck in a shallow local dip instead of finding the true global minimum.
A small, carefully chosen learning rate is what lets training descend smoothly toward the minimum instead of overshooting past it.
Why this matters
Backpropagation is the algorithm that makes "deep learning" actually mean something — it's the mechanism that turns a randomly initialized network into one that's genuinely learned patterns from data. Every architecture covered so far in this series (and every architecture still to come — CNNs, RNNs, transformers) relies on this exact same core loop: predict, measure the error, propagate it backward, update the weights, repeat.
Original Note:
Top comments (0)