How Training Actually Works (Backpropagation and Gradient Descent, From Scratch)

Written by Syed Muhammad Ali Raza

Last article we built self attention from scratch, the mechanism that lets a token look at every other token and decide what actually matters. But I left something hanging on purpose, every single weight matrix in that article, the ones that turned embeddings into queries, keys, and values, I just filled with random numbers. That's genuinely not how a real model works. Those numbers start random, sure, but they don't stay random, they get shaped, over and over, millions and millions of times, into values that actually produce coherent language.

This article is about that shaping process. How does a pile of random numbers turn into something that can finish your sentence, write working code, or explain a joke. The honest answer is two ideas working together, a loss function that measures how wrong the model currently is, and gradient descent, a genuinely simple, almost mechanical process for nudging every single weight a tiny bit in the direction that makes the model a little less wrong. Do that enough times, on enough data, and language understanding falls out the other end. Let's actually build this, by hand, so you see every gear turning.

A real life example before any math, hiking down a mountain in thick fog

Picture standing on the side of a mountain at night, in genuinely thick fog. You can't see the valley below, you can't see more than a few feet in any direction. Your entire goal is to get to the lowest point, but you have no map and no way to see the overall shape of the terrain.

Here's what you actually can do though. You can feel the ground right under your feet. You can tell which direction is downhill from exactly where you're standing, right now, in this immediate spot. So you take a small step in that downhill direction. Then you feel the ground again, from your new position, and take another small step downhill from there. Repeat this, small step, feel the slope, small step, feel the slope, and eventually, slowly, you work your way down toward a low point, even though you never once saw the whole mountain, never had a map, and only ever knew the slope directly beneath your feet at each single moment.

That, genuinely, completely, is gradient descent. The "mountain" is something called the loss landscape, a measure of how wrong the model currently is, across every possible setting of its weights. The "downhill direction" at any specific point is the gradient, a mathematical measurement of exactly which direction, and how much, to nudge each weight to reduce how wrong the model currently is. The "small step" is the actual weight update, applied over and over, millions of times, across an enormous multidimensional mountain range you could never visualize directly, but the mechanism is exactly this simple at every single step, feel the local slope, step downhill, repeat.

A second real life example, learning to shoot free throws

Here's a second angle on the same idea, one that captures why we need backpropagation specifically, not just gradient descent alone.

Imagine you're learning to shoot basketball free throws, and every single part of your shooting motion, your elbow angle, your wrist flick, your knee bend, your follow through, is a separate adjustable dial. You shoot, the ball misses two feet to the left. Now here's the actual hard question, which dial do you turn, and by how much, to fix that specific miss.

A bad basketball coach just says "try again" with no specific feedback. A genuinely good coach traces the miss backward through your form, the ball went left because your wrist flicked slightly sideways at release, which happened because your elbow wasn't quite lined up, which happened because your foot placement was off to begin with. Each part of your motion gets blamed for the miss in proportion to how much it actually contributed to it, and you adjust each one accordingly, a little correction distributed across the whole chain, not just one giant random change.

That backward tracing of blame, from the final mistake all the way back through every contributing step, in proportion to how much each one actually mattered, is exactly what backpropagation does inside a neural network. The loss function tells you how far the shot missed. Backpropagation is the coach tracing that miss backward through every single weight in the network, figuring out precisely how much each one contributed to the error, so gradient descent knows exactly which direction and how far to nudge every single one.

The pieces, defined properly now

With both analogies in your head, here's the actual vocabulary, precisely.

A loss function is a single number that measures how wrong the model's output currently is compared to what it should have been. Low loss, the model's doing well on this example. High loss, it's genuinely off. Every training step computes this number fresh.

Gradient descent is the process of adjusting every weight in the direction that would reduce that loss number, a small amount at a time, using the slope, the gradient, at the model's current position.

Backpropagation is the specific algorithm for actually computing that gradient, for every single weight in the network, efficiently, by working backward from the final loss through every layer that contributed to it, using the chain rule from calculus, which is really just a formal way of doing the "trace the miss backward through each part of the motion" idea from the free throw example.

Let's build all three from scratch

I'm going to build a genuinely tiny neural network, train it on a real, if small, task, and print the loss going down with your own eyes, using nothing but numpy. No frameworks hiding the mechanism from you.

Step 1, the task, and why it needs more than a straight line

We'll train a network to solve XOR, a classic tiny problem specifically chosen because it can't be solved by a simple straight line decision boundary, which forces the network to genuinely learn something nontrivial through this whole process, not just fit an obvious pattern.

import numpy as np

np.random.seed(42)

# XOR truth table, output is 1 only when inputs differ
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])

print("Inputs:")
print(X)
print("Expected outputs:")
print(y)

Step 2, initialize a tiny two layer network, with genuinely random weights

Exactly like the attention article, we start with random numbers. This time, we're actually going to change them through real training instead of leaving them random.

input_size = 2
hidden_size = 4
output_size = 1

W1 = np.random.randn(input_size, hidden_size) * 0.5
b1 = np.zeros((1, hidden_size))
W2 = np.random.randn(hidden_size, output_size) * 0.5
b2 = np.zeros((1, output_size))

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
    return x * (1 - x)

Step 3, the forward pass, making an actual prediction

def forward_pass(X, W1, b1, W2, b2):
    hidden_input = X @ W1 + b1
    hidden_output = sigmoid(hidden_input)

    final_input = hidden_output @ W2 + b2
    final_output = sigmoid(final_input)

    return hidden_output, final_output


hidden_output, prediction = forward_pass(X, W1, b1, W2, b2)
print("Predictions before any training, just random noise:")
print(prediction)

Run this before training, and the predictions are genuinely meaningless, close to random guesses, exactly what you'd expect from a network that's never learned anything yet.

Step 4, the loss function, measuring exactly how wrong we are

def mean_squared_error(predictions, targets):
    return np.mean((predictions - targets) ** 2)

loss = mean_squared_error(prediction, y)
print(f"Initial loss: {loss:.4f}")

Step 5, backpropagation, tracing the error backward, by hand

This is the part most tutorials skip by calling a library function. We're doing it explicitly, so you see the actual chain of blame being traced backward, exactly like the free throw coach.

def backward_pass(X, y, hidden_output, prediction, W2):
    # how wrong was the final output, and in which direction
    output_error = prediction - y
    output_delta = output_error * sigmoid_derivative(prediction)

    # trace that error backward through W2, to see how much
    # the hidden layer contributed to the final mistake
    hidden_error = output_delta @ W2.T
    hidden_delta = hidden_error * sigmoid_derivative(hidden_output)

    # now compute the actual gradient for every single weight,
    # how much each one should be blamed, and in which direction
    grad_W2 = hidden_output.T @ output_delta
    grad_b2 = np.sum(output_delta, axis=0, keepdims=True)
    grad_W1 = X.T @ hidden_delta
    grad_b1 = np.sum(hidden_delta, axis=0, keepdims=True)

    return grad_W1, grad_b1, grad_W2, grad_b2

Read that hidden_error line again, that's the exact "trace the miss backward through the elbow, then the wrist" moment. The error at the output gets propagated backward through W2 to figure out how much the hidden layer's output contributed, which is precisely backpropagation's whole job, distributing blame for a mistake backward through every layer that had a hand in causing it.

Step 6, gradient descent, actually taking the downhill step

learning_rate = 0.5

def train_step(X, y, W1, b1, W2, b2, learning_rate):
    hidden_output, prediction = forward_pass(X, W1, b1, W2, b2)
    loss = mean_squared_error(prediction, y)

    grad_W1, grad_b1, grad_W2, grad_b2 = backward_pass(X, y, hidden_output, prediction, W2)

    # the actual downhill step, nudge every weight a small amount
    # in the direction that reduces the loss
    W1 -= learning_rate * grad_W1
    b1 -= learning_rate * grad_b1
    W2 -= learning_rate * grad_W2
    b2 -= learning_rate * grad_b2

    return W1, b1, W2, b2, loss

Step 7, actually training, watching the loss go down with your own eyes

for epoch in range(10000):
    W1, b1, W2, b2, loss = train_step(X, y, W1, b1, W2, b2, learning_rate)

    if epoch % 1000 == 0:
        print(f"Epoch {epoch}, loss: {loss:.4f}")

_, final_prediction = forward_pass(X, W1, b1, W2, b2)
print("\nFinal predictions after training:")
print(np.round(final_prediction, 3))
print("\nExpected:")
print(y)

Run this yourself and watch the loss number actually fall, epoch after epoch, from something close to random guessing down to genuinely accurate predictions on a problem that literally cannot be solved by a straight line. Nobody hand coded the XOR logic anywhere in this file. The network found it, entirely through repeated small downhill steps, guided by nothing but the loss and the gradients computed from it. That is genuinely, completely, the entire mechanism, scaled up by many orders of magnitude, behind how every model in this whole series learned anything at all.

Scaling this up to an actual LLM

The toy network above has maybe thirteen weights total. A real LLM has billions. The mechanism is genuinely the same, forward pass, compute loss, backpropagate the error, take a downhill step, repeat, just at a scale that's hard to intuitively picture.

A few things change at that scale, worth knowing by name. The loss function for language models is usually cross entropy loss, not the mean squared error we used above, better suited for predicting which token comes next out of a huge vocabulary of possibilities, but it's playing the exact same role, a single number saying how wrong the prediction was. The optimizer, the thing actually applying the gradient to update weights, is usually something more sophisticated than the plain gradient descent we wrote by hand, commonly Adam, which adapts the step size per weight based on recent gradient history, but underneath the sophistication, it's still fundamentally "step in the direction that reduces loss." And instead of training on four tiny XOR examples, models train on enormous datasets, processing data in batches, taking millions or billions of these small downhill steps over the course of training, each one nudging billions of weights by a tiny amount.

def scale_intuition():
    toy_network_weights = 13
    gpt_scale_weights = 175_000_000_000  # roughly, for reference

    print(f"Our toy network: {toy_network_weights} weights")
    print(f"A large LLM: ~{gpt_scale_weights:,} weights")
    print(f"That's roughly {gpt_scale_weights / toy_network_weights:,.0f}x more dials being turned")
    print("Same mechanism though, forward pass, loss, backward pass, small step, repeat")

scale_intuition()

Genuinely the same four steps, at a scale that turns "learns XOR in a few seconds on your laptop" into "learns to write working code and hold a conversation after weeks of training on massive computing clusters." The mechanism didn't get more complicated, it just got applied an almost unimaginable number of times, across an almost unimaginable number of weights.

The honest hard parts I'm not covering here

I want to be straight about what this article did and didn't cover, since oversimplifying training would do you a disservice. We used plain gradient descent with a fixed learning rate, real training almost always uses adaptive optimizers and learning rate schedules that change over the course of training. We trained on the full dataset every single step, real LLM training uses mini batches, small random subsets of the data per step, both for computational reasons and because it turns out to genuinely help the training process. We didn't touch how the actual transformer architecture from the previous article gets trained specifically, attention layers, feedforward layers, and everything else all get their own gradients computed and weights updated through this exact same backpropagation and gradient descent mechanism, just with a lot more moving parts than our tiny two layer network. And we didn't touch the different training stages, an LLM typically goes through pretraining on raw text, then further stages like instruction tuning and reinforcement learning from human feedback, each using this same core mechanism but with different data and different loss signals, which is genuinely a topic worth its own article later in this arc.

Bringing this back to the whole arc

Last article, attention let a model relate tokens to each other within a single forward pass, using weight matrices we just filled with random numbers. This article, those weights stop being random, shaped through millions of small downhill steps, each one nudged by a gradient computed through backpropagation, aimed at reducing a loss that measures exactly how wrong the model currently is. Put these two articles together, and you've genuinely got the two central ideas behind how every model in the entire previous series actually became capable of anything at all, a mechanism for relating information within an input, and a mechanism for learning, from data, what those relationships should actually look like.

---

If you run the training loop yourself and watch that loss number fall on your own screen, that's honestly worth doing even if you already believed me on paper, there's something different about watching thirteen random numbers turn into a working XOR solver right in front of you.