Every AI model learns using the same idea, called gradient descent. I built it from scratch in Python to understand how it works, and in this post I'll explain it as simply as I can.
Every AI model makes mistakes
An AI model is a large set of numbers, called weights. When you give the model an input, it uses those weights to make a guess. At the start, the weights are random, so the guesses can be very bad. To improve, we measure how wrong the guesses are with a single number called the error, and then try to make that error as small as possible.
For example, let's say we're at a shop where every item costs £3, but the model doesn't know that. The model guesses the total price using the formula:
guess = w * x
where x is the number of items bought, and w is the weight: the model's guess for the price of one item.
Now we show it a real example: 2 items cost £6. So the guess is 2w, and we measure the error like this:
error = (2w - 6)²
We square it so that guessing too high and guessing too low both count as bad. Here's the error for a few different weights:
| w | Guess for 2 items | Error |
|---|---|---|
| 1 | £2 | 16 |
| 2 | £4 | 4 |
| 3 | £6 | 0 |
| 4 | £8 | 4 |
| 5 | £10 | 16 |
Notice, the error goes down, hits 0 at w = 3, then goes back up. If you drew it, it would make a bowl shape. Finding the best weight means finding the bottom of this bowl.
The derivative is a compass
The model doesn't know where the bottom of the bowl is. But it can work out the derivative of the error, which is the slope of the bowl where it's standing. The derivative hands us a compass:
- If the derivative is negative, the error is dropping as we go right. So move right to reduce it.
- If the derivative is positive, the error rises as we go right. So move left to reduce it.
- If the derivative is zero, the ground is flat, which means we're at the bottom of the bowl. So stop there.
The goal is always to make the error as small as possible. The derivative tells us which way is downhill, and when it reaches 0, we know we've got to the bottom.
Gradient descent
Gradient descent is the recipe for walking down to the bottom of the bowl. There are 4 steps:
- Make a random guess for the weight.
- Work out the derivative at that weight.
- Take a small step in the opposite direction to the derivative.
- Keep repeating steps 2 and 3 until the derivative is very close to 0.
To work out the next weight, we use this formula:
new_weight = old_weight - (learning_rate * derivative_at_old_weight)
Notice: we subtract, and this is what makes us follow the compass. If the derivative is negative, subtracting a negative means adding, so we move right. If the derivative is positive, subtracting a positive means taking away, so we move left.
The learning rate is a small number (like 0.1) that controls how big each step is. If the learning rate is too big, each step jumps right over the bottom to the other side, and you keep bouncing back and forth without ever settling. If it's too small, you will get there eventually, but it takes a very long time.
Gradient descent example
The shop error (2w - 6)² is the same as 4(w - 3)², so its bottom is also at 3. To keep the numbers simple, I'll use the bowl (x - 3)², where x is the weight:
f(x) = (x - 3)²f'(x) = 2x - 6learning_rate = 0.1
Let's start our guess at x = 1 and see what the derivative is:
f'(1) = 2(1) - 6
f'(1) = -4
The derivative at x = 1 is -4, not 0. Remember, when the derivative is negative, we move right. Let's use our formula to work out the next x:
new_x = old_x - (learning_rate * derivative_at_old_x)
new_x = 1 - (0.1 * -4)
new_x = 1 - -0.4
new_x = 1 + 0.4
new_x = 1.4
Now let's work out the derivative at our new x:
f'(1.4) = 2(1.4) - 6
f'(1.4) = -3.2
It's still negative, but less steep than before, so we keep going. Here's how the next steps go:
| Step | x | Derivative at x |
|---|---|---|
| 0 | 1 | -4 |
| 1 | 1.4 | -3.2 |
| 2 | 1.72 | -2.56 |
| 3 | 1.976 | -2.048 |
| 4 | 2.1808 | -1.6384 |
| 20 | 2.9769 | -0.0461 |
Notice, the steps get smaller as we get closer to the bottom, because the bowl gets flatter. By step 20, x is 2.9769 and the derivative is -0.0461, which is almost 0, so we're almost at the bottom. It never lands on exactly 3, but it gets so close that it doesn't matter.
Finding the derivative with code
In the example above, I worked out by hand that the derivative of (x - 3)² is 2x - 6. But a real AI model is one enormous formula, so working out its derivative by hand isn't possible. So in my code, I estimate the slope by taking a tiny step and seeing how much the function changes:
def numeric_slope(fn, x, h=0.0001):
return (fn(x + h) - fn(x)) / h
Then gradient descent is just a loop:
def gradient_descent(fn, x, learning_rate=0.1, steps=20):
for _ in range(steps):
slope = numeric_slope(fn, x)
x = x - learning_rate * slope
return x
This works for any function I give it, without me working out a single derivative.
Functions with more than one input
So far we've only had one input. A real AI model has millions of weights, but to keep it simple, let's look at a function with two inputs:
f(x, y) = x² + y²
How do we find the slope when there are two inputs? We ask 2 questions:
- If I nudge only x a little bit, and leave y the same, how much does f change?
- If I nudge only y a little bit, and leave x the same, how much does f change?
Each answer is called a partial derivative. A partial derivative is the slope in the direction of one input, with all the other inputs held still. To work one out, you treat the other input as if it were a normal number:
- The partial derivative with respect to x is
2x(y² is treated as a normal number, so it disappears). - The partial derivative with respect to y is
2y.
Putting the partial derivatives together in a list gives you the gradient:
gradient = [2x, 2y]
To find the gradient at the point (1, 2), we put in x = 1 and y = 2:
gradient = [2(1), 2(2)]
gradient = [2, 4]
Gradient descent with 2 inputs
Gradient descent with two inputs works in exactly the same way as with one. The point becomes a list, [x, y], and the formula becomes:
new_point = old_point - (learning_rate * gradient_at_old_point)
Let's go back to:
f(x, y) = x² + y²gradient = [2x, 2y]learning_rate = 0.1
The gradient at (1, 2) is [2, 4], which isn't [0, 0], so let's find a new point:
new_point = [1, 2] - (0.1 * [2, 4])
new_point = [1, 2] - [0.2, 0.4]
new_point = [0.8, 1.6]
Then we find the gradient at (0.8, 1.6):
gradient = [2(0.8), 2(1.6)]
gradient = [1.6, 3.2]
And we keep repeating until the gradient is very close to [0, 0], which happens at the bottom of the bowl, (0, 0).
One thing to know if you try this in Python: [1, 2] - [0.2, 0.4] doesn't work on normal lists, because Python doesn't know how to subtract one list from another. In my code, I wrote small helper functions to scale and subtract lists, one number at a time.
Finding the gradient with code
Again, I worked out by hand that the gradient of x² + y² is [2x, 2y]. In my code, I estimate it instead, by nudging one input at a time and returning both slopes as a list:
def numeric_gradient(fn, x, y, h=0.0001):
slope_x = (fn(x + h, y) - fn(x, y)) / h
slope_y = (fn(x, y + h) - fn(x, y)) / h
return [slope_x, slope_y]
Conclusion
A real model works the same way, just much bigger. It has millions (or billions) of weights instead of one or two, so the gradient is a list of millions of numbers, one for each weight. Training the model means:
- Make a guess and measure the error.
- Work out the gradient of the error for every weight.
- Nudge every weight a small step against its gradient, so the error goes down.
- Repeat, millions of times.
That loop is what people mean when they say a model is "learning".
Top comments (0)