Andrej Karpathy's Neural Networks: Zero to Hero — 1) Intro to Neural Networks and Backpropagation

Introduction

Andrej Karpathy uploaded several lecture videos on YouTube and the accompanying code on GitHub. I think they are excellent lectures, even better than many paid online courses. Here's the link: Neural Networks: Zero to Hero. So, I will summarize them and try to get meaningful insights from them. I'm going to cover all of them lecture by lecture (I hope...)

The first lecture is about backpropagation in neural networks.

 

The Spelled-out Intro to Neural Networks and Backpropagation

In the video, Karpathy said that backpropagation is what you need to train neural networks, and everything else is mainly for efficiency. That is why he explained and demonstrated backpropagation in the very first video.

I totally agree with him. Training neural networks and LLMs is essentially about reducing loss. And fundamentally, all methods for reducing loss are related to backpropagation, directly or indirectly.
 

Building Micrograd

Karpathy built a small project called micrograd. You can see the code here. This is made up of just a few simple lines of code, but it shows us how neural networks are built under the hood. In the video, he demonstrated how to build Micrograd and how it works step by step.
 

Differentiation and Derivative

We all learned differentiation and derivatives, right? What do differentiation and derivatives actually mean?

$$\lim_{h \to 0} \frac{f(x + h) - f(x)}{h}$$

Differentiation and derivatives tell us how much $f(x)$ changes when $x$ changes. That is, they show the effect of a variable and the slope or gradient of a function. Then, if the derivative is 0, that point may be a local maximum or minimum of the function—not always, but it can be.

Actually, Karpathy didn't explain differentiation and derivatives in detail in the video. However, I think this is one of the most important aspects for understanding neural networks. So I'm going to explain this in more detail.
 

Linear Regression and Derivative

This concept is fundamental to linear regression as well. What is the core idea of linear regression? The goal is to minimize the residual sum of squares (RSS).

$$\begin{align*} RSS &= \sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2 \\ &= \sum_{i=1}^{n} \left( y_i - \beta_{1} x_i - \beta_{0} \right)^2 \end{align*}$$

This is what linear regression is all about: finding the optimal $\beta$ values. Then, how to find them? This is where the derivative comes in. The RSS formula is a kind of quadratic function, so when it comes to quadratic functions, the minimum point is the point at which the derivative becomes 0. Therefore, if we differentiate the equation and find where the derivative is zero, we can find the best $\beta_1$ and $\beta_0$ .

To find the optimal $\beta_0$ and $\beta_1$ , we take the partial derivatives of RSS with respect to each parameter and set them equal to zero. I will demonstrate how to derive them.

$$RSS(\beta_0, \beta_1) = \sum_{i=1}^{n} \left(y_i - \beta_1 x_i - \beta_0 \right)^2$$

First, differentiate the RSS with respect to $\beta_0$ :

$$\frac{\partial RSS}{\partial \beta_0} = -2 \sum_{i=1}^{n} \left(y_i - \beta_1 x_i - \beta_0 \right)$$

Then, differentiate the RSS with respect to $\beta_1$ :

$$\frac{\partial RSS}{\partial \beta_1} = -2 \sum_{i=1}^{n} x_i \left(y_i - \beta_1 x_i - \beta_0 \right)$$

At the minimum point, both partial derivatives are zero:

$$\begin{align*} \frac{\partial RSS}{\partial \beta_0} &= 0 \\ \frac{\partial RSS}{\partial \beta_1} &= 0 \end{align*}$$

This gives us the normal equations:

$$\begin{align*} \sum_{i=1}^{n} y_i &= \beta_1 \sum_{i=1}^{n} x_i + n\beta_0 \\ \sum_{i=1}^{n} x_i y_i &= \beta_1 \sum_{i=1}^{n} x_i^2 + \beta_0 \sum_{i=1}^{n} x_i \end{align*}$$

Solving these equations gives the optimal values:

$$\beta_1 = \frac{ \sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y}) }{ \sum_{i=1}^{n} (x_i - \bar{x})^2 }$$

$$\beta_0 = \bar{y} - \beta_1 \bar{x}$$

Here, $\bar{x}$ is the mean of the input values, and $\bar{y}$ is the mean of the target values.

Neural networks are also built from these kinds of linear expressions, usually combined with nonlinear activation functions. However, the way we calculate the parameters is totally different because neural networks are much more complicated and have so many parameters. So it is almost impossible to find the parameters in this way.
 

Finding Your Way in Pitch Darkness

Finding your way in pitch darkness

Assume that while hiking in the mountains, we get lost and trying to get down the mountain. But it is night, so we are in pitch darkness. We can only see a few inches around us. In this case, how can we get down the mountain? The answer is simple: by following the slope downward. At least, if we can see the slope around us, we can tell which way leads downward. This is how we find the optimal point when the equation is so complex that we are not able to solve for the optimum analytically.

def f(x):
  return 3*x**2 - 4*x + 5

f(3.0)

xs = np.arange(-5, 5, 0.25)
ys = f(xs)
plt.plot(xs, ys)

 

Visualization of the quadratic function

This is an example from Karpathy's code. The function is $f(x) = 3x^2 - 4x + 5$ . Its derivative is $\frac{d f(x)}{d x} = 6x - 4$ .

If we solve the equation $6x - 4 = 0$ , the derivative is 0 at $x = 2/3$ . Then, if $x$ is at some other point, how can we move $x$ to find the minimum point of $f(x)$ ?

The answer is simple. If the derivative value at a certain point is greater than 0, we have to decrease $x$ ; if it is less than 0, we have to increase $x$ . Therefore, if we consistently subtract $\lambda (6x - 4)$ from $x$ , with a proper learning rate $\lambda$ , $f(x)$ will converge to its minimum.

This finite-difference approach is useful for building intuition and for gradient checking.

h = 0.000001
x = 2/3
(f(x + h) - f(x))/h

The output is 2.999378523327323e-06, which is almost zero. It is not perfectly exact because floating-point numbers have limited precision, and because this is a finite-difference approximation, but it is close enough for this simple demonstration.

Similarly, when you have an expression with several variables, you can get the slope with respect to a specific variable in this way.

h = 0.0001

# inputs
a = 2.0
b = -3.0
c = 10.0

d1 = a*b + c
c += h
d2 = a*b + c

print('d1', d1)
print('d2', d2)
print('slope', (d2 - d1)/h)

 
Output:

d1 4.0
d2 4.0001
slope 0.9999999999976694

This is also Karpathy's code. This example shows how $d$ changes when $c$ changes from 10.0. The gradient is 1.0, of course, since the derivative $\frac{d}{dc}(ab + c)$ is 1.
 

Calculating the Gradient

Karpathy demonstrates a hands-on example of how to calculate the gradients. Let's see one of his examples.

$$\begin{align*} a &= -2.0 \\ b &= 3.0 \\ d &= a \times b \\ e &= a + b \\ f &= d \times e \end{align*}$$

When $f$ is the final output, we should calculate the gradients with respect to all intermediate values. Let's do this one by one.

The gradient of $f$ with respect to itself is 1 because $\frac{d}{df}f = 1$ . Easy, right?

The important thing is that addition passes the gradient through. For example, if $x = a + b$ and $\frac{d}{dx}f(x) = k$ , the gradients of $a$ and $b$ are also $k$ . For subtraction, the subtracted term receives the negative of the upstream gradient. And when it comes to multiplication, the upstream gradient is multiplied by the other variable. If $x = a \times b$ , the gradient of $a$ is $k \times b$ because the local derivative with respect to $a$ is $b$ .

Therefore, the gradient of $e$ is $\frac{d}{de}f$ . $f$ is $d \times e$ , so the gradient $e$ is $d=-6.0$ . On the other hand, the gradient of $d$ is 1.0.

Finally, the gradient of $a$ is the contribution from $e$ plus the contribution from $d$ . So it is $-6 + 3 = -3$ . The gradient of $b$ is also the contribution from $e$ plus the contribution from $d$ , and that is $-6 - 2 = -8$ .

The computation graph for the final output looks like this:

What if we want to minimize $f$ by tuning the value of $a$ ? By subtracting the gradient, $-3\lambda$ , $f$ will get smaller since the function with respect to $a$ is an upward-opening quadratic function. If $\lambda$ is 0.1, we subtract -0.3 from $a$ . Then $a$ becomes -1.7. As a result, $f$ becomes -6.63, which is smaller.

Now, let's apply this algorithm to a neural network.
 

Neural Networks and Gradient Descent

I organized another example from Karpathy in the image above. This is a very simple neural network architecture that he made. Actually, this is not the whole story yet. What we want to minimize is the loss function. So, if $o$ is $\hat{y}$ , gradient descent should minimize $\sum_{i=1}^{n} \left( y_i - o_i \right)^2$

In this way, Karpathy shows hands-on code that runs gradient descent on a simple MLP.

Here's the MLP training loop Karpathy builds in the Micrograd lecture:

class Neuron:

  def __init__(self, nin):
    self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]
    self.b = Value(random.uniform(-1,1))

  def __call__(self, x):
    # w * x + b
    act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)
    out = act.tanh()
    return out

  def parameters(self):
    return self.w + [self.b]

class Layer:

  def __init__(self, nin, nout):
    self.neurons = [Neuron(nin) for _ in range(nout)]

  def __call__(self, x):
    outs = [n(x) for n in self.neurons]
    return outs[0] if len(outs) == 1 else outs

  def parameters(self):
    return [p for neuron in self.neurons for p in neuron.parameters()]

class MLP:

  def __init__(self, nin, nouts):
    sz = [nin] + nouts
    self.layers = [Layer(sz[i], sz[i+1]) for i in range(len(nouts))]

  def __call__(self, x):
    for layer in self.layers:
      x = layer(x)
    return x

  def parameters(self):
    return [p for layer in self.layers for p in layer.parameters()]

x = [2.0, 3.0, -1.0]
n = MLP(3, [4, 4, 1])
n(x)

xs = [
  [2.0, 3.0, -1.0],
  [3.0, -1.0, 0.5],
  [0.5, 1.0, 1.0],
  [1.0, 1.0, -1.0],
]
ys = [1.0, -1.0, -1.0, 1.0] # desired targets

for k in range(20):

  # forward pass
  ypred = [n(x) for x in xs]
  loss = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred))

  # backward pass
  for p in n.parameters():
    p.grad = 0.0
  loss.backward()

  # update
  for p in n.parameters():
    p.data += -0.1 * p.grad

  print(k, loss.data)

With his Micrograd code, you can see how the gradient descent algorithm works step by step. You can also calculate the gradient of each variable on your own. I strongly recommend doing these hands-on examples. After watching the video, I was able to clearly understand how gradient descent works, why we should use zero out gradients, why ReLU function is the most efficient, and so on. This is definitely worth your time.

---

Conclusion

I have shown Karpathy's demonstrations of gradient descent. As he said, this is the core concept for training neural networks. The rest is just for efficiency. Reducing loss using gradients: this is what makes neural network training possible and, ultimately, helped usher in the AI era.