Before I Could Build GPT, I Had to Teach a Machine to Learn

30 July 2026.

I didn't start this journey because I wanted to build another AI chatbot.

I started because I was using AI tools regularly, and I realised something slightly uncomfortable: I knew how to use them, but I couldn't properly explain what was happening underneath.

I could ask ChatGPT to write code, explain a concept, or help debug something. But if someone asked me, "How does a language model actually learn?", I would have struggled to give an answer beyond a few familiar terms.

Neural networks. Training data. Transformers. Attention. Weights.

I knew the vocabulary. I wanted to understand the machinery.

As a JavaScript developer, I also had a preference: I wanted to build things myself. I didn't want to import a library, call a train() function and pretend I understood what happened.

So I set myself a challenge:

Could I understand the fundamentals of AI by implementing them in JavaScript, from scratch?

No TensorFlow. No PyTorch. No machine-learning framework hiding the mathematics. Just JavaScript, arrays, loops, and a lot of patience.

I had no idea how many times I would end up staring at a number that had suddenly become NaN.

But that was part of the journey.

---

1. I had to stop thinking of AI as magic

Before getting anywhere near GPT, I needed to understand a much smaller problem.

Imagine I give a program two numbers:

const x1 = 2;
const x2 = 3;

And I want it to predict their sum.

The answer is obviously 5. I can write:

const result = x1 + x2;

But that isn't machine learning. I've already told the program exactly what to do.

What if I don't tell it the relationship? What if I only give it examples?

Input 1	Input 2	Expected output
2	3	5
4	1	5
5	5	10
7	2	9

Now the program needs to discover the relationship from the examples.

This is the basic idea behind supervised learning: provide examples with known answers, make predictions, measure the errors, and adjust the model so that future predictions improve.

Supervised learning is a loop: show the model examples with known answers, make a prediction, compare it with the expected output, and adjust the weights.

The important change in thinking was this:

Instead of programming the answer, I was programming a system that could learn how to produce the answer.

That distinction became the foundation of everything that followed.

2. What is a neuron, really?

A neuron is a small mathematical operation.

It receives inputs, multiplies them by weights, adds a bias, and produces an output.

For two inputs:

$$z = x_1 w_1 + x_2 w_2 + b$$

Here:

• $x_1, x_2$ are inputs.
• $w_1, w_2$ are weights.
• $b$ is a bias.
• $z$ is the result before activation.

A neuron scales its inputs by weights, adds a bias to get the weighted sum z, then applies an activation f to produce the output ŷ.

Let's implement it.

function neuron(x1, x2, w1, w2, bias) {
  return x1 * w1 + x2 * w2 + bias;
}

const prediction = neuron(
  2,
  3,
  0.5,
  0.8,
  0.1
);

console.log(prediction);

The weights determine how strongly each input contributes to the result. The bias lets the model shift the output.

At first, these values can be random. The model doesn't know the correct relationship yet.

And that leads to the next question:

How does it know whether its prediction is good or bad?

3. Loss: giving the model a way to measure its mistakes

Suppose the expected answer is 5, but our neuron predicts 4.

We need a way to measure that difference.

One simple option is squared error:

$$L = (\hat{y} - y)^2$$

Where:

• $\hat{y}$ is the prediction.
• $y$ is the expected answer.
• $L$ is the loss.

In JavaScript:

function loss(prediction, expected) {
  const difference = prediction - expected;
  return difference * difference;
}

console.log(loss(4, 5)); // 1
console.log(loss(2, 5)); // 9

A prediction of 2 produces a larger loss than a prediction of 4.

Squared error L = (ŷ − y)² is a parabola centred on the expected answer. ŷ = 4 gives L = 1, ŷ = 2 gives L = 9, and the loss is zero only when ŷ = y = 5.

This gives us a measurable objective: reduce the loss.

I found this idea surprisingly important. A model doesn't simply "know" that it made a mistake. We define a mathematical function that measures how wrong its prediction was.

And then we need to use that measurement to improve the model.

4. Gradient descent: how do we improve the weights?

Imagine the loss function as a landscape.

Our weights determine where we stand on that landscape. Some combinations produce a high loss; others produce a lower loss.

Gradient descent tells us which direction to move the weights to reduce the loss.

The basic update rule is:

$$w_{\text{new}} = w_{\text{old}} - \eta \frac{\partial L}{\partial w}$$

Here, $\eta$ is the learning rate.

The derivative tells us how sensitive the loss is to a particular weight. The learning rate controls how large an update we make.

Gradient descent steps downhill on the loss. A well-chosen learning rate converges toward the minimum; one that is too large overshoots and the steps grow.

In code, the update itself is straightforward:

function updateWeight(weight, gradient, learningRate) {
  return weight - learningRate * gradient;
}

The difficult part is calculating the gradient correctly.

That led me to one of the most important concepts in the entire journey: backpropagation.

5. Backpropagation: tracing the mistake backwards

Let's return to our neuron:

$$\hat{y} = x_1 w_1 + x_2 w_2 + b$$

And use squared error:

$$L = (\hat{y} - y)^2$$

We want to know how changing $w_1$ , $w_2$ , or $b$ affects the loss.

Using the chain rule:

$$\frac{\partial L}{\partial w_1} = 2(\hat{y} - y)x_1$$

Similarly:

$$\frac{\partial L}{\partial w_2} = 2(\hat{y} - y)x_2$$

And:

$$\frac{\partial L}{\partial b} = 2(\hat{y} - y)$$

These gradients tell us how each parameter contributed to the error.

The forward pass computes ŷ and the loss. The backward pass applies the chain rule so that ∂L/∂w₁ = 2(ŷ − y)x₁, and likewise for w₂ and b.

This is the intuition behind backpropagation: calculate how the error at the output depends on the intermediate calculations and parameters, then pass that information backwards through the network.

It isn't magic. It's calculus applied systematically.

Once I understood that, neural networks became much less mysterious.

6. From one neuron to a trainable network

A single neuron is useful for understanding the basic mathematics, but real models need many neurons connected across layers.

I moved from simple linear models to a trainable multilayer perceptron.

One of the architectures I worked with was:

2 inputs
   |
   v
4 neurons
   |
   v
Activation function
   |
   v
1 output

The network could learn a simple relationship such as:

$$y = x_1 + x_2$$

A 2–4–1 multilayer perceptron: two inputs, four hidden neurons with an activation function, and one output, fully connected layer to layer.

The architecture introduced another important idea: activation functions.

Without suitable nonlinear activations, stacking linear transformations still produces an overall linear transformation. That limits what the network can learn.

So I explored several activation functions.

ReLU

$$\operatorname{ReLU}(x) = \max(0, x)$$

Simple, fast, and commonly used.

Tanh

$$\tanh(x)$$

Produces values between -1 and 1.

Leaky ReLU

Unlike ordinary ReLU, it allows a small negative slope for negative inputs.

This helps avoid neurons that stop producing useful gradients for negative inputs.

GELU

GELU is smoother than ReLU and is commonly used in Transformer architectures.

The four activation functions I explored. Without a nonlinearity, stacking layers would still be a single linear transformation.

I implemented GELU myself and checked its derivative numerically.

That brought me to another lesson.

7. Gradient checking: how do you know your mathematics is correct?

When implementing backpropagation manually, a small mathematical mistake can quietly ruin training.

A gradient might be incorrect. The model might still run. The loss might even change. But the learning behaviour could be wrong.

So I implemented numerical gradient checking.

The idea is simple.

If we slightly increase a parameter, calculate the loss, then slightly decrease it and calculate the loss again, we can estimate its derivative:

$$\frac{\partial L}{\partial w} \approx \frac{L(w + \epsilon) - L(w - \epsilon)}{2\epsilon}$$

Numerical gradient checking compares a secant through L(w − ε) and L(w + ε) with the analytic tangent from backpropagation. If they agree, the derivative is likely correct.

Then we compare that numerical estimate with the gradient produced by our backpropagation implementation.

If they are sufficiently close, that's evidence that our derivative implementation is correct.

This became one of my favourite debugging techniques because it gave me a way to test the mathematics rather than just hoping the code was right.

8. And then came the bugs

Not everything worked smoothly.

There were exploding gradients. There were NaN values. There were dead neurons. There were situations where a network would train for a little while and then produce completely unusable numbers.

Loss over epochs for three runs: healthy training, exploding gradients, and dead neurons that flatten out and then become NaN.

One of the lessons was that a correct-looking architecture doesn't guarantee stable training.

The learning rate matters. Weight initialization matters. Activation functions matter. Gradient calculations matter.

A model can fail for reasons that aren't obvious from looking at the final output.

And that's why implementing these pieces myself was so useful. I wasn't just seeing that training failed. I could inspect the values and understand where the failure might be happening.

9. What I took away from Part 1

By this point, I had a much clearer picture of how a basic neural network learns.

The process looks something like this:

1. Give the model an input.
2. Calculate a prediction using weights and biases.
3. Compare the prediction with the expected output.
4. Calculate the loss.
5. Use backpropagation to calculate gradients.
6. Update the weights using gradient descent.
7. Repeat.

That simple loop is an important foundation for understanding much larger models.

But there was still a huge gap between a small neural network that learns to add two numbers and a language model that can generate paragraphs.

A language model has to process sequences. It needs to represent text, understand relationships between tokens, and predict what comes next.

That meant I was ready to move beyond ordinary neural networks.

I had learned how to teach a machine to learn. Now I wanted to understand how a machine learns language.

---

Coming next: Part 2 — The Transformer: How GPT Learns to Pay Attention.

In the next article, I'll explore tokenization, embeddings, positional information, Query-Key-Value projections, self-attention, multi-head attention, residual connections and LayerNorm — the building blocks that eventually led to our own GPT implementation.

If you'd like to follow along, subscribe to the series so you don't miss Part 2.

Project journey started: 30 July 2026.