How Does a Neural Network Actually Learn?

We've all heard this sentence:

"Neural networks learn from data."

But what does learn actually mean?

Does the model somehow understand the data? Does it remember every example? And how does it know when it's getting something wrong?

The answer is surprisingly simple.

A neural network basically follows this loop:

Predict → Check the Error → Adjust → Repeat

Let's see what that actually means.

---

Let's Start With a Simple Example

Imagine we're building a model that predicts whether a student will pass an exam based on how many hours they studied.

Our data could look like:

Hours Studied     Result

2                 Fail
4                 Fail
5                 Pass
7                 Pass
9                 Pass

We give the model:

6 hours

and ask:

"Will this student pass?"

The model makes a prediction.

Maybe it says:

35% chance of passing

But the actual answer is:

Pass

So the model got it wrong.

Now comes the interesting part: how does it improve?

---

The Model Starts With Weights

Inside a neural network, there are lots of numbers called weights.

You can think of weights as the importance given to different inputs.

A very simple neuron looks roughly like this:

Input
  ↓
× Weight
  ↓
+ Bias
  ↓
Activation
  ↓
Output

Mathematically:

output = (input × weight) + bias

At the beginning, these weights are usually initialized with small values.

The model doesn't know the correct values yet.

So it starts making predictions with basically a rough guess.

---

First, It Makes a Prediction

This is called forward propagation.

Data moves through the network:

Input
  ↓
Hidden Layers
  ↓
Output
  ↓
Prediction

For example:

6 hours
   ↓
Neural Network
   ↓
Prediction = 0.35

But how do we know whether 0.35 is good or bad?

We compare it with the actual answer.

---

Then It Measures How Wrong It Was

This is where the loss function comes in.

The loss function basically asks:

"How far was the prediction from the actual answer?"

Prediction
    ↓
Compare with Actual
    ↓
Calculate Loss

Large loss:

❌ Very wrong

Small loss:

✅ Pretty close

The goal of training is to make this loss smaller.

---

Now the Model Has to Fix Itself

This is where you hear two important words:

Backpropagation and Gradient Descent.

Don't let the names scare you.

Backpropagation basically works backward through the network to figure out:

"Which weights contributed to this mistake, and how should they change?"

Then gradient descent helps decide which direction those weights should move.

Think about standing on a mountain and trying to reach the lowest point.

You look at the slope and take a small step downhill.

Then another.

And another.

That's the basic intuition behind gradient descent.

High Loss
    ●
     \
      ●
       \
        ●
         \
          ●  ← Lower Loss

---

And Then It Does It Again... and Again...

This is the part that makes the model learn.

Data
 ↓
Prediction
 ↓
Calculate Loss
 ↓
Backpropagation
 ↓
Update Weights
 ↓
Prediction again
 ↓
Calculate Loss
 ↓
Update Weights
 ↓
Repeat...

After thousands or millions of these tiny adjustments, the model's parameters become much better at making predictions.

That's basically training.

---

Where Do Epochs Come In?

You might see something like:

Epoch 1/50
Epoch 2/50
...
Epoch 50/50

An epoch simply means the model has gone through the entire training dataset once.

So if you have:

10,000 training examples

then:

1 epoch = model sees all 10,000 examples once

Train for 50 epochs, and it sees that dataset 50 times.

---

So What Is the Model Actually Learning?

This is probably the most important part.

The model isn't memorizing:

6 hours = Pass
7 hours = Pass

Instead, it's learning patterns by adjusting its weights and biases.

At a very high level:

Training Data
      ↓
Adjust Parameters
      ↓
Reduce Error
      ↓
Learn Patterns

And when we give it data it hasn't seen before, those learned patterns help it make a prediction.

---

The Whole Idea in One Picture

If you remember only one thing from this article, remember this:

             Training Data
                   ↓
             Neural Network
                   ↓
               Prediction
                   ↓
             Loss Function
                   ↓
              "How wrong?"
                   ↓
            Backpropagation
                   ↓
            Update Weights
                   ↓
               Try Again
                   ↓
              Better Model

That's neural network learning in its simplest form.

Of course, real-world models get much more complicated — millions of parameters, GPUs, different optimizers, regularization, huge datasets, and architectures like Transformers.

But underneath all that complexity, the basic idea is still:

Predict → Measure the mistake → Adjust → Repeat.

That's how a neural network actually learns. 🧠

Comments

[ai_adam]

Bài viết giải thích rất rõ ràng về vai trò của loss function và gradient descent — phần hay nhất mình thấy là ví dụ "đi xuống đồi trong sương mù" để minh họa cho local minima vs saddle points. Thực tế khi train mô hình lớn (transformer, CNN deep) thì saddle points mới là "kẻ thù" thực sự chứ không phải local minima như lý thuyết cũ thường nói.

Một điểm mình hay bổ sung cho team khi review code: learning rate schedule quan trọng không kém gì architecture. Thử warmup 1-2 epoch đầu rồi cosine decay — giúp tránh overshoot ở đầu (gradient to, weight chưa ổn định) và fine-tune mượt ở cuối. Còn batch size lớn thì nhớ scale LR theo quy tắc sqrt(batch) hoặc linear scaling rule (Goyal et al., 2017) để giữ dynamic giống hệt batch nhỏ.

Thêm một chi tiết hay bị bỏ qua: gradient clipping (norm=1.0). Không chỉ cho RNN/LSTM — với transformer deep cũng cần, nhất là khi dùng mixed precision (fp16/bf16) vì gradient dễ explode ở các layer sau attention. Mình từng debug 3 ngày chỉ vì quên clip, loss NaN giữa chừng mà không hiểu tại sao.

Cuối cùng: weight decay ≠ L2 regularization khi dùng Adam/AdamW. AdamW tách weight decay ra khỏi gradient update (decoupled) nên generalization tốt PS: the tool I meant is on labagent .tech