DEV Community: Logit Solutions AI

How does a rolling ball skip small bumps?

Syed Abdul Hanan — Sun, 01 Nov 2026 01:20:00 +0000

Momentum, step by step: give gradient descent a memory, and it rolls past the traps that stop it.

Plain gradient descent looks only at the slope under its feet. Every step is

w ← w − η ∇L(w)

and nothing else. That makes it simple, but it also makes it forgetful: the moment the ground goes flat, it stops. Momentum fixes this with one extra number per weight, a velocity that remembers the slopes it has already rolled down. The examples below follow the CS878 lectures on gradient descent and scaling the optimization, and every number comes from running the update rule.

The trap

Take one weight and a loss with two dips:

L(w) = 0.12 (w − 2.5)² − 0.55 · exp(−(w + 1)² / 0.12)

Start at A, w = −3.5, where the loss is 4.32. There is a shallow dip B at w = −0.904 (loss 0.881) and the real minimum C at w = 2.5 (loss 0). Between them sits a bump only 0.13 higher than B.

Plain gradient descent with η = 0.1 takes big steps on the steep slope, smaller and smaller ones as it nears B, and by step 21 it has arrived. At B the gradient is zero, so every step after that is zero too. At step 50, at step 100, it is still in B, at a loss of 0.88.

Plain gradient descent stops in the shallow dip B: the gradient is zero there, so learning stops.

Add a velocity

Momentum keeps a velocity v and moves along it:

v_t = β v_(t−1) + ∇L(w_t)
w_(t+1) = w_t − η v_t

With β = 0.9, each step keeps 90% of the old velocity and adds the new gradient. (This is the form PyTorch uses. Lecture notes often write v = βv + (1 − β)∇L, which only rescales the learning rate.)

Run it from A with the same η = 0.1. By step 6 the ball is near B and its velocity is about 6. The gradient at B is zero, but the velocity isn't, so the ball keeps going: it clears the bump at step 7, overshoots all the way to w = 4.71, swings back without ever returning to the bump, and settles in C by step 89.

Momentum carries the ball over the bump; plain GD's ghost is still stuck in B.

What v remembers

Unroll the velocity and it becomes a weighted sum of every gradient so far:

v_t = ∇L_t + β ∇L_(t−1) + β² ∇L_(t−2) + …

The latest gradient counts fully, the one before 90%, then 81%, 73%, 66% and so on. The weights add up to 1/(1 − β) = 10, so momentum remembers about ten steps. With β = 0.1 they would be 1, 0.1, 0.01: it forgets almost at once. In practice β is usually around 0.9.

Eight bars, heights β^k, sum 1/(1 − β) = 10: gradients that agree stack toward 10; gradients that flip cancel, staying within ±0.53.

Agree or cancel

That memory does two different things depending on the gradients:

When they agree (+1, +1, +1 …), they stack: v grows 1, 1.9, 2.71, 3.44 … toward 1/(1 − β) = 10. The step grows up to ten times.
When they flip sign (+1, −1, +1 …), they cancel: v goes 1, −0.1, 0.91, −0.18, 0.84, −0.25 … and settles at about ±1/(1 + β) ≈ 0.53. The step stays small.

A narrow valley

Now two weights and a valley that is steep across and almost flat along: L = ½(w₁² + 50 w₂²), starting at (−4, 1), where the loss is 33. Count the steps until the loss drops below 0.001, with η = 0.015:

Plain GD: 298 steps. A bigger rate doesn't help: at η = 0.045, each step multiplies the across direction by 1 − 0.045 · 50 = −1.25 and it blows up. So plain GD has to crawl along the floor.
Momentum (β = 0.9): 73 steps. The along direction keeps agreeing, so it builds speed there. But the across direction overshoots and swings from side to side.

The narrow valley: plain GD crawls along the floor; momentum is faster but swings across.

Nesterov: look before you step

Nesterov momentum changes one thing: it takes the gradient where the velocity is about to carry the weight, not where the weight is now.

v_t = β v_(t−1) + ∇L(w_t − η β v_(t−1))

If that look-ahead point is already uphill, the gradient there points back, and the step brakes before it overshoots. On the same valley, Nesterov needs 18 steps, with almost no swing.

Nesterov looks ahead: the gradient at the look-ahead point brakes the swing. 298 · 73 · 18.

Try it in PyTorch

Every weight in a network gets its own velocity, with the same shape as the weight. In PyTorch it is one line:

opt = torch.optim.SGD(model.parameters(), lr=0.015,
                      momentum=0.9, nesterov=True)

And the velocity itself, with gradients that flip sign:

v, out = 0.0, []
for g in [1, -1, 1, -1, 1]:
    v = 0.9 * v + g
    out.append(round(v, 2))
print(out)

It prints [1.0, -0.1, 0.91, -0.18, 0.84]. Change every g to 1 and v climbs toward 10.

Key takeaways

Plain gradient descent has no memory: it stops wherever the slope is zero, even in a shallow dip.
Momentum keeps a velocity, v = βv + ∇L, and steps along it, so it can roll through flat spots and over small bumps.
v is a sum of past gradients weighted by β^k, about 1/(1 − β) = 10 steps of memory at β = 0.9.
Agreeing gradients stack (up to 10×); flipping ones cancel. That is why momentum speeds up along valleys.
Nesterov takes the gradient at the look-ahead point, so it brakes before overshooting: 298 → 73 → 18 steps in the narrow valley.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

Why train on a few examples at a time?

Syed Abdul Hanan — Sun, 01 Nov 2026 00:50:00 +0000

Stochastic and mini-batch gradient descent: why every real training loop takes small, noisy steps.

Gradient descent, as we wrote it earlier in this series, has one quiet assumption. To take a single step, it looks at every training example. That is fine for four examples. It is not fine for a million.

Real training never does it that way. It takes a small random handful of examples, a mini-batch, and steps on that alone. This article shows why that works, what it costs, and how it looks in PyTorch. The numbers come from a tiny worked example and from the CS878 Deep Learning slides (lectures 07-08).

The gradient is an average

The loss of a model is an average over the training examples: L = (1/N) Σ ℓᵢ, where ℓᵢ is the loss on example i. A derivative of an average is the average of the derivatives, so the gradient is an average too: ∇L = (1/N) Σ ∇ℓᵢ.

Take a model with two weights, ŷ = w·x, a squared loss ℓᵢ = ½(ŷᵢ − yᵢ)², and four examples:

x₁ = (4, −1), y₁ = 1
x₂ = (0, 1), y₂ = 3
x₃ = (1, 1), y₃ = 3
x₄ = (1, −1), y₄ = 1

Start at w = (0, 0). There each example's gradient is −yᵢ·xᵢ, so the four arrows are (−4, 1), (0, −3), (−3, −3) and (−1, 1).

Put them tip to tail and they add up to (−8, −4). Divide by four and you get the true gradient, ∇L = (−2, −1). A step with learning rate ¼ moves the other way, to w = (½, ¼).

Four examples, four arrows, added tip to tail and divided by four: the true gradient (−2, −1).

Now make it a million examples. The slides use n = 2²⁰, about a million images. Every single step must touch all million. That is a lot of work for one small move.

A mini-batch is a cheap, honest guess

So take a shortcut. Shuffle the data, cut it into small batches, and average the gradients of one batch only:

g_B = (1/B) Σ over i in the batch of ∇ℓᵢ

With our four examples and B = 2, say the shuffle gives batches {1, 3} and {2, 4}.

Batch one: ½[(−4, 1) + (−3, −3)] = (−7/2, −1). Too long.
Batch two: ½[(0, −3) + (−1, 1)] = (−1/2, −1). Too short.

One-step guesses from them would land at (7/8, ¼) and (1/8, ¼), on either side of the true landing point (½, ¼). But their average is ½[(−7/2, −1) + (−1/2, −1)] = (−2, −1), exactly the true gradient.

That is the key property. Over the random choice of batch, the expected value of g_B is ∇L. The estimate is noisy, but it is unbiased: it does not lean in any direction.

Batch one's gradient is too long, batch two's too short; their average lands exactly on the true gradient.

How noisy is it?

If one example's gradient has a spread of σ around the true gradient, a batch of B examples has a spread of σ/√B (when the data set is much bigger than the batch).

That square root matters. Four times the batch costs four times the compute but only halves the noise. Going from 1 to 64 examples cuts the spread by a factor of 8. Past a point, a bigger batch buys very little extra accuracy for its cost. A small batch already points the right way.

The cloud of mini-batch gradients around the true one shrinks like σ/√B, while the cost grows like B.

A race on one budget

Give three methods the same budget: one pass over 1,024 examples, called one epoch, with the same learning rate.

Full batch takes one careful step. After a whole epoch it is still far from the bottom.
Pure SGD (B = 1) takes 1,024 shaky steps. It reaches the bottom but keeps jittering around it.
Mini-batch (B = 64) takes 16 steady steps and ends near the bottom with only a small wobble.

There is a hardware reason too. A GPU computes many examples in parallel. A batch of 64 is one GPU call that keeps all its lanes busy, so the mini-batch epoch is 16 calls instead of 1,024.

If you plot the loss against iterations, as the lecture does, full batch is smooth, SGD is jagged, and mini-batch sits in between. That middle ground is why mini-batch SGD is the default.

One pass over 1,024 examples, all three at once: full batch takes 1 step, B = 64 takes 16, B = 1 takes 1,024; a B = 64 step lights all 64 GPU lanes together.

Try it in PyTorch

In PyTorch the batch size lives in the DataLoader. One batch is one iteration, one optimizer step. One full pass over the data is an epoch.

loader = DataLoader(data, batch_size=64, shuffle=True)
opt = optim.SGD(net.parameters(), lr=0.1)
for epoch in range(3):
    for xb, yb in loader:
        loss = loss_fn(net(xb), yb)
        opt.zero_grad()
        loss.backward()
        opt.step()

MNIST has 60,000 training images. In batches of 64 that is ⌈60,000 / 64⌉ = 938 iterations per epoch, and the last batch holds the 32 images left over. You can check it without any real data:

import torch.utils.data as D
import torch
X = torch.zeros(60000, 1)
dl = D.DataLoader(X, 64)
*_, last = dl
print(len(dl), len(last))

Output:

938 32

Change 64 to 1 and you get pure SGD: 60,000 steps per epoch. Change it to 60000 and you are back to full batch: one step per epoch. The optimizer is the same optim.SGD either way. The batches are what make it mini-batch SGD.

The PyTorch loop: one batch of 64 images flows through the network together, and 60,000 / 64 rounds up to 938 steps per epoch.

Noise can even help

At a flat saddle point the true gradient is almost zero, so full-batch descent barely moves. A mini-batch gradient keeps a little noise, and that is enough to shake the weights loose and let them roll downhill.

The noise does not fix everything. In a long, narrow valley the steps still zig-zag from wall to wall. Taming that zig-zag is the job of momentum, the next idea in this series.

Key takeaways

The loss is an average over examples, so its gradient is an average of per-example gradients.
A mini-batch gradient is a cheap estimate of the true gradient. It is noisy but unbiased: on average it is exactly right.
Its noise shrinks like σ/√B, so four times the batch only halves the noise at four times the cost.
In one epoch, mini-batches take many cheap steps that run in parallel on a GPU, between smooth full batch and jagged pure SGD.
In PyTorch, batch_size in the DataLoader sets it: 60,000 images in batches of 64 is 938 steps per epoch.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

Too big, it explodes. Too small, it crawls.

Syed Abdul Hanan — Sun, 01 Nov 2026 00:20:00 +0000

The learning rate, step by step: why one number decides whether training works at all.

Backpropagation tells every weight which way to move: its gradient. It doesn't say how far. That's the job of the learning rate, written η (eta). Every step of gradient descent is

w ← w − η ∇L

The gradient comes from the model. The learning rate is a setting you choose, and it is the first one people tune. This article shows what it does on the smallest possible example, why there is a hard limit on how big it can be, and how practitioners pick it and change it during training. The ideas follow the CS878 Deep Learning lecture on learning rates.

One weight on a bowl

Take a single weight and the loss L(w) = w², a bowl with its bottom at w = 0. The gradient is 2w, so one step is

w ← w − η · 2w = (1 − 2η) · w

Start at w = 2, where the loss is 4, and try four learning rates:

η = 0.01: each step multiplies w by 0.98. After ten steps, w has only reached 1.634 and the loss is still 2.67. It crawls.
η = 0.4: each step multiplies w by 0.2. Two becomes 0.4, then 0.08, then 0.016. Fast.
η = 0.9: each step multiplies w by −0.8. The weight jumps to the other side of the bowl each time (−1.6, 1.28, −1.024, 0.819), but the jumps shrink, so it still lands.
η = 1.1: each step multiplies w by −1.2. The jumps grow: −2.4, 2.88, −3.456, 4.147. The loss climbs 5.76, 8.29, 11.94, 17.20. It explodes.

One weight on the bowl L = w², four learning rates: crawls, fast, zig-zags, explodes.

Why the edge is at one

Everything depends on the factor 1 − 2η. After t steps, w_t = (1 − 2η)^t · w₀.

Below η = 0.5, the factor is between 0 and 1: the weight slides in without overshooting.
At η = 0.5, the factor is 0: one jump lands exactly on the bottom.
Between 0.5 and 1, the factor is between −1 and 0: the weight flips sides but shrinks.
At η = 1, the factor is −1: it bounces between 2 and −2 forever.
Past 1, the factor is below −1: it grows every step.

Nothing here is special to w². For any bowl L = (λ/2) w², with curvature λ (the second derivative), the factor is 1 − ηλ. So training is stable only if 0 < η < 2/λ, and one jump lands at η = 1/λ. For L = w², λ = 2: the edge is 1 and the one jump is 0.5.

The factor 1 − 2η decides everything: slides in below 0.5, one jump at 0.5, zig-zags up to 1, explodes past 1.

Millions of directions

A real network has millions of weights, so the loss has millions of directions, and they are not equally steep. Take two weights and L = ½(w₁² + 10 w₂²): one direction with curvature 1 (flat), one with curvature 10 (steep). Each step multiplies w₁ by (1 − η) and w₂ by (1 − 10η).

Starting from (4, 1):

η = 0.02: the steep direction shrinks fast (×0.8 per step) but the flat one barely moves (×0.98). After ten steps the weights are at (3.27, 0.11): done on the steep side, stuck on the flat side.
η = 0.21: 0.21 is past 2/10, the edge of the steep direction. w₂ is multiplied by −1.1 each step: 1, −1.1, 1.21, −1.331, 1.464. It explodes.
η = 2/11 ≈ 0.18: the best single fixed rate, 2/(λ_min + λ_max). Both directions shrink by 9/11 per step, and after ten steps the weights are at (0.54, 0.13).

The steep direction sets the speed limit, η < 2/λ_max, and the flat direction then crawls. Even the best fixed rate shrinks the error by only (κ − 1)/(κ + 1) = 9/11 per step, where κ = λ_max/λ_min = 10 is how uneven the valley is.

The valley L = ½(w₁² + 10w₂²): the steep side sets the speed limit; the best fixed step is 2/11.

Picking a starting rate

Two quick checks get you most of the way:

Before training, look at the first loss. For softmax with C classes and small random weights, it should be about ln C, which is 2.30 for ten classes. If it isn't, something else is wrong.
Try a few learning rates a factor of ten apart: 1e-1, 1e-2, 1e-3, 1e-4. Run each for about a hundred iterations and keep the one whose loss drops fast without blowing up. Too small barely moves; too big turns into NaN.

Trying 1e-4 to 1e-1 from the same starting loss of 2.30: keep the one that drops fast.

Try it in PyTorch

The bowl example, at the learning rate that explodes:

w, lr = 2.0, 1.1
for t in range(4):
    w = w - lr * 2 * w
    print(round(w, 3))

It prints −2.4, 2.88, −3.456, 4.147. Change lr to 0.5 and w lands on 0 in one step.

In a real training loop the learning rate is one argument of the optimizer. A quick sweep tries a few values a factor of ten apart:

import torch
# model = your network
for lr in (1e-1, 1e-2, 1e-3, 1e-4):
    opt = torch.optim.SGD(model.parameters(), lr=lr)
    ...            # train ~100 iterations, keep the lr whose loss drops fast

Shrinking the rate during training (decay schedules) is its own episode.

Key takeaways

The learning rate sets how far each gradient step goes; it is chosen, not learned.
On a bowl with curvature λ, each step multiplies the error by 1 − ηλ: stable only if η < 2/λ, one jump at η = 1/λ.
In a real network the steepest direction sets the speed limit, so the flat directions crawl.
Start by checking the first loss (ln C), then try rates a factor of ten apart and keep the one that drops fast.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

How does PyTorch find every gradient by itself?

Syed Abdul Hanan — Sat, 31 Oct 2026 23:50:00 +0000

Autograd, step by step: PyTorch records the forward pass as a graph, then replays it backward.

In the last article we did backpropagation by hand. We sent the error back through a tiny network, one node at a time, and found each weight's gradient. It worked, but nobody does that for a real network with millions of weights.

In PyTorch you write the forward pass, call L.backward(), and every gradient is there. This article shows what happens in between. The engine that does it is called autograd, short for automatic differentiation.

We'll use the same network and the same numbers as last time, from the CS878 Deep Learning slides.

The forward pass builds the graph

Here is the network in PyTorch:

x  = torch.tensor(1.0)
w1 = torch.tensor(1/3, requires_grad=True)
w2 = torch.tensor(2.0, requires_grad=True)
y  = torch.tensor(0.0)
z  = x * w1
s  = z.relu()
yh = s * w2
L  = (yh - y)**2

The first four lines make the leaves of the graph: the input x, the target y and the two weights. Only the weights have requires_grad=True. That flag tells autograd we will want their gradients.

Every line after that runs one operation, and autograd records it. x * w1 makes a multiply node that holds 1/3. z.relu() adds a ReLU node, still 1/3. Multiplying by w2 gives 2/3, and the squared error gives the loss, 4/9.

You can see the record in what each result prints. z prints as tensor(0.3333, grad_fn=<MulBackward0>). The value is the number we expected. The grad_fn part is new: it is the function that made z.

Typing the forward pass builds the graph, one node per line: the loss is 4/9.

grad_fn links make the backward graph

Every result keeps its grad_fn, and every grad_fn keeps links to the functions that made its inputs. Follow those links from the loss and you have the whole graph, pointing backward.

One line of code can hold more than one operation. L = (yh - y)**2 is a subtraction and then a power, so PyTorch records two nodes: L.grad_fn is PowBackward0, and its input comes from SubBackward0.

The weights asked for gradients, so each gets an extra node at the very end of the backward graph, called AccumulateGrad. Think of it as a bucket: it is where the weight's .grad will be stored. The input x and the target y did not ask for gradients, so autograd skips them.

The backward graph: every operation is now its backward function, the arrows point down, and each weight has an AccumulateGrad bucket.

L.backward() walks the graph in reverse

Calling L.backward() starts with a gradient of 1 at the loss and visits the nodes in reverse order. Each node knows one small rule: take the gradient from above and multiply it by its own local derivative.

Pow: the local derivative of d² is 2d, so d gets 1 × 2d = 4/3.
Sub: the local derivative of ŷ − y with respect to ŷ is 1, so ŷ gets 4/3. The path to y is skipped.
Mul: a multiply node swaps values. w2 gets 4/3 × s = 4/9, and s gets 4/3 × w2 = 8/3.
ReLU: z was positive, so the gate is open and passes 8/3 through.
Mul: w1 gets 8/3 × x = 8/3. The path to x is skipped.

The buckets fill: w1.grad is tensor(2.6667), or 8/3, and w2.grad is tensor(0.4444), or 4/9. Those are the numbers we found by hand.

After L.backward(): the buckets hold 8/3 and 4/9, the same numbers as the hand-worked backprop.

When a value is used twice, gradients add

Take y = w * w with w = 3. The multiply node has two inputs, and both are w. On the way back, each path sends 3 (the other input's value). The two contributions meet in w's bucket and add up to 6, which is the derivative of w², 2w. This is the multivariable chain rule: when a value feeds several paths, its gradient is the sum over the paths.

The bucket keeps adding across calls too. Run the forward and backward pass again and w.grad goes from 6 to 12. That's why every training loop calls optimizer.zero_grad() (or sets the gradients to None) before each backward pass.

Running backward twice: the bucket jumps from 6 to 12, so training loops call zero_grad.

Why go backward?

A network has one loss and millions of weights. You could also push derivatives forward, from one weight up to the loss. That's forward-mode differentiation, and it needs one pass per weight: a million weights, a million passes.

Reverse mode, the one autograd uses, starts from the single loss and reaches every weight in one sweep. It costs about as much as a couple of forward passes, whatever the number of weights. With one output and many inputs, going backward is the only sensible direction.

Reverse mode: one wavefront sweeps down from the loss through every layer and reaches all 36 weights in a single pass.

The graph is rebuilt every step

After backward(), PyTorch frees the graph. The next forward pass records a new one. Because the graph is built while your code runs, ordinary Python works inside a model: an if can add a node on one step and leave it out on the next, and a loop can run a different number of times.

0 the raw `if` endraw adds a ×2 node." width="503" height="739">

The same code records two different graphs: with s > 0 the if adds a ×2 node.

Try it in PyTorch

import torch
w = torch.tensor(3.,
  requires_grad=True)
(w * w).backward()
(w * w).backward()
print(w.grad)

It prints tensor(12.), not 6, because the second backward pass adds to the first. Add w.grad = None between the two backward calls and it prints tensor(6.).

Key takeaways

Autograd records every operation of the forward pass as a graph. Each result's grad_fn is the function that made it.
L.backward() walks that graph in reverse. Each node multiplies the gradient from above by its local derivative.
Weights with requires_grad=True collect their gradient in .grad through AccumulateGrad. Tensors without the flag are skipped.
Gradients from several paths add up, and .grad keeps adding across steps, so training loops call zero_grad().
Reverse mode gets every gradient in one sweep, and the graph is rebuilt on every forward pass, so plain Python control flow works.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

Who's to blame when a network gets it wrong?

Syed Abdul Hanan — Sat, 31 Oct 2026 23:20:00 +0000

Backpropagation, step by step: how one backward pass hands every weight its share of the blame.

When a neural network makes a wrong prediction, training has to fix it. But a network has many weights, and each one played some part in the mistake. Before we can change anything, we need to know how much each weight is to blame.

That's what backpropagation does. After the network makes its prediction (the forward pass), it sends the error back through the network (the backward pass). On the way, every weight picks up a number, its gradient, that says how much the loss would change if that weight moved a little. That number is the weight's share of the blame.

The examples below are the ones from the CS878 Deep Learning lecture slides, with the same numbers.

A tiny graph, forward

Start with something small enough to see all at once:

f(x, y, z) = (x + y) · z

Draw it as a graph with two steps. An add node computes d = x + y, and a multiply node computes f = d · z.

Put in x = 1, y = −3 and z = 4. The add node gives d = 1 + (−3) = −2. The multiply node gives f = −2 · 4 = −8. That's the forward pass: values flow from the inputs to the output.

The forward pass. The add node turns 1 and −3 into −2, and the multiply node turns −2 and 4 into −8.

The same graph, backward

Now go the other way. We want to know how f changes when each input changes: ∂f/∂x, ∂f/∂y and ∂f/∂z.

Start at the output with a gradient of 1, because f changes exactly as much as f does.

A multiply node swaps: each input gets the upstream gradient times the other input's value. For f = d · z, the local gradients are ∂f/∂z = d and ∂f/∂d = z. So z gets 1 × (−2) = −2 (the value of d) and the sum gets 1 × 4 = 4 (the value of z).

An add node copies: it passes the gradient straight through to every input. For d = x + y, both local gradients are 1, so x and y each get 4 × 1 = 4.

Check one of them with the chain rule: ∂f/∂x = ∂f/∂d · ∂d/∂x = 4 · 1 = 4.

The backward pass. Starting from 1 at the output, the multiply node sends −2 to z and 4 to the sum; the add node passes 4 to both x and y.

Every node does the same job

Look at what each node did. It took the gradient arriving from above, the upstream gradient, and multiplied it by its own local gradient. For any node, with in one of its inputs and out its output:

∂L/∂in = ∂L/∂out · ∂out/∂in (upstream times local)

That's the whole algorithm. Each node only needs to know its own small derivative. It never has to see the rest of the network. The chain rule from the last episode, applied node by node, from the output back to the inputs.

A real network

Now a tiny network: one input x = 1, a first weight w₁ = 1/3, a ReLU, a second weight w₂ = 2, and a squared-error loss against a target of y = 0.

Forward:

z = x · w₁ = 1/3
s = max(0, z) = 1/3 (the ReLU's output; σ usually means the sigmoid)
ŷ = s · w₂ = 2/3
L = (ŷ − y)² = 4/9

The network predicts 2/3 when the answer is 0. That's the mistake, and the loss of 4/9 measures it.

The network after its forward pass. It predicts 2/3 instead of 0, a loss of 4/9.

Handing out the blame

Backward, from the loss, one node at a time:

At the output: ∂L/∂ŷ = 2(ŷ − y) = 4/3.
The multiply node ŷ = s · w₂ swaps. The second weight gets 4/3 × s = 4/3 × 1/3 = 4/9, and the other branch gets 4/3 × w₂ = 4/3 × 2 = 8/3.
The ReLU's input was positive (z = 1/3), so the gate is open and its local gradient is 1: z gets 8/3 × 1 = 8/3. Had z been negative, the gate would be shut and pass back nothing.
The multiply node z = x · w₁ swaps too. The first weight gets 8/3 × x = 8/3 × 1 = 8/3.

Multiply the factors along the path and you get the slides' chain rule, ∂L/∂w₁ = ∂L/∂ŷ · ∂ŷ/∂s · ∂s/∂z · ∂z/∂w₁ = 4/3 · 2 · 1 · 1.

So the first weight gets six times as much of the blame as the second: 8/3 = 6 × 4/9. One backward pass found both numbers.

The backward pass through the network. The first weight's gradient is 8/3, the second weight's is 4/9.

One step downhill

Gradient descent then moves every weight at once, each against its own gradient, scaled by a learning rate α:

w ← w − α · ∂L/∂w

With α = 0.05, the first weight moves six times as far as the second: w₁ goes from 1/3 to 1/3 − 0.05 · 8/3 = 0.20, and w₂ from 2 to 2 − 0.05 · 4/9 ≈ 1.98. Run the forward pass again with the new weights: z = 0.20, s = 0.20, ŷ ≈ 0.40, and the loss drops from 0.44 to about 0.16.

One gradient step with α = 0.05. The weights become 0.20 and 1.98, and the loss falls from 0.44 to 0.16.

Why it matters

A real network has millions of weights. Nudging them one at a time would take millions of forward passes per step. Backpropagation gets all of them in one backward pass that costs about as much as the forward pass. It does this by reusing work: the 4/3 at the output was computed once and shared by both weights.

It became the standard way to train neural networks after Rumelhart, Hinton and Williams showed in 1986 that it lets hidden layers learn useful features.

Try it in PyTorch

PyTorch runs backpropagation for you. Build the loss, call backward(), and every weight's gradient appears in .grad:

import torch
w = torch.tensor([1/3, 2.],
  requires_grad=True)
L = (w[1] * w[0].relu())**2
L.backward()
print(w.grad)

It prints tensor([2.6667, 0.4444]): 8/3 and 4/9, the same numbers we found by hand. (Here x = 1 and y = 0, so they drop out of the formula.) Change 1/3 to −1/3 and both gradients become zero: the ReLU outputs 0, so no blame gets through.

Key takeaways

The forward pass computes values; the backward pass sends gradients from the loss back to every weight.
Each node multiplies the gradient from above by its own local gradient.
A multiply node swaps: each input gets the gradient times the other's value. An add node copies: it passes the gradient straight through. An open ReLU multiplies by 1.
In the slides' network, one backward pass gives the first weight 8/3 and the second 4/9, and one step with α = 0.05 takes the loss from 0.44 to 0.16.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

How does one nudge ripple through ten layers?

Syed Abdul Hanan — Sat, 31 Oct 2026 22:50:00 +0000

The chain rule, the idea that lets a network see through every one of its layers, with real numbers.

A neural network is a chain. The input goes into the first layer, the first layer's output goes into the second, and so on, until the last layer produces an answer and a loss, one number that says how wrong that answer is.

Training keeps asking one question: if I nudge this weight a little, how much does the loss move? For a weight ten layers down, the nudge has to travel through every layer above it before it reaches the loss. The chain rule tells you how big it is when it arrives.

A network is a chain

Written as math, a ten-layer network is a function inside a function inside a function:

f₁₀( ⋯ f₂( f₁(x) ) ⋯ )

Picture it as a tower of dials on one shaft: the input at the bottom, a dial for each layer above it. Turn the bottom dial a little and every dial above it turns too, but not by the same amount. Each layer has its own ratio, which can make the turn bigger or smaller.

A network as a tower of dials. Turn the input at the bottom and every layer above it turns; the blue wedges show how far each one moved.

Two links, real numbers

Take just two layers. The first triples its input, y = 3x, and the second squares that, z = y². Start the input at x = 1: then y = 3 and z = 9.

Now raise the input by 0.01, to x = 1.01. Then y = 3.03: it moved 0.03, three times the nudge, so its ratio is 3.

And z = 3.03² = 9.1809: it moved about 0.18, six times y's own move of 0.03, so its ratio is 6.

Why six? Write out (3 + h)² = 9 + 6h + h². For a tiny h, the h² hardly matters, so the change is about 6h. That ratio is the slope of y² at y = 3: its derivative.

Close on two links. Raising the input by 0.01 moves the first layer by 0.03 (ratio 3) and the second by 0.18 (ratio 6).

The ratios multiply

The first link multiplies the nudge by 3 and the second by 6, so together it's 3 × 6 = 18. The real numbers agree: z moved 0.18 for a nudge of 0.01, which is 18 times.

That's the chain rule. Written with derivatives:

dz/dx = dz/dy · dy/dx = 6 × 3 = 18

Each layer only needs to know its own slope: find each link's ratio where it stands, then multiply.

You've seen this on a clock

A clock is a chain too. The minute hand turns 12 times as fast as the hour hand, and the second hand turns 60 times as fast as the minute hand. So the second hand turns 12 × 60 = 720 times as fast as the hour hand: you multiply the ratios along the chain, exactly as with the layers.

Ten layers, ten ratios

A network with ten layers multiplies ten ratios, and a product of many numbers moves fast in both directions.

If every layer halves the turn, the top moves 0.5¹⁰ ≈ 0.001 as far as the input. The nudge has all but vanished.
If every layer doubles it, the top moves 2¹⁰ = 1024 times as far. The nudge has exploded.

Ten layers that each double the turn. The bottom barely moves, and the top spins 1024 times as far.

This is why very deep networks were once so hard to train: the signal about how a deep weight affects the loss either faded away or grew out of control.

The same rule in a real network

Here is a small three-layer network: two inputs, three hidden neurons with ReLU, and one output, scored by squared error against a target of 1. With inputs 2 and 1, the hidden neurons come out at 1, 0.5 and 0.2, the output is 3, and the loss is (3 − 1)² = 4.

Pick one weight, w = 0.5, from the first input to the top hidden neuron, and follow its path to the loss. It has three links:

the hidden neuron moves 2 times as much as w (the input it multiplies is 2);
the output moves 3 times as much as that hidden neuron (its weight is 3);
the loss moves 4 times as much as the output (the slope of (y − 1)² at y = 3).

So ∂L/∂w = 2 × 3 × 4 = 24. Check it: raise w from 0.50 to 0.51, and the loss goes from 4.00 to 4.24.

A three-layer network. One weight's path to the loss has ratios 2, 3 and 4, so the loss moves 24 times as much as the weight.

Try it in PyTorch

PyTorch applies the chain rule for you. torch.func.grad returns a function's derivative, even when the function is a chain of other functions:

from torch import tensor, func
y = lambda x: 3 * x
z = lambda y: y ** 2
f = lambda x: z(y(x))
x = tensor(1.)
print(func.grad(f)(x))

It prints tensor(18.), the same 3 × 6 we found by hand. Add a third layer that halves its input, and the gradient of the longer chain should come out at 9.

Key takeaways

A network is a chain of layers, and a nudge at the bottom ripples through every layer above it.
Each layer scales the nudge by its own ratio: its local slope, the derivative.
The chain rule: multiply the ratios along the chain. dz/dx = dz/dy · dy/dx.
With many layers the product can shrink to almost nothing (0.5¹⁰ ≈ 0.001) or blow up (2¹⁰ = 1024).
In a real network, a weight's effect on the loss is the product of the ratios along its path: 2 × 3 × 4 = 24.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

How do you find a valley blindfolded?

Syed Abdul Hanan — Sat, 31 Oct 2026 22:20:00 +0000

Gradient descent, the walk every neural network takes while it learns, step by step with real numbers.

Imagine standing in the hills with a blindfold on, trying to reach the bottom of the valley. You can't see the valley or the path. All you have is what you feel under your feet: which way the ground tilts, and how steeply.

That's enough. If the ground tilts down to your left, step left, then feel again. Repeat until the ground is flat. That loop is gradient descent, and it's how almost every neural network is trained.

The ground is the loss

A network in training is in the same position. Its loss, one number that says how wrong its answers are, depends on its weights: change a weight and the loss goes up or down.

Picture a network with just two weights. For every pair of weights you can compute the loss, so you can draw it as a height over a flat map of the two weights. That gives a landscape. Where you stand on the map is the current weights. The height under your feet is the current loss. Training means finding the lowest point.

In our example the landscape is a long valley:

loss = (w₁ − 3)² + 3 · (w₂ − 1)²

The bottom is at w₁ = 3 and w₂ = 1, where the loss is 0. We start at w₁ = 0 and w₂ = 3, where the loss is 21.

The valley the network can't see. The ball is the current weights; the blue patch is the only ground it can feel.

The network never sees this picture. A real network has millions of weights, so its landscape can't be drawn at all. Like the hiker, it only knows the ground where it stands.

The slope is the gradient

The thing you feel under your feet has a name: the gradient. It's the slope of the loss in every weight's direction at once, and it points uphill, the way the loss grows fastest.

At our starting point the gradient is (−6, 12). The 12 says the loss climbs steeply if weight 2 grows. The −6 says it also climbs if weight 1 shrinks. Going uphill would be a bad idea, so gradient descent does the opposite and steps against the gradient, downhill.

The gradient points uphill (grey arrow). The step goes the other way, downhill (blue).

How big is a step?

Each step is the gradient times a small number called the learning rate, written η (eta):

w ← w − η · ∇L

With a learning rate of 0.1, the first step moves the weights by 0.1 × (−6, 12) = (−0.6, 1.2) the other way. They go from (0, 3) to (0.6, 1.8), and the loss drops from 21 to 7.7.

Then you feel the slope again and take the next step. Here is what twenty steps do:

after 1 step, the loss is 7.7;
after 2 steps, 4.0;
after 5 steps, 0.97;
after 10 steps, 0.10;
after 20 steps, 0.001, almost zero.

Notice that the steps get shorter by themselves. Near the bottom the ground flattens, so the gradient shrinks, and a small gradient times the learning rate makes a small step. Nobody tells the network to slow down near the minimum. The slope does it.

Twenty real steps at learning rate 0.1. The loss goes from 21 to almost zero, and the steps shrink as the ground flattens.

The learning rate is your stride

The gradient gives the direction, and the learning rate decides how far you go each time. To see why that matters, look along the valley floor from the side, where it's a simple curve, loss = (w − 3)², starting at w = 0 with a loss of 9.

Learning rate 0.1: the hops shrink as the curve flattens, and the ball settles at the bottom.
Learning rate 0.01: tiny steps. After twenty of them the loss is still 4.0. You're walking the right way, just very slowly.
Learning rate 1.1: giant leaps. Every step overshoots the bottom and lands higher on the other side. Four steps later the loss has grown from 9 to 38.7, and it keeps growing.

The valley floor from the side. At 1.1 every step leaps over the bottom and climbs (grey arcs); at 0.1 the hops shrink and the ball settles at the bottom (blue).

There's a simple reason for the blow-up. On this curve, each step multiplies your distance from the bottom by (1 − 2η). With η = 0.1 that's 0.8, so you get 20% closer every step. With η = 1.1 it's −1.2: you land on the other side, 20% farther away than before.

That's why the learning rate is usually the first setting people tune. A common recipe from the lecture notes: try 0.1, 0.01, 0.001 and 0.0001, and keep the one that makes the loss drop fast within the first hundred steps.

Try it in PyTorch

PyTorch can compute the gradient for you. torch.func.grad turns a function into its slope:

from torch import tensor, func
L = lambda w: (w - 3) ** 2
w = tensor(0.)
for _ in range(20):
  w -= 0.1 * func.grad(L)(w)
print(w)

It prints tensor(2.9654): twenty steps from 0 get within 0.035 of the bottom at 3. Change 0.1 to 1.1 and run it again to watch the steps leap out of the valley.

Key takeaways

Training is a walk down the loss landscape, where your position is the weights and your height is the loss.
The gradient is the slope under your feet. It points uphill, so each step goes against it: w ← w − η · ∇L.
Steps shrink by themselves near the bottom, because the slope flattens.
The learning rate is your stride. Too small and you crawl, too big and you leap out of the valley.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

This is what a confident mistake looks like

Syed Abdul Hanan — Sat, 31 Oct 2026 21:50:00 +0000

Cross-entropy, the loss behind almost every classifier, as a landscape and as a bet.

A classifier doesn't just pick an answer. It gives every class a score, and softmax turns those scores into probabilities: cat, dog, bird. To learn, the model needs one number that says how good those probabilities were. For classifiers, that number is almost always the cross-entropy loss.

One number, one log

Cross-entropy looks at a single probability: the one the model gave to the right answer. Call it p. The loss is minus its log:

loss = −log(p)

If the picture is a cat and the model said 67% cat, the loss is −log(0.67) ≈ 0.4. If it said 1% cat, the loss is −log(0.01) ≈ 4.6.

The loss as a landscape

Take a classifier with three classes, and hold the bird score still. Two numbers are left: the cat score and the dog score. For every pair, cross-entropy gives a loss. Make that loss the height, and you get a landscape.

The loss over the cat and dog scores: a landscape. The ball is a confidently wrong model: cat 1%, loss 4.61.

Where the cat score wins, the ground is flat and close to zero. A model that's confident and correct sits there at almost no cost: at 98% cat, the loss is 0.02.

Where the cat score wins, the ground is flat and close to zero.

Where the dog score wins, the ground climbs, and it never levels off. Far enough up that ramp the loss grows by one for every extra point of dog score. There is no flat spot for a confident mistake.

Where the dog score wins, the ground keeps climbing.

Training rolls downhill

Training follows the slope. For softmax with cross-entropy, the slope with respect to each score is just the prediction minus the truth, where the truth is 1 for the right class and 0 for the others. So each training step moves the scores by that amount, times a learning rate:

z ← z − (p − y)

Start at the confident mistake, cat 1% and dog 99%, with a learning rate of 1. Three steps take the cat from 1% to 7%, to 32%, to 64%, and the loss falls from 4.61 to 0.44. The ramp is steep exactly where the model is most wrong, so the push back is strongest there.

Three real gradient-descent steps from the top of the ramp: cat 1% → 64%.

The same loss as a bet

Here's another way to see it. Think of the model's probabilities as a bet: 100 chips spread across the answers, 67 on cat, 24 on dog and 9 on bird. When the answer turns out to be a cat, cross-entropy sends a bill: minus the log of the cat's share, −log(67/100) ≈ 0.4.

The model bets 100 chips. It's a cat, so the bill is −log(0.67) ≈ 0.4.

Halve the chips on the truth, and the bill grows by the same amount every time, log(2) ≈ 0.69:

67 chips cost 0.40;
34 chips cost 1.09;
17 chips cost 1.79;
one chip costs 4.61;
zero chips: there is no limit.

Every halving adds 0.69. One chip costs 4.6; at zero, the bill has no limit.

That's why a confident mistake costs so much: being sure of the wrong answer means betting almost nothing on the truth, and every halving of that bet keeps adding to the bill.

Try it in PyTorch

You never write the log yourself. nn.CrossEntropyLoss takes the raw scores and the right class and does the softmax and the log for you:

import torch
from torch import nn
z = torch.tensor([[2., 1, 0]])
y = torch.tensor([0])  # cat
print(nn.CrossEntropyLoss()(z, y))

It prints tensor(0.4076), the 0.4 from above. Pass it probabilities instead of raw scores and it applies softmax a second time, and the loss comes out wrong.

Key takeaways

Cross-entropy = −log(probability of the right answer).
Over the scores it is a landscape: flat where the model is right, a ramp with no top where it is confidently wrong.
Its slope is prediction minus truth, so training pushes hardest where the model is most wrong.
As a bet: every halving of the chips on the truth adds 0.69, and zero chips has no limit.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

How does a network know it's wrong?

Syed Abdul Hanan — Sat, 31 Oct 2026 21:20:00 +0000

Loss functions turn every mistake into one number. Here's how squared and absolute loss score the same misses very differently.

A neural network can't feel embarrassed when it's wrong. It needs a way to measure its mistakes, so it knows which direction to improve.

That measurement is the loss function. It turns all of a network's errors into a single number, and training is the process of pushing that number down.

One number for "how wrong"

A loss function compares the network's predictions with the right answers and returns one number:

Zero means every prediction is perfect.
Bigger means worse.

That's it. Everything a network learns, it learns by trying to make this number smaller.

A simple example

Say a network draws a line to predict three data points. It doesn't fit perfectly. Its predictions miss the three points by 3, 2, and 1.

How bad is that, in one number? It depends on which loss function you pick.

The network's line misses three points by 3, 2 and 1.

Squared loss

The squared loss squares each miss, then adds them up:

3² = 9
2² = 4
1² = 1

The total is 9 + 4 + 1 = 14. When you average it over all the data points, this is called the mean squared error, or MSE. Here, that's 14 ÷ 3 ≈ 4.7.

Squaring makes every miss positive, and it punishes big misses much more than small ones. A miss of 3 costs nine times as much as a miss of 1.

Squared loss: the misses 3, 2 and 1 cost 9, 4 and 1, for a total of 14.

Absolute loss

The absolute loss is simpler. It just adds up the size of each miss, ignoring whether the prediction was too high or too low:

3 + 2 + 1 = 6.

Averaged over the data points, this is called the mean absolute error, or MAE. Here, that's 6 ÷ 3 = 2.

What one outlier does

Now a new point arrives, and the network misses it by 5.

Squared, that one miss costs 5² = 25. That's more than all the other misses combined (14).
The absolute loss only adds 5.

The totals jump from 14 to 39 for squared loss, but only from 6 to 11 for absolute loss.

So one unusual point can dominate the squared loss. If your data has a few wild outliers, a network trained with squared loss will bend over backwards to reduce them, sometimes at the expense of fitting everything else.

One new point misses by 5. Squared, it costs 25, more than the other three misses combined. Absolute only adds 5.

How training uses the loss

Training adjusts the network's weights to push the loss down, one small step at a time. Each step asks: "If I nudge this weight a little, does the loss go up or down?" Then it moves the weight in the direction that lowers the loss.

Which loss you choose shapes what the network cares about. Squared loss cares a lot about big mistakes. Absolute loss treats every unit of error the same.

Training nudges the weights to push the loss down, one small step at a time.

Quick check

Your data has a few wild outliers. Which loss stays calmer: squared, or absolute?

Answer: absolute. It grows in step with each miss, while squaring blows big misses up.

Key takeaways

A loss function turns all of a network's errors into one number. Zero is perfect.
Squared loss squares each miss (3, 2, 1 → 9 + 4 + 1 = 14). Averaged, it's the mean squared error.
Absolute loss adds the miss sizes (3 + 2 + 1 = 6). Averaged, it's the mean absolute error.
Squared loss is very sensitive to outliers: a single miss of 5 costs 25.
Training adjusts the weights, step by step, to push the loss down.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

How do raw scores become probabilities?

Syed Abdul Hanan — Sat, 31 Oct 2026 20:50:00 +0000

Softmax, the last step of almost every classifier, explained with three numbers.

When a neural network looks at a photo and decides "cat," it doesn't jump straight to an answer. Its last layer produces a raw score for every possible class. Those scores are useful, but they aren't probabilities yet.

Softmax is the small function that turns them into probabilities. It takes two steps, and you can do both by hand.

Raw scores are not probabilities

Say a network has three classes: cat, dog, and bird. For one image, its last layer gives these raw scores (often called logits):

Cat: 2
Dog: 1
Bird: 0

Cat clearly scores highest. But these numbers have two problems if you want to read them as probabilities. They can be negative, and they don't add up to 1. A score of 2 doesn't tell you "how sure" the network is.

The network's last layer gives one raw score per class: cat 2, dog 1, bird 0.

Step 1: take the exponential

First, softmax raises the number e (about 2.718) to the power of each score. This is called taking the exponential.

Cat: e² ≈ 7.4
Dog: e¹ ≈ 2.7
Bird: e⁰ = 1

The exponential has a useful property: its result is always positive, even when the score is negative. A score of −3 becomes e⁻³ ≈ 0.05, small but still positive. It also stretches the gaps, so higher scores pull further ahead.

Step 1: the exponential turns 2, 1 and 0 into about 7.4, 2.7 and 1. Every result is positive.

Step 2: divide by the total

Next, add the three results: 7.4 + 2.7 + 1 ≈ 11.1. Then divide each one by that total.

Cat: 7.4 ÷ 11.1 ≈ 67%
Dog: 2.7 ÷ 11.1 ≈ 24%
Bird: 1 ÷ 11.1 ≈ 9%

Together, they add up to 100%. Now the numbers are real probabilities: all positive, and they sum to one.

As a formula, for a class with score zᵢ: softmax(zᵢ) = e^(zᵢ) ÷ (the sum of e^(z) over all classes).

Step 2: divide each exponential by the total. Cat gets 67%, dog 24%, and bird 9%, adding up to 100%.

Why "soft" max?

A hard max would simply pick the winner and give it everything: cat 100%, dog 0%, bird 0%. That throws away information about how close the race was.

Softmax is the soft version. The winner gets the biggest share, but the others keep a share too. That matters during training, because the network can learn from how confident it was, not just whether it was right.

A hard max would give cat everything and the others nothing. Softmax keeps a share for every class.

Quick check

Add 10 to every score: cat 12, dog 11, bird 10. Do the probabilities stay the same, or does cat's share grow?

Answer: they stay the same. Adding 10 multiplies every exponential by the same amount, e¹⁰. That common factor cancels when you divide by the total. Only the differences between scores matter.

Key takeaways

A classifier's last layer outputs raw scores, which can be negative and don't sum to one.
Softmax takes the exponential of each score, which makes every value positive.
Then it divides by the total, so the results add up to 100%.
Scores of 2, 1 and 0 become about 67%, 24% and 9%.
Shifting every score by the same amount doesn't change the result. Only the gaps matter.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

Without one bend, 100 layers act like one

Syed Abdul Hanan — Sat, 31 Oct 2026 20:20:00 +0000

Why every neural network needs an activation function, explained with two simple equations.

Deep networks stack layer after layer. But there's a catch hiding in that stack. Without one small ingredient, a hundred layers can do exactly what one layer does, and nothing more.

That ingredient is the activation function. Here's why it matters, in a few lines of algebra.

What a layer does

Every layer in a neural network does two things:

It multiplies and adds: each input is multiplied by a weight, the results are summed, and a bias is added.
It passes the result through an activation function: a small bend.

The first step is simple arithmetic. The second step is the one that's easy to overlook, and it's the one that makes depth worth having.

What happens without the bend

Let's remove the activation function and watch what happens with two tiny layers.

Layer one doubles its input and adds one: h = 2x + 1.
Layer two triples that and subtracts four: y = 3h − 4.

Now put them together. Substitute the first into the second: y = 3(2x + 1) − 4 = 6x − 1.

Two layers, and the result is a single multiply-and-add. It's still just one straight line.

Two layers without a bend: 2x + 1, then 3h − 4. Combined, they collapse into a single line, 6x − 1.

Stacking doesn't help

This isn't a coincidence of the numbers we picked. Any multiply-and-add fed into another multiply-and-add is still a multiply-and-add.

So stack a hundred of these layers, and they still collapse into one. All that depth, and the network can only draw a straight line.

Add the bend: ReLU

Now add an activation function. The most popular one is called ReLU (rectified linear unit). Its rule is:

Negative numbers become zero.
Positive numbers pass straight through.

So −3 becomes 0, and 2 stays 2. That's it. It looks almost too simple to matter.

ReLU: negative inputs become zero, positive inputs pass straight through.

From bends to curves

That one kink at zero is enough to stop the collapse. The layers can't be merged into a single line anymore.

Better still, bends can be combined. Add two ReLUs, one of them flipped, and you get a peak. Combine many of them, with different positions and slopes, and they trace a curve, one straight piece at a time.

This is how networks build complicated functions out of very simple parts.

Many ReLU pieces, each a straight segment with one bend, combine to trace a curve.

Sigmoid vs ReLU

Older networks often used a smooth S-shaped curve called sigmoid, which squashes any number into the range between 0 and 1.

Today, ReLU is the standard choice in most networks, because it's simple and fast to compute. It also tends to make deep networks easier to train.

Sigmoid, a smooth S-curve, was common in older networks. ReLU is the standard choice today.

Quick check

A network has ten layers, with no activation functions. Can it draw a curve, or only a straight line?

Answer: only a straight line. Without a bend, all ten layers collapse into one multiply-and-add.

Key takeaways

Every layer multiplies, adds, and then applies an activation function.
Without the activation, any stack of layers collapses into a single straight line, like 2x + 1 and 3h − 4 becoming 6x − 1.
ReLU turns negatives into zero and passes positives through. That one bend stops the collapse.
Many bends combined can trace curves.
Sigmoid was common in older networks. ReLU is the default today.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

How do hidden layers bend a straight line?

Syed Abdul Hanan — Sat, 31 Oct 2026 19:50:00 +0000

They don't. They join many straight lines into shapes, and that's where a network's power comes from.

A single neuron can only draw one straight line. Yet neural networks recognize faces, read handwriting, and trace curved boundaries everywhere. So where do the curves come from?

The surprising answer: from straight lines, lots of them, combined by a hidden layer.

One line can't fence in a cluster

A neuron splits the plane in two with a straight line. Points on one side get a "yes," and points on the other side get a "no."

That works when the classes sit on opposite sides of the page. But look at this case: the blue points form a cluster in the middle, surrounded by the others. Wherever you draw one straight line, it can't fence the cluster in.

The blue points sit in the middle. A single straight line can't separate them from the points around them.

Add a hidden layer

So add a layer of neurons between the inputs and the output. This is a hidden layer. Give it three neurons.

Each hidden neuron draws its own line, and each one asks a single question about any point: "Is this point on my side?" The answer is yes (1) or no (0).

The output neuron asks for agreement

The output neuron adds up the three answers. It only fires when the sum passes 2.5, so it needs all three to say yes.

The only place where all three lines agree is the region between them. Together, the three lines fence off a triangle.

Three hidden neurons draw three lines. The output fires only where all three agree, which is a triangle.

A worked example

Take a point inside the triangle. All three neurons say yes, so the sum is 1 + 1 + 1 = 3. That passes 2.5, so the output fires.

Now take a point just past one of the lines. Only two neurons say yes, so the sum is 0 + 1 + 1 = 2. That's below 2.5, so the output stays off.

That threshold turns the output neuron into an "AND" of all the hidden neurons.

More neurons, rounder shapes

The triangle is a rough fit, and its corners catch a few outsiders. The fix is simple: use more hidden neurons.

With six neurons, you get six lines and a tighter hexagon.
With twenty, the edge looks almost like a circle.

The edge is never truly curved. It's made of many short straight pieces, like a polygon with lots of sides. Add enough of them, and the difference stops mattering.

More hidden neurons means more straight edges. With twenty, the boundary looks almost like a circle.

Go deeper

A second hidden layer can combine the shapes from the first one. That lets a network fence off things a single shape can't, like two separate islands, or a ring with a hole in the middle.

This is why depth matters. Each layer builds on the shapes from the layer before it.

A second hidden layer combines shapes: two islands, or a ring with a hole.

Quick check

A hidden layer has five neurons. Does the output fence off a pentagon, or a perfect circle?

Answer: a pentagon. Five neurons draw five straight lines, so the edge has (at most) five sides. A perfect circle would need infinitely many.

Key takeaways

One neuron draws one straight line.
A hidden layer draws many lines, and the output neuron fires only where they agree.
Three lines make a triangle, six a hexagon, and twenty something close to a circle.
More layers combine shapes into islands, rings, and other complex regions.
Hidden layers never bend a line. They join many straight ones into a curve.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.