DEV Community: Wesam Khallaf — Author of PyTorch From Ground Up

Backpropagation by Hand: Two Layers, a Pen, and Then Autograd Agrees

Wesam Khallaf — Author of PyTorch From Ground Up — Sun, 23 Aug 2026 07:20:10 +0000

in the last piece i did autograd, and the whole point of it was that you call loss.backward() once and pytorch fills in every gradient for you, and a couple of people basically said the same thing back to me, ok that is nice but on a real network with more than one layer what is it actually doing in there. so this time i want to do the thing you will genuinely never do inside a training loop, i want to take a two-layer network and compute every gradient by hand, one node at a time, and then type the same thing into pytorch and watch autograd land on the exact same numbers. the one-neuron version was the warm-up. this is the one where the chain rule has to travel back through a hidden layer, and where the interesting failures live.

The network, small enough to hold in your head

Two layers. one input, a hidden neuron with a ReLU, then an output neuron, then a squared-error loss. that is it. i am going to write it as plain scalars first because the whole trick is visible that way and nothing is hiding inside a matrix.

x ──▶ h1 = w1·x + b1 ──▶ a1 = ReLU(h1) ──▶ h2 = w2·a1 + b2 ──▶ L = (h2 − y)²

four parameters to find gradients for: w1, b1, w2, b2. and i am picking numbers so there is nothing to hide behind:

x  = 1.0
w1 = 2.0    b1 = 0.0
w2 = 3.0    b2 = 1.0
y  = 2.0     (the target)

Forward pass, left to right, writing everything down

you compute the answer first, and you keep every intermediate value, because the backward pass needs them. this is not optional bookkeeping, the local derivatives literally reuse these numbers.

h1 = w1·x + b1 = 2·1 + 0   = 2.0
a1 = ReLU(h1)  = max(0, 2)  = 2.0
h2 = w2·a1 + b2 = 3·2 + 1   = 7.0
L  = (h2 − y)²  = (7 − 2)²   = 25.0

loss is 25. now the only question that matters is how each of the four parameters should move to make that 25 smaller, and that is exactly what a gradient is, ∂L/∂w1 and so on, the slope of the
loss with respect to that one number while everything else is held still.

Backward pass, one node at a time, right to left

the entire method is this, and it does not get more complicated no matter how deep the net is: at each node you take the gradient arriving from the right, and you multiply it by that node's own
local derivative, and you pass the result further left. that is the chain rule and there is nothing else to it.

loss → h2. L = (h2 − y)², so the derivative of the loss with respect to h2 is 2(h2 − y).

∂L/∂h2 = 2·(7 − 2) = 10.0

h2 → the output parameters. h2 = w2·a1 + b2. the derivative of h2 with respect to w2 is just a1, and with respect to b2 it is 1, so multiply each by the 10 coming in:

∂L/∂w2 = 10 · a1  = 10 · 2 = 20.0
∂L/∂b2 = 10 · 1   = 10.0

so far identical in spirit to the one-neuron case. here is the part that only shows up once you have a hidden layer.

h2 → a1, i.e. keep going left into the first layer. h2 = w2·a1 + b2, so the derivative of h2 with respect to a1 is w2. the gradient does not stop at the output layer, it flows backthrough w2 into the hidden activation:

∂L/∂a1 = 10 · w2 = 10 · 3 = 30.0

a1 → h1, through the ReLU. ReLU passes its input when it was positive and its derivative is 1 there, 0 when the input was negative. our h1 was 2, positive, so the gate is open:

∂a1/∂h1 = 1     (because h1 > 0)
∂L/∂h1  = 30 · 1 = 30.0

h1 → the first-layer parameters. h1 = w1·x + b1, derivative with respect to w1 is x, with respect to b1 is 1:

∂L/∂w1 = 30 · x = 30 · 1 = 30.0
∂L/∂b1 = 30 · 1 = 30.0

done, by hand, four gradients:

∂L/∂w1 = 30    ∂L/∂b1 = 30    ∂L/∂w2 = 20    ∂L/∂b2 = 10

notice the shape of what happened. the 10 computed at the output got carried all the way back to the first layer, multiplied by w2 on the way through, then by the ReLU's 1, then by x. every
gradient in the network is that same number from the loss, multiplied by a chain of local slopes between it and the parameter. deeper nets are just longer chains.

Now ask PyTorch the same question

same numbers, same five-ish lines, mark the four parameters as needing gradients, run forward, call backward once, print:

import torch

x  = torch.tensor(1.0)
w1 = torch.tensor(2.0, requires_grad=True)
b1 = torch.tensor(0.0, requires_grad=True)
w2 = torch.tensor(3.0, requires_grad=True)
b2 = torch.tensor(1.0, requires_grad=True)
y  = torch.tensor(2.0)

h1 = w1 * x + b1
a1 = torch.relu(h1)
h2 = w2 * a1 + b2
loss = (h2 - y) ** 2

loss.backward()

print(f"loss   = {loss.item()}")     # 25.0
print(f"dL/dw1 = {w1.grad.item()}")  # 30.0
print(f"dL/db1 = {b1.grad.item()}")  # 30.0
print(f"dL/dw2 = {w2.grad.item()}")  # 20.0
print(f"dL/db2 = {b2.grad.item()}")  # 10.0

all four match, and the loss matches. autograd traced the same chain of operations, stored the same intermediate values on the forward pass, and multiplied the same local derivatives on the way back.
the only thing it did that you did not is bookkeeping, it just did it without asking you and it would do it the same way with ten million parameters instead of four.

if you read nothing else in this article, run that block and put it next to the hand calculation above. the whole point is the moment where the pen and the computer agree.

The part that turns this into a debugging skill

keep the network the same but break the hidden neuron on purpose. flip w1 to -2 and leave everything else. now the first layer produces a negative pre-activation:

h1 = w1·x + b1 = -2·1 + 0 = -2.0
a1 = ReLU(-2)  = 0.0
h2 = w2·a1 + b2 = 3·0 + 1 = 1.0
L  = (1 − 2)²   = 1.0

run backward on this one and look at what comes out:

w1 = torch.tensor(-2.0, requires_grad=True)   # the only change
b1 = torch.tensor(0.0,  requires_grad=True)
w2 = torch.tensor(3.0,  requires_grad=True)
b2 = torch.tensor(1.0,  requires_grad=True)
x, y = torch.tensor(1.0), torch.tensor(2.0)

h1 = w1 * x + b1
a1 = torch.relu(h1)
h2 = w2 * a1 + b2
loss = (h2 - y) ** 2
loss.backward()

print(w1.grad.item(), b1.grad.item())   # 0.0 0.0
print(w2.grad.item(), b2.grad.item())   # 0.0 -2.0

three of the four gradients are zero, and the loss is not, the network is wrong and mostly getting no signal about it. and because you just did the hand version you can say exactly why each zero
happened instead of guessing.

w1 and b1 are zero because the ReLU's derivative is 0 when its input was negative, and that 0 sits in the middle of the chain, so everything behind it gets multiplied down to nothing. that is
the dead-ReLU problem and this is it in miniature, a neuron whose input is negative gets no gradient and stops learning, quietly, no error printed.
but look at w2, it is also zero, and the ReLU is not the reason. ∂L/∂w2 = ∂L/∂h2 · a1, and a1is 0 here, so the weight that reads from the dead neuron gets a zero gradient too, because you
cannot learn how to weight an input that is always zero. so a single dead neuron in the hidden layer takes out both the parameters feeding it and the weight reading out of it.

b2 is the survivor, ∂L/∂b2 = ∂L/∂h2 = -2, the output bias sits after the dead neuron so the zero never reaches it, and it keeps learning. that is the whole reason this is worth doing on two
layers and not one, on one neuron you just see "gradient is zero", on two you see the zero spread along the chain and stop where the chain does.

this is the payoff of the exercise. you will never backprop by hand in a real loop, autograd owns that. but the day a model just sits there and does not improve and you print a gradient and it is 0,
you will not be staring at it, you will go straight to asking what on the path back is multiplying by zero, a dead ReLU, a detach you forgot about, an activation that saturated, and you will know that
a zero at one node zeros everything behind it but not in front of it, because you have already watched it happen on four parameters you could hold in your head.

Two things that may be confusing at first glance

the gradient does not stop at the layer that produced the loss. the most common beginner mental model is that each layer computes its own gradient locally and independently. it does not. the 10
from the output travelled all the way to w1. every layer's gradient depends on every layer after it, that dependence is the · w2 step, and it is why the order is strictly right to left and why you cannot parallelise the backward pass across depth the way you can the forward one.

gradients accumulate, they do not overwrite. if you run loss.backward() twice without clearing,
the second set of numbers gets added onto the first and you will read 60, 60, 40, 20 on the second call and wonder what happened. that is deliberate, it is what lets you sum gradients over several mini-batches, but it means optimizer.zero_grad() before every backward is load-bearing, not boilerplate. i wrote about why that is in the autograd piece.

Try it before you close the tab

take the healthy network again, x=1, w1=2, b1=0, w2=3, b2=1, y=2, and this time add a second hidden step of your own, say multiply a1 by another weight w3 = 0.5 before the output. write out
the full chain for ∂L/∂w1 on paper, there is just one more factor in it now, then check against autograd. it takes a few minutes and it is worth more than rereading this.

then the mean one, set b1 = -5 so the hidden ReLU dies, predict which of the four gradients go to zero before you run it, and see if you called it right. if you can predict the zeros you have
actually got the chain rule, if you cannot you have more to gain from running it than from any amount of reading.

what is the worst "model just would not learn and the gradient was zero" you have hit? for me the first real one was a dead ReLU exactly like this, a whole layer of them, and it cost me most of a day before i thought to print a gradient. i am curious whether that is the usual first one or whether everyone finds their own way to multiply by zero.

This is one chapter's worth of an idea from my book, PyTorch From Ground Up, which builds everything from tensors upward so nothing stays vague. If it helped:
8 chapters are free, no email required, there's a free one-page tensor cheat-sheet here, every example runs in
the companion notebooks on GitHub, and the full book is on Leanpub or in
paperback and Kindle on Amazon.

PyTorch Autograd Explained: What .backward() Actually Does

Wesam Khallaf — Author of PyTorch From Ground Up — Wed, 12 Aug 2026 20:50:08 +0000

most autograd tutorials show you requires_grad, then .backward(), then .grad, and then they hand you a training loop, and you come out of it able to run autograd and still not able to say what it did, and i think that is backwards. so in this one you do the derivative first, by hand, on a tiny network that fits on one page, and only once you have your own number do we run pytorch and check it against .grad. the two agree to the decimal. and that is the point where autograd stops being magic, because you already did the thing it does, you just did it slower.

First, what a gradient actually is

Skip this section if you are comfortable with it, but most confusion about autograd is really confusion about what it is producing.

Imagine standing on a hillside in fog. You cannot see the bottom, but you can feel that the ground tilts. Step in the direction it tilts downward and you get lower. That is a gradient: the slope of the ground under your feet.

Now put that on a graph. The horizontal axis is one adjustable number inside the model, a parameter. The vertical axis is the loss. The curve shows how the loss changes as the parameter
changes, and somewhere on it there is a lowest point.

Take y = x². Its slope at position x is 2x. At x = 3 the slope is 6, steep and positive, so the loss climbs to the right and you should step left. At x = -2 it is -4, so you step right. At x = 0 it is 0, and you have arrived.

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
y.backward()
print(x.grad)      # tensor(6.)

PyTorch produced the slope without being told the formula. That is autograd, and the rest of this article is how.

Turning on the tape

Tensors do not track gradients by default. That would be wasted work on your input data, which is never adjusted. You opt in with requires_grad=True:

x = torch.tensor(3.0, requires_grad=True)
print(x.requires_grad)    # True

From this moment every operation involving x gets recorded. Think of it as a receipt. Each multiplication, addition and power you apply gets written down, along with enough information to reverse it later.

The computation graph

The receipt has a proper name, the computation graph. It is a chain of nodes where each node is an operation and each edge carries a tensor from one operation into the next.

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2        # node: power
z = 2 * y + 1     # nodes: multiply, then add

Three lines of Python, four values, three operation nodes:

x (leaf)      **2         ×2          +1        z
  3.0    →    9.0    →   18.0   →    19.0   →  19.0

This is the forward pass. It runs left to right, computes the answer, and as a side effect builds the graph. Nothing has been differentiated yet. The graph exists purely so that something can walk it backwards.

Two things are worth noticing here. First, x is a leaf — you created it directly rather than computing it from something else. Second, every result node remembers the operation that produced it. You can see this:

print(z.grad_fn)     # <AddBackward0 object at ...>
print(y.grad_fn)     # <PowBackward0 object at ...>
print(x.grad_fn)     # None  ← leaves have no history

grad_fn is the recording. AddBackward0 is not the addition, it is the instruction for reversing the addition.

Calling .backward()

z.backward()
print(x.grad)     # tensor(12.)

Check it by hand. z = 2x² + 1, so dz/dx = 4x, and at x = 3 that is 12. Correct.

But the interesting part is not that the answer is right, it is how it was produced, because PyTorch never formed the expression 4x at all. It walked the graph right to left and at each node multiplied the incoming gradient by that node's own local slope:

z = 19          +1              ×2              **2            x.grad
start           slope 1         slope 2         slope 2x = 6
grad 1     →    grad 1     →    grad 2     →    grad 12

Reading right to left: the +1 node has slope 1, so the gradient passes through unchanged. The ×2 node has slope 2, so the gradient doubles. The x² node has slope 2x, which at x = 3 is 6, so the gradient multiplies by 6. Altogether 1 × 2 × 6 = 12.

That is the whole of backpropagation. It is the chain rule, applied one node at a time, right to left, automatically. Nobody ever writes down 4x.

This also explains why the forward pass has to happen first, and why it has to store its intermediate values. The x² node's local slope is 2x, which needs the value of x that went in. Every node keeps whatever it needs to compute its own derivative later. This is where the memory goes during training, and it is why a batch that fits in memory for inference can still run you out of memory when training.

Now do it on an actual network, by hand

Everything above is one variable. Here is the smallest thing that is honestly a neural network:
one input, one weight, one bias, a ReLU, and a squared-error loss. Five operations. Enough to show every part of the chain rule, small enough to hold in your head.

Numbers: x = 2.0, w = 3.0, b = -1.0, target y = 2.0.

Forward pass, left to right, writing down everything:

h = w·x + b     = 3·2 + (-1)  = 5.0
a = ReLU(h)     = max(0, 5)   = 5.0
L = (a - y)²    = (5 - 2)²    = 9.0

Loss is 9. Now the question that matters: how should w and b change to make it smaller?

Backward pass, one node at a time, right to left. At each node, local derivative times the
gradient arriving from the right.

Loss to a. L = (a - y)², so ∂L/∂a = 2(a - y) = 2(5 - 2) = 6.0

a to h, through the ReLU. ReLU's derivative is 1 if its input was positive and 0 if it was
negative. Our h was 5, positive, so ∂a/∂h = 1.0 and therefore
∂L/∂h = 6.0 × 1.0 = 6.0

h to w and b. Since h = w·x + b:

∂h/∂w = x = 2.0    →    ∂L/∂w = 6.0 × 2.0 = 12.0
∂h/∂b = 1.0        →    ∂L/∂b = 6.0 × 1.0 = 6.0

So by hand: ∂L/∂w = 12 and ∂L/∂b = 6.

Now ask PyTorch the same question:

x = torch.tensor(2.0)
w = torch.tensor(3.0, requires_grad=True)
b = torch.tensor(-1.0, requires_grad=True)
y = torch.tensor(2.0)

h = w * x + b
a = torch.relu(h)
loss = (a - y) ** 2

loss.backward()

print(f"loss  = {loss.item()}")     # 9.0
print(f"dL/dw = {w.grad.item()}")   # 12.0
print(f"dL/db = {b.grad.item()}")   # 6.0

Exact match, all three. Autograd traced the same five operations and applied the same chain rule. It just did it without asking you.

If you read nothing else here, run that block and compare it to the hand calculation above. The whole point of the exercise is the moment the numbers agree.

What happens when the ReLU is closed

Change one number. Keep w = 3 and b = -1, but feed x = -1.0. Now h = 3·(-1) + (-1) = -4, which is negative, so ReLU outputs 0 and its local derivative is also 0.

x = torch.tensor(-1.0)
w = torch.tensor(3.0, requires_grad=True)
b = torch.tensor(-1.0, requires_grad=True)
y = torch.tensor(2.0)

h = w * x + b          # -4.0
a = torch.relu(h)      #  0.0
loss = (a - y) ** 2    # (0 - 2)² = 4.0
loss.backward()

print(w.grad.item())   # 0.0
print(b.grad.item())   # 0.0

The loss is 4, so the network is wrong, and yet both gradients are exactly zero. No learning
signal reaches w or b at all.

This follows straight from the hand calculation. ∂a/∂h = 0, and every gradient behind that node gets multiplied by zero on its way through. The ReLU is a gate, and this one is shut.

That is the dying-ReLU problem, and it is worth meeting it here rather than three months later in a model with fifty layers. When a neuron's input is negative for every example in your data, its gradient is permanently zero and it never learns again. LeakyReLU exists because it lets a small gradient through instead.

More generally: when a model stops learning and you cannot see why, the question to ask is what is multiplying the gradient by zero on the way back.

Three mechanics that catch nearly everyone

1. Only leaves keep their `.grad`

Intermediate results do not store their gradient. It is computed, used to keep the chain going, and thrown away, because keeping every intermediate gradient in a real model would be enormous. If you want one, ask before calling backward:

x = torch.tensor(2.0, requires_grad=True)
y = x ** 3
y.retain_grad()        # keep y's gradient too
z = y * 5
z.backward()

print(x.grad)    # tensor(60.)   dz/dx = 15x² = 60
print(y.grad)    # tensor(5.)    dz/dy = 5

Without retain_grad(), y.grad is None and you get a warning rather than an error, which is why people spend twenty minutes confused by it.

2. Gradients accumulate, they do not overwrite

This one causes more silently broken training loops than anything else in PyTorch.

x = torch.tensor(3.0, requires_grad=True)

for i in range(3):
    y = x ** 2
    y.backward()
    print(f"step {i}  grad = {x.grad.item()}")

step 0  grad = 6.0
step 1  grad = 12.0
step 2  grad = 18.0

The gradient of x² at x = 3 is 6, every time. It reads 12 and then 18 because .grad is added to, not replaced.

This is deliberate. It is what lets you accumulate gradients over several mini-batches and take one larger step, which is how people train with an effective batch size their GPU cannot hold.
But it means that unless you clear it, every step of your training loop is stepping on a sum of all previous gradients, and your loss curve will do something strange that is very hard to diagnose from the outside.

The fix is x.grad.zero_(), or in a real loop, optimizer.zero_grad() before every loss.backward(). That line is not boilerplate. It is load-bearing.

3. `no_grad` and `detach` — switching the recorder off

During evaluation you do not need gradients, and building the graph costs both time and memory.

x = torch.tensor(3.0, requires_grad=True)

with torch.no_grad():
    y = x ** 2
    print(y.requires_grad)   # False

Inside the block no graph is built. This is why every evaluation loop you have ever copied is wrapped in torch.no_grad().

.detach() is the narrower tool. It gives you a tensor that shares the same data but has no connection to the graph:

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
y_val = y.detach()

print(y_val)                  # tensor(9.)
print(y_val.requires_grad)    # False

Use it when you want a value without dragging its history along — logging, or using a model's output as a target that should not be differentiated through. It is the mechanism behind stop-gradient tricks and target networks, and it is also, occasionally, the reason your gradient is unexpectedly zero.

One call, every parameter

Everything so far had one or two parameters so the arithmetic stayed visible. Nothing changes at scale.

w = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(1.0, requires_grad=True)
x = torch.tensor(3.0)

y = w * x + b            # 2·3 + 1 = 7
loss = (y - 5) ** 2      # (7 - 5)² = 4

loss.backward()

print(w.grad)   # tensor(12.)
print(b.grad)   # tensor(4.)

One backward() filled in both. In a model with eleven million parameters, the same single call fills in all eleven million, because the graph reaches every one of them and the chain rule multiplies along every path. Nothing about the mechanism is different. There are just more nodes.

The summary worth keeping

requires_grad=True marks a tensor as something you want the gradient of. Model parameters get it automatically; your data does not need it.
The forward pass computes the answer and records a graph as a side effect. Each node stores what it needs to reverse itself later.
.backward() walks that graph in reverse, multiplying the incoming gradient by each node's local derivative. That is the chain rule, and it is all backpropagation is.
Gradients land in .grad on leaf tensors only. Use retain_grad() for intermediates.
Gradients accumulate. zero_grad() before every backward, always.
torch.no_grad() for evaluation, .detach() for a single value without its history.
A zero gradient means something on the path back multiplied by zero. A closed ReLU is the usual suspect.

Try this before you close the tab

Take x = 4.0 with requires_grad=True, compute y = x**3 - 2*x, call backward(), and print
x.grad. Then work out 3x² - 2 at x = 4 on paper and check whether they agree. It takes
about ninety seconds and it is worth more than rereading this article.

Then the harder version: put a second weight after the ReLU in the tiny network above, w2 = 0.5, and write out the full chain rule for ∂L/∂w before you run it.

This is one chapter's worth of an idea from my book, PyTorch From Ground Up, which builds
everything from tensors upward so nothing stays vague. If it helped: 8 chapters are free, no email required,
there's a free one-page tensor cheat-sheet here, every example runs
in the companion notebooks on GitHub, and
the full book is on Leanpub or in
paperback and Kindle on Amazon.

More in this series

Shape mechanics, the part that has to be solid before any of this makes sense:

Your CNN's Advantage Is One Assumption — and I Measured What Happens When It Breaks

Wesam Khallaf — Author of PyTorch From Ground Up — Sat, 08 Aug 2026 12:19:03 +0000

A small convolutional network beats a plain flatten-and-feed-it-forward network by 7.0 points on CIFAR-10. That's convolutions, pooling, normalisation and skip connections doing honest work.

Then I shuffled the rows of every image, destroying no information at all, and that 7.0-point margin fell to 0.3.

Same architecture. Same data, in a strict sense I'll defend in a moment. Almost the entire advantage, gone.

The experiment

Take one fixed permutation of the 32 row indices. Apply it to every image in the training set and every image in the test set — the same permutation, every time.

import torch

g = torch.Generator().manual_seed(1234)
row_perm = torch.randperm(32, generator=g)

def shuffle_rows(x):        # x: (C, H, W)
    return x[:, row_perm, :]

print(row_perm[:8].tolist())   # [15, 9, 8, 1, 4, 12, 30, 7]

That's the whole intervention. Then train two models twice each — once on natural images, once on shuffled ones:

Model	Params	Natural rows	Shuffled rows
Flatten → 512 → 10 (MLP)	1,578,506	51.4%	51.7%
Small CNN	94,538	58.4%	52.0%
CNN's margin		+7.0 pts	+0.3 pts

The baseline is a real fully-connected network, not a single linear layer — Flatten → Linear(3072, 512) → ReLU → Linear(512, 10). It has the capacity to learn anything the CNN can; what it lacks is any reason to look at pixels near each other.

Two things in that table are worth sitting with. The CNN wins the natural case with sixteen times fewer parameters — that's the prior paying for itself. And in the shuffled case it doesn't just lose its lead; it drops 6.4 points in absolute terms, down to roughly where the linear model already was.

"You destroyed the data" — no, and this is the important part

This is the objection everyone raises, so let's take it seriously, because the experiment is worthless if the objection holds.

A fixed permutation is a bijection. Nothing is added, nothing is removed, nothing is averaged or blurred:

img = torch.arange(3*32*32, dtype=torch.float32).reshape(3, 32, 32)
sh  = shuffle_rows(img)

print(sh.shape)                                     # torch.Size([3, 32, 32])

# every value still present, exactly once
print(torch.equal(sh.flatten().sort().values,
                  img.flatten().sort().values))     # True

# and it's perfectly reversible
inv = torch.empty_like(row_perm)
inv[row_perm] = torch.arange(32)
print(torch.equal(sh[:, inv, :], img))              # True

Every pixel is still there, with its original value. The transformation is invertible, so no information has been lost in any sense that information theory would recognise. What changed is which pixels sit next to which.

And here's the control experiment that settles it: the fully-connected model scores the same either way — 51.4% natural, 51.7% shuffled, a difference well inside run-to-run noise. If the shuffle had damaged the data, the linear model would have suffered too. It didn't, because it never used the layout in the first place — after flatten(), position 400 is just position 400, and a fixed permutation of the input columns is something the first layer absorbs by permuting its own weights. The network is free to relearn the identical function; only the column labels moved.

So the shuffle removes exactly one thing: the usefulness of the assumption that neighbouring pixels are related. And that assumption turns out to be worth 6.7 of the CNN's 7.0 points.

Which is the uncomfortable version of the finding: the convolutions, the pooling, the normalisation — all of it — were converting one true fact about photographs into 7 points of accuracy. Take the fact away and the machinery has nothing left to convert.

What a convolution actually assumes

Written out, a conv layer makes three claims about your data, none of which are claims about tensors:

Locality. A 3×3 kernel only ever sees a 3×3 neighbourhood. This is a bet that meaningful patterns are local — that to recognise an edge you need nearby pixels and not distant ones. True of photographs. Not true of a shuffled photograph, where the pixels that formed an edge are now scattered across the image.

Weight sharing. The same kernel slides over every position, so a feature detected in the top-left uses identical weights to the same feature bottom-right. This is a bet that what a pattern is doesn't depend on where it is. It's also where the parameter savings come from — and it's why the shuffled case doesn't merely lose the advantage but wastes capacity, since the kernel is now sharing weights across positions that have nothing in common.

Hierarchy. Stacked layers assume small patterns compose into larger ones: edges into corners, corners into shapes. A shuffled image has edges nowhere, so there is nothing to compose.

All three are statements about the world, not the maths. When they hold, they're enormously valuable — a prior that good is worth more than a lot of data. When they don't hold, depth doesn't manufacture them.

Try it yourself

Roughly this, on CPU, in a few minutes:

import torch, torchvision
from torchvision import transforms

g = torch.Generator().manual_seed(0)
row_perm = torch.randperm(32, generator=g)

norm = transforms.Normalize([0.5] * 3, [0.5] * 3)
base = [transforms.ToTensor(), norm]
shuffled = base + [transforms.Lambda(lambda x: x[:, row_perm, :])]

train_nat = torchvision.datasets.CIFAR10(root="./data", train=True, download=True,
                                         transform=transforms.Compose(base))
train_shuf = torchvision.datasets.CIFAR10(root="./data", train=True, download=True,
                                          transform=transforms.Compose(shuffled))
# ...and the same two for train=False

Then train any small CNN and any Flatten → Linear → ReLU → Linear on each of the four loaders. Four runs. Watch the CNN's lead evaporate while the fully-connected model shrugs.

Two things to keep honest while you do it: build the permutation once, outside the transform, or you'll get a different shuffle per image, which really does destroy information and proves nothing. And use the same seed and epoch count across all four runs, or you're measuring your own variance.

My run, for the record: torch 2.13.0 on CPU, manual_seed(0), Adam at 1e-3, batch 128 train / 256 eval, 5 epochs, normalised to mean 0.5 / std 0.5, row permutation seeded 1234.

Where this actually matters

This isn't a party trick. It's the question to ask before choosing any architecture: what is this thing assuming about my data, and is the assumption true?

Tabular data. Columns have no spatial relationship — age next to postcode next to income is an arbitrary ordering you could permute at will. A CNN over columns is the shuffled experiment, permanently. This is a large part of why gradient-boosted trees still win on tabular problems.
Spectrograms and time series. These do have local structure along at least one axis, so a 1-D convolution is a real prior rather than a fashion choice. Neighbouring time steps are genuinely related.
Channel order. Permuting the channel axis of an image is not the same operation and is nearly harmless, because a conv kernel spans all channels at once — there's no locality assumption along that axis to break. Worth knowing, because it's the first thing people try to use as a counter-example.
Attention. Self-attention makes no locality assumption at all, which is exactly why transformers need positional encoding bolted on and why they need more data to reach the same place on images. Fewer assumptions is not automatically better; it means less free knowledge and more required evidence.

The honest caveats

One dataset, one small architecture, one seed, at 32×32, five epochs — nobody's state of the art. The direction of this result is robust and it's the standard inductive-bias argument that the literature has made for years. The specific numbers are mine and yours will differ — the point of running it isn't the decimal places, it's watching a 7-point advantage turn into a rounding error while the data stays intact.

The takeaway

Architecture isn't magic and it isn't a leaderboard. It's a bet about the structure of your data, made before training starts. A convolution bets that nearby things belong together. On photographs that bet pays enormously. Break its truth without touching a single pixel value and the same architecture is worth about a third of a point.

So when someone asks why your model isn't learning, the useful question often isn't "should I go deeper" — it's whether the architecture's assumptions were ever true of your data.

This experiment is from Chapter 8 of **PyTorch From Ground Up, Volume 2, which I'm writing now. The whole book works this way: no solution gets introduced until you've watched the failure it fixes — no convolution before a linear layer fails, no skip connection before a 56-layer network loses to a 20-layer one.

Run it yourself: open the Chapter 8 notebook in Colab — every code block from the chapter, in order, CPU-only, nothing to install. The four-model comparison is the slow cell, about 15 minutes.

Volume 2 isn't finished yet. Volume 1 — Foundations, the tensors-and-training-loop book underneath all of this — is out, and 8 of its chapters are free with no email required.

What's an architecture choice that quietly bought you nothing? I collect these — the ones that surprise me end up as experiments.

PyTorch `permute` vs `transpose`: What's the Difference (and the `reshape` Bug That Scrambles Your Images)

Wesam Khallaf — Author of PyTorch From Ground Up — Sun, 02 Aug 2026 18:23:12 +0000

You loaded an image, got a tensor shaped (batch, height, width, channels), and your convolution wants (batch, channels, height, width). Stack Overflow says permute. Someone else says transpose. And reshape(2, 3, 28, 28) gives you the right shape too — so why is everyone making this complicated?

Because two of those three are the same tool, and the third one silently destroys your data.

The short answer

transpose(dim0, dim1) swaps exactly two dimensions. permute(...) reorders all of them in one call, and you must list every dimension. transpose is a special case of permute. Both return a view — no data is copied, only the strides change — which also means both leave you with a non-contiguous tensor.

reshape is not in this family at all. It reinterprets the flat memory under a new shape without moving anything, so it can produce the shape you asked for while completely scrambling what the numbers mean.

import torch

t = torch.arange(24).reshape(2, 3, 4)

print(t.transpose(0, 1).shape)     # torch.Size([3, 2, 4])  — swapped dims 0 and 1
print(t.permute(2, 0, 1).shape)    # torch.Size([4, 2, 3])  — full reorder

`transpose` — swap two axes

transpose(dim0, dim1) takes two dimension indices and swaps them. Everything else stays put.

t = torch.arange(24).reshape(2, 3, 4)

print(t.shape)                  # torch.Size([2, 3, 4])
print(t.transpose(0, 1).shape)  # torch.Size([3, 2, 4])
print(t.transpose(1, 2).shape)  # torch.Size([2, 4, 3])

The order of the two arguments doesn't matter — t.transpose(0, 1) and t.transpose(1, 0) are the same thing. A swap is a swap.

On a 2-D tensor this is the matrix transpose you already know, and .T is the shorthand:

m = torch.arange(6).reshape(2, 3)
print(m.T.shape)                # torch.Size([3, 2])
print(m.transpose(0, 1).shape)  # torch.Size([3, 2])  — identical

One caution on .T: on tensors with more than two dimensions, .T reverses every dimension, and modern PyTorch has deprecated that behaviour — it warns now and is slated to become an error. If you want the "transpose the last two axes" behaviour on a batch of matrices, use .mT, which is explicit and safe:

t = torch.arange(24).reshape(2, 3, 4)
print(t.mT.shape)   # torch.Size([2, 4, 3]) — swaps the last two dims, batch untouched

Reserve plain .T for 2-D matrices. On anything higher, say what you mean with permute or .mT.

`permute` — state the whole new order

permute doesn't swap; it rewrites the dimension order in full. You pass the old index of each dimension in the position you want it to end up.

t = torch.arange(24).reshape(2, 3, 4)
print(t.permute(2, 0, 1).shape)   # torch.Size([4, 2, 3])

Read permute(2, 0, 1) as: "new dim 0 is old dim 2, new dim 1 is old dim 0, new dim 2 is old dim 1." Not "move dim 2 somewhere" — you are writing out the destination order, left to right.

Two consequences worth internalising:

You must list every dimension. Miss one and you get RuntimeError: number of dims don't match in permute.
Any reorder you can do with a chain of transposes, you can do in one permute:

t = torch.arange(24).reshape(2, 3, 4)

via_permute    = t.permute(1, 2, 0)                    # torch.Size([3, 4, 2])
via_transposes = t.transpose(0, 1).transpose(1, 2)     # torch.Size([3, 4, 2])

print(torch.equal(via_permute, via_transposes))   # True

Same result. The permute version says the destination shape out loud; the chained version makes the reader simulate two swaps in their head. Prefer permute for anything beyond a single swap.

The one that actually bites: `permute` is not `reshape`

Here is the bug this article exists for. You have a batch of images in (N, H, W, C) — the layout you get from OpenCV, PIL, TensorFlow, and most image files — and PyTorch convolutions need (N, C, H, W).

Both of these produce a tensor of the right shape:

img = torch.arange(1 * 2 * 2 * 3).reshape(1, 2, 2, 3)   # N=1, H=2, W=2, C=3

print(img.permute(0, 3, 1, 2).shape)   # torch.Size([1, 3, 2, 2])  ✅
print(img.reshape(1, 3, 2, 2).shape)   # torch.Size([1, 3, 2, 2])  ⚠️ same shape!

Same shape. No error. No warning. Now look at what's actually inside the first channel:

print(img.permute(0, 3, 1, 2)[0, 0])
# tensor([[0, 3],
#         [6, 9]])            ← the red value of each of the 4 pixels ✅

print(img.reshape(1, 3, 2, 2)[0, 0])
# tensor([[0, 1],
#         [2, 3]])            ← pixel 0's R, G, B and then pixel 1's R ❌

The permute version collected the red channel: pixels are stored as (R,G,B)(R,G,B)…, so the reds are elements 0, 3, 6, 9. That's a real red channel.

The reshape version just took the first four numbers in memory and called them "channel 0" — that's one whole pixel plus a third of the next one. Your "red channel" is now a mixture of red, green and blue from different pixels. The tensor is the right shape, the model trains, the loss goes down a little, and the accuracy is quietly terrible. Nothing on screen ever tells you.

The rule: permute moves dimensions. reshape re-cuts the same flat sequence of numbers into new brackets. If you want to change what an axis means, you need permute — always.

If you want the full picture on how reshape re-cuts memory, that's covered in Reshape vs View in PyTorch.

Real code: the two reorders you'll actually write

Images — channels-last to channels-first:

img = torch.randn(2, 28, 28, 3)          # N, H, W, C  (from PIL / OpenCV)
x = img.permute(0, 3, 1, 2)              # N, C, H, W  (what nn.Conv2d wants)
print(x.shape)                           # torch.Size([2, 3, 28, 28])

Attention — splitting into heads. This is the one place where reshape and permute correctly appear back-to-back, and seeing why makes the distinction click:

B, S, E, H = 2, 5, 8, 2                  # batch, seq_len, embed_dim, heads

x = torch.randn(B, S, E)                 # torch.Size([2, 5, 8])
x = x.reshape(B, S, H, E // H)           # torch.Size([2, 5, 2, 4])  — split, don't move
x = x.permute(0, 2, 1, 3)                # torch.Size([2, 2, 5, 4])  — move heads forward

The reshape is legitimate here because it only splits the last axis — the 8 embedding values were already laid out contiguously, so cutting them into 2 groups of 4 doesn't reorder anything. The permute then genuinely moves the head axis in front of the sequence axis. Splitting an axis is reshape work; moving an axis is permute work.

Both of them break contiguity

transpose and permute never touch the underlying data. They change the tensor's strides — the step size PyTorch takes through memory for each dimension:

a = torch.arange(6).reshape(2, 3)
print(a.stride(), a.is_contiguous())            # (3, 1) True

b = a.permute(1, 0)
print(b.shape, b.stride(), b.is_contiguous())   # torch.Size([3, 2]) (1, 3) False

That's why they're free. It's also why the very next .view() you call will blow up:

b.view(6)
# RuntimeError: view size is not compatible with input tensor's size and stride
# (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

Two fixes, and the choice matters:

b.reshape(6)                # works — copies quietly when it has to
b.contiguous().view(6)      # works — reorders memory first, then views

Use .reshape() by default. Reach for an explicit .contiguous() when you're about to do many operations on the permuted tensor and want the memory laid out well once, up front, rather than copied repeatedly.

Common mistakes and gotchas

Using reshape to convert NHWC → NCHW. The headline bug. Right shape, scrambled channels, no error. Use permute(0, 3, 1, 2).
Reading permute(2, 0, 1) as instructions to move things. It's a destination list: new dim i is old dim args[i]. Say it out loud before you trust it.
Forgetting a dimension. permute needs all of them → RuntimeError: number of dims don't match in permute.
Using .T on 3-D+ tensors. Deprecated (it reverses all dims, which is almost never what you want). Use .mT for a batch of matrices, or permute to say it exactly.
Calling .view() right after a permute. Guaranteed contiguity error. .reshape() or .contiguous().view().
Assuming a transpose changed your data. It changed the strides only. print(x.stride()) next to print(x.shape) when a shape bug won't make sense — the strides usually tell you the real story.

Recap

transpose(a, b) swaps two dimensions; permute(...) rewrites the whole dimension order and needs every index listed. They do the same kind of work, and permute is the one to reach for whenever more than one swap is involved. Both are free view operations that leave the tensor non-contiguous, so the next .view() will error — use .reshape().

And the part that costs people a weekend: reshape is not a reordering tool. It can hand you a correctly-shaped tensor full of scrambled values, with nothing on screen to warn you. When an axis needs to mean something different, permute it.

More in this series: Reshape vs View · Broadcasting Explained · What unsqueeze Does · What keepdim Does

A question for you: what's the shape bug that cost you the most time? I'm collecting the ones that hit hardest — drop it in the comments and I'll write up the fix.

This is one idea from my book PyTorch From Ground Up, which builds everything from tensors upward so nothing stays vague. If it helped, you can grab the free PyTorch tensor cheat-sheet here, run every example from the book in the companion notebooks on GitHub, get the digital edition on Leanpub, or find the full paperback on Amazon here.

What Does `keepdim` Do in PyTorch? (And the Silent Bug When You Forget It)

Wesam Khallaf — Author of PyTorch From Ground Up — Fri, 24 Jul 2026 08:26:45 +0000

You wrote x.sum(dim=1) to get a row total, divided the original tensor by it, and either got RuntimeError: The size of tensor a (3) must match the size of tensor b (2) — or, worse, got no error at all and numbers that were quietly wrong. Someone told you to add keepdim=True and it fixed itself. But nobody said why.

The short answer

Every reduction — sum, mean, max, min — collapses a dimension, and by default that dimension disappears from the shape. keepdim=True tells PyTorch to keep it as a size-1 axis instead. A (2, 3) tensor reduced over dim=1 becomes (2,) normally, but (2, 1) with keepdim=True. Same numbers either way — the only difference is a single 1 in the shape. That 1 is what lets the result broadcast back against the original tensor.

import torch

m = torch.tensor([[1., 2., 3.],
                  [4., 5., 6.]])

print(m.sum(dim=1).shape)                 # torch.Size([2])
print(m.sum(dim=1, keepdim=True).shape)   # torch.Size([2, 1])

That's the whole feature. The interesting part is why the missing 1 breaks things — sometimes loudly, sometimes silently.

First: what does `dim` do in a reduction?

Before keepdim makes sense, you need the mental model for dim. The dimension you name is the one that disappears.

m = torch.tensor([[1., 2., 3.],
                  [4., 5., 6.]])   # shape (2, 3)

print(m.sum())          # tensor(21.)          — no dim: collapse everything to a scalar
print(m.sum(dim=0))     # tensor([5., 7., 9.]) — collapse the rows, shape (3,)
print(m.sum(dim=1))     # tensor([ 6., 15.])   — collapse the columns, shape (2,)

dim=0 adds down each column (the size-2 axis vanishes, leaving (3,)). dim=1 adds across each row (the size-3 axis vanishes, leaving (2,)). If you can predict the output shape before running, you understand reductions. The dim you name is gone.

So what does `keepdim` actually do?

By default the collapsed axis is deleted. keepdim=True leaves it behind as a length-1 placeholder:

row_sum   = m.sum(dim=1)                 # shape (2,)
row_sum_k = m.sum(dim=1, keepdim=True)   # shape (2, 1)

print(row_sum)
# tensor([ 6., 15.])
print(row_sum_k)
# tensor([[ 6.],
#         [15.]])

Identical values. The keepdim version is just wrapped so each row sum sits in its own row. That shape — (2, 1) instead of (2,) — is the entire point, because of what happens next.

Why size-1 matters: broadcasting back against the original

The most common reason to reduce a tensor is to then use that result on the original — normalize each row to sum to 1, subtract each column's mean, divide by a per-row max. Every one of those is original (op) reduction, and for it to line up, the reduction has to broadcast.

PyTorch broadcasting aligns shapes from the right and treats a 1 as "stretch me to fit." A (2, 1) sum broadcasts cleanly against a (2, 3) tensor: the 1 stretches across the 3 columns. A bare (2,) sum does not line up at all.

# Correct: keepdim=True → (2, 1) broadcasts against (2, 3)
normed = m / m.sum(dim=1, keepdim=True)
print(normed)
# tensor([[0.1667, 0.3333, 0.5000],
#         [0.2667, 0.3333, 0.4000]])
print(normed.sum(dim=1))   # tensor([1., 1.]) — every row sums to 1 ✅

Drop the keepdim and this exact line either crashes or lies to you. Which one you get depends on the shape — and that's the part almost every tutorial skips.

The trap: the same bug fails two completely different ways

Case 1 — a non-square tensor: it errors (the lucky case)

Our m is (2, 3). Without keepdim, the row sum is (2,), and PyTorch tries to align (2, 3) against (2,) from the right: 3 vs 2. Mismatch.

m = torch.tensor([[1., 2., 3.],
                  [4., 5., 6.]])          # (2, 3)

normed = m / m.sum(dim=1)                  # (2, 3) / (2,)
# RuntimeError: The size of tensor a (3) must match the size of
# tensor b (2) at non-singleton dimension 1

This is the good outcome. PyTorch stops you immediately, and the error text even names the two sizes that don't match. (If that message looks familiar, it's the same shape-mismatch error covered in PyTorch Broadcasting Explained.)

Case 2 — a square tensor: it runs, and it's silently wrong (the dangerous case)

Now make the tensor square. The row-sum vector is length 3, the tensor has 3 columns, so the shapes happen to broadcast — just not the way you meant.

m = torch.tensor([[1., 2., 3.],
                  [4., 5., 6.],
                  [7., 8., 9.]])          # (3, 3)

wrong   = m / m.sum(dim=1)                 # (3, 3) / (3,)  — no error!
correct = m / m.sum(dim=1, keepdim=True)   # (3, 3) / (3, 1)

print(wrong.sum(dim=1))    # tensor([0.4250, 1.2500, 2.0750])  ← NOT 1s ❌
print(correct.sum(dim=1))  # tensor([1., 1., 1.])              ← correct ✅

No exception. No warning. The row sums aren't 1 because the (3,) vector aligned to the columns instead of the rows — PyTorch divided the wrong way. Your data looks normalized, your code ran clean, and the mistake surfaces as a mysteriously bad model three hours later.

This is why you add keepdim=True by reflex, not by debugging. On a square batch it's the difference between correct and quietly-corrupted, with nothing on screen to tell you which one you got.

A rule you can actually remember

Whenever you reduce a tensor and then use the result in arithmetic with the original, pass keepdim=True.

That's it. If the reduction is the final answer (you just want the row totals and you're done), you don't need it. The moment the reduction feeds back into the original tensor, you do.

Does `keepdim` work on `max`, `min`, and `argmax` too?

Yes — max(dim) and min(dim) return two tensors (the values and their indices), and keepdim keeps the axis on both:

vals, idxs = m.max(dim=1, keepdim=True)
print(vals.shape, idxs.shape)   # torch.Size([3, 1]) torch.Size([3, 1])

This is handy for a "divide each row by its own max" style normalization, where you need that (3, 1) shape to broadcast. argmax(dim, keepdim=True) works the same way.

Common mistakes and gotchas

Forgetting keepdim on a square tensor. The headline trap: it broadcasts to the wrong axis with no error. Rows-or-columns bugs on square data are almost always a missing keepdim.
Thinking keepdim changes the numbers. It never does. The values are identical; only the shape gains a 1. If your totals changed, something else is wrong.
Reaching for unsqueeze after the fact. x.sum(dim=1).unsqueeze(1) gives the same (N, 1) shape, but x.sum(dim=1, keepdim=True) says it in one step and can't get the axis position wrong.
Assuming "it ran" means it was right. As Case 2 shows, running successfully is not the same as broadcasting correctly. Print .shape on the reduction before you divide.
Using keepdim when you don't need it. If the reduced value is your final output (not fed back into the original tensor), the trailing 1 is just clutter — drop it.

Recap

keepdim=True keeps a reduced dimension as a size-1 axis instead of deleting it, so the result still broadcasts against the original tensor. Forget it and you get one of two failures: a shape-mismatch RuntimeError on non-square tensors (annoying but safe), or a silent wrong answer on square tensors (dangerous). The habit that saves you: any time you reduce and then combine with the original — normalizing rows, centering columns, scaling by a per-row max — pass keepdim=True.

What Does `unsqueeze` Do in PyTorch? (And Why Your Model Keeps Asking For It)

Wesam Khallaf — Author of PyTorch From Ground Up — Sun, 19 Jul 2026 11:03:14 +0000

You passed a single image to your model and got ValueError: expected 4D input (got 3D input). Someone on Stack Overflow said "just add .unsqueeze(0)" — and it worked. But nobody explained what it did, or how you'd know to reach for it next time.

The short answer

unsqueeze(dim) inserts a new axis of size 1 at position dim. The data doesn't change at all — only the shape gains an extra 1. A tensor of shape (4,) becomes (1, 4) with unsqueeze(0) or (4, 1) with unsqueeze(1). Its inverse, squeeze(dim), removes a size-1 axis. Both are free: no memory is allocated and no data is copied.

import torch

v = torch.tensor([1, 2, 3, 4])   # shape (4,)

row = v.unsqueeze(0)             # (1, 4) — one row
col = v.unsqueeze(1)             # (4, 1) — one column

print(row.shape, col.shape)
# torch.Size([1, 4]) torch.Size([4, 1])

Same four numbers, three different shapes. That's the whole idea.

Why does PyTorch need size-1 dimensions at all?

Because a shape isn't just a size — it's a meaning. In PyTorch, shape (4,) means "four numbers." Shape (1, 4) means "one row of four." Shape (4, 1) means "four rows of one." Those are three different claims about your data, and PyTorch takes them literally.

That's why so many frustrating shape errors happen not because your data is the wrong size, but because it's missing a dimension — or has one too many. A model expects a batch of inputs and you hand it a single sample. A loss function wants a column and you give it a flat row. The fix, nearly every time, is one call.

The number you pass is where the new axis goes. Dimension 0 is the front, dimension 1 is next, and negative indexing works too — unsqueeze(-1) always appends a new last axis no matter how many dimensions the tensor already has.

t = torch.randn(3, 4)

print(t.unsqueeze(0).shape)    # torch.Size([1, 3, 4])
print(t.unsqueeze(1).shape)    # torch.Size([3, 1, 4])
print(t.unsqueeze(-1).shape)   # torch.Size([3, 4, 1])

The `None` shortcut (same thing, fewer characters)

There's a terser spelling you'll see constantly in real codebases: put None inside an index expression. Wherever you place None, a new size-1 axis appears.

v = torch.tensor([1, 2, 3, 4])

row = v[None, :]     # same as unsqueeze(0)
col = v[:, None]     # same as unsqueeze(1)

print(row.shape, col.shape)
# torch.Size([1, 4]) torch.Size([4, 1])

Both spellings do exactly the same thing. None indexing is shorter; unsqueeze is more explicit about where the axis lands. Use whichever reads more clearly — you'll meet both in the wild.

What does `squeeze` do in PyTorch?

squeeze is the inverse: it removes dimensions whose size is 1. With no argument it removes all of them; with a dim argument it removes only that one — and only if it really is size 1.

t = torch.randn(1, 3, 1, 4)

print(t.shape)                # torch.Size([1, 3, 1, 4])
print(t.squeeze().shape)      # torch.Size([3, 4])     — all 1s gone
print(t.squeeze(0).shape)     # torch.Size([3, 1, 4])  — only dim 0
print(t.squeeze(1).shape)     # torch.Size([1, 3, 1, 4]) — dim 1 is size 3, unchanged

Look at that last line. Squeezing dimension 1 did nothing, because dimension 1 has size 3, not 1. squeeze only removes axes that are actually size 1, so it's safe to call even when you're not sure.

Safe, but not always wise — see the gotchas below.

A worked example: the batch dimension

This is the case that sends most beginners to Google, so let's walk it end to end.

Most PyTorch models expect a batch — shape (N, ...), where N is how many samples you're passing. A single CIFAR-style image is shape (3, 32, 32): three channels, 32 by 32 pixels. That's three dimensions, and a lot of layers want four.

Here's the part that trips people up, because it's changed over the years: a bare nn.Conv2d will actually accept your unbatched image just fine. Modern PyTorch added no-batch-dim support to many nn modules, so this runs:

import torch
import torch.nn as nn

conv = nn.Conv2d(3, 16, kernel_size=3)
sample = torch.randn(3, 32, 32)      # one image — 3 dims

print(conv(sample).shape)            # torch.Size([16, 30, 30]) — no error

So why does everyone tell you to unsqueeze? Because the moment there's a normalization layer in the stack, the tolerance ends. BatchNorm2d computes statistics across the batch, so it genuinely cannot work without a batch axis:

model = nn.Sequential(
    nn.Conv2d(3, 16, kernel_size=3),
    nn.BatchNorm2d(16),
)

model(sample)

ValueError: expected 4D input (got 3D input)

Read it literally: it got 3 dimensions and wanted 4. Nothing is wrong with your image — it's just not in a batch. unsqueeze(0) wraps it in a batch of exactly one:

batch = sample.unsqueeze(0)          # (1, 3, 32, 32)
output = model(batch)

print(batch.shape)                   # torch.Size([1, 3, 32, 32])
print(output.shape)                  # torch.Size([1, 16, 30, 30])

The lesson generalizes: whether a single layer tolerates a missing batch dimension is a detail that varies by layer and by version. Whether your model does is not worth gambling on. Add the batch axis and the question disappears.

And on the way out, you often want that batch axis gone again, or a trailing singleton removed — a regression head that outputs (32, 1) won't line up with targets shaped (32,):

preds = torch.randn(32, 1)           # model output
print(preds.squeeze(-1).shape)       # torch.Size([32])

unsqueeze going in, squeeze coming out. That pairing shows up in almost every inference script you'll ever write.

The other big use: lining up shapes for broadcasting

The second place unsqueeze earns its keep is broadcasting. To subtract a per-channel mean from an image, the mean has to be shaped so it aligns with the channel axis — not the width axis.

mean = torch.tensor([0.485, 0.456, 0.406])   # (3,)
img = torch.randn(3, 224, 224)               # (C, H, W)

img - mean

RuntimeError: The size of tensor a (224) must match the size of tensor b (3)
at non-singleton dimension 2

Broadcasting aligns shapes from the right, so (3,) tries to line up with the 224-wide last axis and fails. Give it two trailing size-1 axes and it lines up correctly:

mean = mean[:, None, None]           # (3, 1, 1)
img = img - mean                     # broadcasts to (3, 224, 224)

print(mean.shape, img.shape)
# torch.Size([3, 1, 1]) torch.Size([3, 224, 224])

The 1s are placeholders that say "stretch me along this axis." That's what makes the subtraction apply per channel instead of per pixel column. If the right-to-left alignment rule is new to you, I wrote up how PyTorch broadcasting works in full here.

Are `squeeze` and `unsqueeze` expensive?

No — they're free. Both return a view of the same data with adjusted strides. No memory is allocated, nothing is copied, so you can call them freely, even inside a hot loop.

v = torch.arange(4)
u = v.unsqueeze(0)

print(u.data_ptr() == v.data_ptr())   # True
print(u.stride())                     # (4, 1)

Identical data pointer means no copy happened. unsqueeze just added a stride entry. This is the same principle behind view and reshape: any shape change expressible as a stride change is a view; anything else needs a copy.

Common mistakes and gotchas

Bare squeeze() eating your batch dimension. If your batch size happens to be 1, squeeze() with no argument silently deletes it along with every other singleton — and your shapes break several lines later, far from the cause. Prefer squeeze(dim), always.
Confusing unsqueeze(1) with unsqueeze(-1). They're the same only for 1-D tensors. On a (3, 4) tensor, unsqueeze(1) gives (3, 1, 4) while unsqueeze(-1) gives (3, 4, 1).
Expecting squeeze(dim) to raise on a non-1 axis. It doesn't — it quietly returns the tensor unchanged. Convenient, but it means a typo'd dim fails silently.
Reaching for reshape where unsqueeze is clearer. x.reshape(1, *x.shape) works, but x.unsqueeze(0) says what you mean.
Forgetting the batch axis at inference time. Training works because your DataLoader adds the batch dimension for you. Then you feed a single sample by hand and it breaks. That's not a new bug — it's the DataLoader no longer doing you a favor.
Trusting that "it ran" means the shape was right. Since some nn modules accept unbatched input, a missing batch dimension can survive several layers before something downstream complains — or worse, quietly produce output of the wrong rank. Print .shape.

Recap

unsqueeze(dim) inserts a size-1 axis at dim; squeeze(dim) removes one. v[None, :] is the same thing written inline. Both are free views — no copy. Use unsqueeze(0) to add a batch dimension for a single sample, unsqueeze(-1) or [:, None, None] to align shapes for broadcasting, and squeeze(-1) to drop a trailing singleton from model outputs.

Reshape vs View in PyTorch: What's the Difference (and When `view()` Breaks)

Wesam Khallaf — Author of PyTorch From Ground Up — Fri, 17 Jul 2026 06:59:46 +0000

You called .view() on a tensor, got a RuntimeError: view size is not compatible with input tensor's size and stride, switched to .reshape(), and it just... worked. So what's actually going on — and which one should you use?

The short answer

reshape and view both give you the same numbers under a new shape. The difference is one word: contiguity. view() returns a zero-copy view but requires the tensor to be contiguous — if it isn't, it errors. reshape() returns a view when it can and silently makes a copy when it can't, so it never errors on contiguity. Use reshape() by default; reach for view() only when you specifically need to guarantee shared memory.

Why reshape vs view exists at all

A tensor's data and its shape are two separate things. The data is a flat row of numbers sitting in memory; the shape is just the lens you look at it through. Reshaping changes the lens without touching the numbers — twelve elements can be seen as a 3×4 matrix, a 4×3, a 2×6, or a flat row of 12.

import torch

t = torch.arange(1, 13)     # 12 elements: 1..12
print(t.reshape(3, 4))
print(t.reshape(2, 6))
print(t.reshape(12))

The only constraint is that the product of the new shape must equal the total number of elements (numel()). Twelve elements can become (3, 4) because 3 × 4 = 12, but not (3, 5) because 3 × 5 = 15.

When the tensor is contiguous — which it almost always is right after you create it — view and reshape do exactly the same thing: return a zero-cost view of the same memory.

t = torch.arange(12)
a = t.view(3, 4)      # view — always free
b = t.reshape(3, 4)   # reshape — also free here
print(a.data_ptr() == t.data_ptr())   # True
print(b.data_ptr() == t.data_ptr())   # True

Same data pointer means no copy happened. So far, no difference.

What does "contiguous" mean in PyTorch?

A tensor is contiguous when its elements — read left-to-right along the last dimension, then the next dimension out, and so on — sit in that same order in memory. Most tensors are contiguous from birth. The operations that break contiguity are transpose, permute, and expand: they change the strides without moving the data.

t = torch.arange(6).reshape(2, 3)
print(t.is_contiguous())   # True
print(t.stride())          # (3, 1)

tt = t.T                   # transpose
print(tt.is_contiguous())  # False
print(tt.stride())         # (1, 3)

After the transpose the strides are reversed. The numbers are still in the same flat row 0 1 2 3 4 5, but reading them in the new dimension order no longer walks through memory in order. That is non-contiguous — and it's exactly the situation where view() breaks.

When `view()` breaks (the error you googled)

tt = t.T                       # non-contiguous
tt.view(6)                     # RuntimeError!

You'll see something like:

RuntimeError: view size is not compatible with input tensor's size and stride
(at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

PyTorch is telling you the operation isn't safe as a zero-copy view. You have two fixes:

print(tt.reshape(6))               # works — quietly copies
print(tt.contiguous().view(6))     # also works — rearranges memory first, then views

reshape is almost always the right default. Call .contiguous().view(...) only when you have a reason to force the copy explicitly.

Watch out: reshape doesn't guarantee a view

Because reshape returns a view sometimes and a copy other times, you cannot rely on it to share memory. If you need in-place edits to propagate back to the original, use view — and if it errors, that's PyTorch telling you the operation genuinely can't be done without copying.

The `-1` shortcut

You can pass -1 for exactly one dimension and PyTorch infers its size from the total element count:

t = torch.arange(24)
print(t.reshape(4, -1).shape)     # (4, 6)
print(t.reshape(-1, 3).shape)     # (8, 3)
print(t.reshape(2, 3, -1).shape)  # (2, 3, 4)

At most one -1 per call. This is everywhere in real code — you know some dimensions and let PyTorch solve for the last.

A worked example: flatten before a linear layer

The single most common reshape in practice is collapsing spatial dimensions before a fully-connected layer, while keeping the batch dimension intact:

batch = torch.randn(32, 3, 28, 28)   # N, C, H, W
flat = batch.flatten(1)              # flatten from dim 1 onward
print(flat.shape)                    # torch.Size([32, 2352])

flatten(1) keeps dimension 0 (the batch) and merges the rest into one. Note the trap: flatten() with no argument collapses everything, batch included — inside a model's forward pass, always use flatten(1).

Common mistakes and gotchas

Calling view() after transpose/permute. That's the classic RuntimeError. Use reshape() or .contiguous().view().
Assuming reshape shares memory. Sometimes it copies. If you need a guaranteed view, use view().
Using bare flatten() in a forward pass. It eats the batch dimension. Use flatten(1).
Wrong element count. reshape(3, 5) on 12 elements raises RuntimeError: shape '[3, 5]' is invalid for input of size 12. The product must equal numel().

Recap

view and reshape both reinterpret the same flat memory under a new shape. view requires contiguity and always returns a true view; reshape returns a view when it can and copies when it must, so it never errors on contiguity. Default to reshape; use view when you specifically need shared memory.

This is one idea from my book PyTorch From Ground Up, which builds everything from tensors upward so nothing stays vague. If it helped, you can grab the free PyTorch tensor cheat-sheet here, run every example from the book in the companion notebooks on GitHub, or find the full paperback on Amazon here or find a digital copy of the book on leanpup here

PyTorch Broadcasting Explained: The 3 Rules (and the Silent Bug That Bites Everyone)

Wesam Khallaf — Author of PyTorch From Ground Up — Thu, 16 Jul 2026 08:09:38 +0000

You added two tensors of different shapes and PyTorch didn't complain — it just returned something bigger than you expected. Or you got a RuntimeError about sizes that don't match and no idea which dimension is wrong. Both come down to one feature: broadcasting.

The short answer

PyTorch broadcasting follows exactly three rules: align shapes from the right, each aligned pair of dimensions must be equal or one of them must be 1, and any size-1 dimension is virtually stretched to match the other. If a pair is neither equal nor 1, you get a RuntimeError. The stretch never copies memory — it's implemented with a stride of 0. Learn the three rules once and broadcasting stops surprising you.

The core example

Add a column of three numbers to a row of four:

import torch

a = torch.tensor([[1], [2], [3]])   # shape (3, 1)
b = torch.tensor([[10, 20, 30, 40]]) # shape (1, 4)
c = a + b
print(c.shape)
print(c)

Output:

torch.Size([3, 4])
tensor([[11, 21, 31, 41],
        [12, 22, 32, 42],
        [13, 23, 33, 43]])

No loop, no manual copying. PyTorch stretched the column across four columns and the row down three rows, then added element by element. Critically, the stretch is virtual — no memory is allocated for the repeated values, which is why broadcasting is fast.

The 3 broadcasting rules

Align from the right. Line the shapes up starting at the last dimension. If one tensor has fewer dimensions, pad it with 1s on the left.
Each pair must match. For each aligned pair, the sizes must be equal, or one of them must be 1.
Size-1 dims stretch. Any dimension of size 1 is virtually repeated to match the other tensor along that axis.

Walking the core example through the rules:

        dim0   dim1
a         3      1   ->  stretches to 4
b         1      4   ->  stretches to 3   (b padded to (1, 4))
result    3      4

The output shape takes the max of each aligned pair: (3, 4).

Worked examples

Scalar + matrix. A scalar has shape () and pads to match anything:

m = torch.ones(3, 4)
print((m + 10).shape)   # (3, 4)

Vector + matrix. A (4,) vector aligns with the last dim of a (3, 4) matrix — it's implicitly treated as (1, 4), then stretched down three rows:

v = torch.tensor([1.0, 2.0, 3.0, 4.0])   # (4,)
m = torch.ones(3, 4)                      # (3, 4)
print((m + v).shape)   # (3, 4)

Per-channel image normalization. Subtract a (3, 1, 1) mean from a (3, 224, 224) image — this is standard ImageNet normalization, and the size-1 dims are exactly what broadcasting knows how to stretch:

img = torch.randn(3, 224, 224)
mean = torch.tensor([.485, .456, .406])
mean = mean[:, None, None]   # (3, 1, 1)
out = img - mean             # (3, 224, 224)
print(out.shape)

Why does broadcasting fail with a RuntimeError?

If a pair of aligned dimensions is neither equal nor 1, PyTorch raises a RuntimeError — it's protecting you from a silent mistake:

a = torch.ones(3)
b = torch.ones(5)
a + b   # RuntimeError!

RuntimeError: The size of tensor a (3) must match the size of tensor b (5)
at non-singleton dimension 0

The fix is always the same: reshape one tensor so the mismatched dimension becomes size 1 (with unsqueeze or reshape), then let broadcasting do the rest. If the shapes genuinely shouldn't combine, the error just saved you from a bug.

The silent broadcasting bug (this one has no error)

The real danger isn't the RuntimeError — that message tells you exactly what's wrong. The danger is when broadcasting succeeds silently and hands you a bigger tensor than you meant:

a = torch.randn(3)      # (3,)
b = torch.randn(3, 1)   # (3, 1)
c = a - b
print(c.shape)          # (3, 3)  <-- probably wrong!

You likely wanted an element-wise (3,) result. Instead (3,) was padded to (1, 3), (3, 1) stayed as is, and you got a 3×3 matrix. No error, no warning — just quietly wrong, and it compounds ten lines later. This is the number-one broadcasting bug.

Broadcasting and memory: the stride-0 trick

The stretch is virtual. Internally PyTorch uses a stride of 0 along the stretched dimension, so the same single row or column is reused for every position:

a = torch.ones(1, 4)
e = a.expand(3, 4)     # explicit broadcast
print(e.shape)         # (3, 4)
print(e.stride())      # (0, 1)  <- stride 0!
print(e.numel())       # 12 "virtual" elements
print(a.numel())       # 4 actual elements

The expanded tensor reports 12 elements but only 4 exist in memory. Stride 0 means "don't advance — reuse the same row." That's why broadcasting is fast even when the virtual expanded tensor would be huge.

Common mistakes and gotchas

Subtracting (3,) and (3, 1) expecting (3,). You get (3, 3). Make both shapes match exactly for element-wise.
Mismatched non-1 dimensions. (3,) + (5,) raises a RuntimeError at the non-singleton dimension.
Not knowing which dim stretched. Print .shape before and after every broadcast until the habit is automatic.
Forgetting to add size-1 dims. Per-channel ops need an explicit unsqueeze/None to line up the axes.

A three-line habit that catches every broadcasting bug

# 1. Print both shapes
print(a.shape, b.shape)
# 2. Align from the right on paper:  a: 3,1   b: 1,4  -> out 3,4 — is that what you want?
# 3. If not, unsqueeze or reshape to control which dim stretches

Print, align, decide. That habit catches broadcasting bugs before they turn into a wrong result downstream.

Recap

Align from the right, each pair must be equal or one must be 1, and size-1 dims stretch for free. The output takes the max of each pair. A mismatch raises a RuntimeError; the sneakier failure is a silent blow-up that returns the wrong shape — so print your shapes.

DEV Community: Wesam Khallaf — Author of PyTorch From Ground Up

Backpropagation by Hand: Two Layers, a Pen, and Then Autograd Agrees

The network, small enough to hold in your head

Forward pass, left to right, writing everything down

Backward pass, one node at a time, right to left

Now ask PyTorch the same question

The part that turns this into a debugging skill

Two things that may be confusing at first glance

Try it before you close the tab

More in this series

PyTorch Autograd Explained: What .backward() Actually Does

First, what a gradient actually is

Turning on the tape

The computation graph

Calling .backward()

Now do it on an actual network, by hand

What happens when the ReLU is closed

Three mechanics that catch nearly everyone

1. Only leaves keep their .grad

2. Gradients accumulate, they do not overwrite

3. no_grad and detach — switching the recorder off

One call, every parameter

The summary worth keeping

Try this before you close the tab

More in this series

Your CNN's Advantage Is One Assumption — and I Measured What Happens When It Breaks

The experiment

"You destroyed the data" — no, and this is the important part

What a convolution actually assumes

Try it yourself

Where this actually matters

The honest caveats

The takeaway

PyTorch `permute` vs `transpose`: What's the Difference (and the `reshape` Bug That Scrambles Your Images)

The short answer

transpose — swap two axes

permute — state the whole new order

The one that actually bites: permute is not reshape

Real code: the two reorders you'll actually write

Both of them break contiguity

Common mistakes and gotchas

Recap

What Does `keepdim` Do in PyTorch? (And the Silent Bug When You Forget It)

The short answer

First: what does dim do in a reduction?

So what does keepdim actually do?

Why size-1 matters: broadcasting back against the original

The trap: the same bug fails two completely different ways

Case 1 — a non-square tensor: it errors (the lucky case)

Case 2 — a square tensor: it runs, and it's silently wrong (the dangerous case)

A rule you can actually remember

Does keepdim work on max, min, and argmax too?

Common mistakes and gotchas

Recap

What Does `unsqueeze` Do in PyTorch? (And Why Your Model Keeps Asking For It)

The short answer

Why does PyTorch need size-1 dimensions at all?

The None shortcut (same thing, fewer characters)

What does squeeze do in PyTorch?

A worked example: the batch dimension

The other big use: lining up shapes for broadcasting

Are squeeze and unsqueeze expensive?

Common mistakes and gotchas

Recap

Reshape vs View in PyTorch: What's the Difference (and When `view()` Breaks)

The short answer

Why reshape vs view exists at all

What does "contiguous" mean in PyTorch?

When view() breaks (the error you googled)

Watch out: reshape doesn't guarantee a view

The -1 shortcut

A worked example: flatten before a linear layer

Common mistakes and gotchas

Recap

PyTorch Broadcasting Explained: The 3 Rules (and the Silent Bug That Bites Everyone)

The short answer

The core example

The 3 broadcasting rules

Worked examples

Why does broadcasting fail with a RuntimeError?

1. Only leaves keep their `.grad`

3. `no_grad` and `detach` — switching the recorder off

`transpose` — swap two axes

`permute` — state the whole new order

The one that actually bites: `permute` is not `reshape`

First: what does `dim` do in a reduction?

So what does `keepdim` actually do?

Does `keepdim` work on `max`, `min`, and `argmax` too?

The `None` shortcut (same thing, fewer characters)

What does `squeeze` do in PyTorch?

Are `squeeze` and `unsqueeze` expensive?

When `view()` breaks (the error you googled)

The `-1` shortcut