Your CNN's Advantage Is One Assumption — and I Measured What Happens When It Breaks

A small convolutional network beats a plain flatten-and-feed-it-forward network by 7.0 points on CIFAR-10. That's convolutions, pooling, normalisation and skip connections doing honest work.

Then I shuffled the rows of every image, destroying no information at all, and that 7.0-point margin fell to 0.3.

Same architecture. Same data, in a strict sense I'll defend in a moment. Almost the entire advantage, gone.

The experiment

Take one fixed permutation of the 32 row indices. Apply it to every image in the training set and every image in the test set — the same permutation, every time.

import torch

g = torch.Generator().manual_seed(1234)
row_perm = torch.randperm(32, generator=g)

def shuffle_rows(x):        # x: (C, H, W)
    return x[:, row_perm, :]

print(row_perm[:8].tolist())   # [15, 9, 8, 1, 4, 12, 30, 7]

That's the whole intervention. Then train two models twice each — once on natural images, once on shuffled ones:

Model	Params	Natural rows	Shuffled rows
Flatten → 512 → 10 (MLP)	1,578,506	51.4%	51.7%
Small CNN	94,538	58.4%	52.0%
CNN's margin		+7.0 pts	+0.3 pts

The baseline is a real fully-connected network, not a single linear layer — Flatten → Linear(3072, 512) → ReLU → Linear(512, 10). It has the capacity to learn anything the CNN can; what it lacks is any reason to look at pixels near each other.

Two things in that table are worth sitting with. The CNN wins the natural case with sixteen times fewer parameters — that's the prior paying for itself. And in the shuffled case it doesn't just lose its lead; it drops 6.4 points in absolute terms, down to roughly where the linear model already was.

"You destroyed the data" — no, and this is the important part

This is the objection everyone raises, so let's take it seriously, because the experiment is worthless if the objection holds.

A fixed permutation is a bijection. Nothing is added, nothing is removed, nothing is averaged or blurred:

img = torch.arange(3*32*32, dtype=torch.float32).reshape(3, 32, 32)
sh  = shuffle_rows(img)

print(sh.shape)                                     # torch.Size([3, 32, 32])

# every value still present, exactly once
print(torch.equal(sh.flatten().sort().values,
                  img.flatten().sort().values))     # True

# and it's perfectly reversible
inv = torch.empty_like(row_perm)
inv[row_perm] = torch.arange(32)
print(torch.equal(sh[:, inv, :], img))              # True

Every pixel is still there, with its original value. The transformation is invertible, so no information has been lost in any sense that information theory would recognise. What changed is which pixels sit next to which.

And here's the control experiment that settles it: the fully-connected model scores the same either way — 51.4% natural, 51.7% shuffled, a difference well inside run-to-run noise. If the shuffle had damaged the data, the linear model would have suffered too. It didn't, because it never used the layout in the first place — after flatten(), position 400 is just position 400, and a fixed permutation of the input columns is something the first layer absorbs by permuting its own weights. The network is free to relearn the identical function; only the column labels moved.

So the shuffle removes exactly one thing: the usefulness of the assumption that neighbouring pixels are related. And that assumption turns out to be worth 6.7 of the CNN's 7.0 points.

Which is the uncomfortable version of the finding: the convolutions, the pooling, the normalisation — all of it — were converting one true fact about photographs into 7 points of accuracy. Take the fact away and the machinery has nothing left to convert.

What a convolution actually assumes

Written out, a conv layer makes three claims about your data, none of which are claims about tensors:

Locality. A 3×3 kernel only ever sees a 3×3 neighbourhood. This is a bet that meaningful patterns are local — that to recognise an edge you need nearby pixels and not distant ones. True of photographs. Not true of a shuffled photograph, where the pixels that formed an edge are now scattered across the image.

Weight sharing. The same kernel slides over every position, so a feature detected in the top-left uses identical weights to the same feature bottom-right. This is a bet that what a pattern is doesn't depend on where it is. It's also where the parameter savings come from — and it's why the shuffled case doesn't merely lose the advantage but wastes capacity, since the kernel is now sharing weights across positions that have nothing in common.

Hierarchy. Stacked layers assume small patterns compose into larger ones: edges into corners, corners into shapes. A shuffled image has edges nowhere, so there is nothing to compose.

All three are statements about the world, not the maths. When they hold, they're enormously valuable — a prior that good is worth more than a lot of data. When they don't hold, depth doesn't manufacture them.

Try it yourself

Roughly this, on CPU, in a few minutes:

import torch, torchvision
from torchvision import transforms

g = torch.Generator().manual_seed(0)
row_perm = torch.randperm(32, generator=g)

norm = transforms.Normalize([0.5] * 3, [0.5] * 3)
base = [transforms.ToTensor(), norm]
shuffled = base + [transforms.Lambda(lambda x: x[:, row_perm, :])]

train_nat = torchvision.datasets.CIFAR10(root="./data", train=True, download=True,
                                         transform=transforms.Compose(base))
train_shuf = torchvision.datasets.CIFAR10(root="./data", train=True, download=True,
                                          transform=transforms.Compose(shuffled))
# ...and the same two for train=False

Then train any small CNN and any Flatten → Linear → ReLU → Linear on each of the four loaders. Four runs. Watch the CNN's lead evaporate while the fully-connected model shrugs.

Two things to keep honest while you do it: build the permutation once, outside the transform, or you'll get a different shuffle per image, which really does destroy information and proves nothing. And use the same seed and epoch count across all four runs, or you're measuring your own variance.

My run, for the record: torch 2.13.0 on CPU, manual_seed(0), Adam at 1e-3, batch 128 train / 256 eval, 5 epochs, normalised to mean 0.5 / std 0.5, row permutation seeded 1234.

Where this actually matters

This isn't a party trick. It's the question to ask before choosing any architecture: what is this thing assuming about my data, and is the assumption true?

• Tabular data. Columns have no spatial relationship — age next to postcode next to income is an arbitrary ordering you could permute at will. A CNN over columns is the shuffled experiment, permanently. This is a large part of why gradient-boosted trees still win on tabular problems.
• Spectrograms and time series. These do have local structure along at least one axis, so a 1-D convolution is a real prior rather than a fashion choice. Neighbouring time steps are genuinely related.
• Channel order. Permuting the channel axis of an image is not the same operation and is nearly harmless, because a conv kernel spans all channels at once — there's no locality assumption along that axis to break. Worth knowing, because it's the first thing people try to use as a counter-example.
• Attention. Self-attention makes no locality assumption at all, which is exactly why transformers need positional encoding bolted on and why they need more data to reach the same place on images. Fewer assumptions is not automatically better; it means less free knowledge and more required evidence.

The honest caveats

One dataset, one small architecture, one seed, at 32×32, five epochs — nobody's state of the art. The direction of this result is robust and it's the standard inductive-bias argument that the literature has made for years. The specific numbers are mine and yours will differ — the point of running it isn't the decimal places, it's watching a 7-point advantage turn into a rounding error while the data stays intact.

The takeaway

Architecture isn't magic and it isn't a leaderboard. It's a bet about the structure of your data, made before training starts. A convolution bets that nearby things belong together. On photographs that bet pays enormously. Break its truth without touching a single pixel value and the same architecture is worth about a third of a point.

So when someone asks why your model isn't learning, the useful question often isn't "should I go deeper" — it's whether the architecture's assumptions were ever true of your data.

---

This experiment is from Chapter 8 of **PyTorch From Ground Up, Volume 2, which I'm writing now. The whole book works this way: no solution gets introduced until you've watched the failure it fixes — no convolution before a linear layer fails, no skip connection before a 56-layer network loses to a 20-layer one.

Run it yourself: open the Chapter 8 notebook in Colab — every code block from the chapter, in order, CPU-only, nothing to install. The four-model comparison is the slow cell, about 15 minutes.

Volume 2 isn't finished yet. Volume 1 — Foundations, the tensors-and-training-loop book underneath all of this — is out, and 8 of its chapters are free with no email required.

What's an architecture choice that quietly bought you nothing? I collect these — the ones that surprise me end up as experiments.