The transformer was built for sequences of words. The Vision Transformer's one radical move is to make an image look like a sentence — and then run the exact same encoder from language on it, with no convolutions anywhere. That sounds like a hack, and in a sense it is, but it works, and building a live visualizer for it made the whole idea click. Here's the pipeline, end to end.
An image is worth 16×16 words
A ViT never sees a whole image. It chops the image into a grid of fixed, non-overlapping patches — classically 16×16 pixels, which is where the paper's title comes from. A 224×224 image at patch size 16 gives a 14×14 grid, so N = 196 patches. Each patch's 16·16·3 = 768 pixel values get flattened into a vector and linearly projected to a D-dimensional token. That projection has a slick implementation: it's literally a Conv2d(3, D, kernel=16, stride=16) — one conv whose stride equals its kernel, so it produces exactly one embedding per patch.
Add a [CLS] token and positional embeddings
Two more pieces before the encoder. First, prepend one extra learnable [CLS] token, so the sequence length is L = N + 1 = 197; its final state after the encoder is the whole-image summary the classifier reads. Second, a bag of patches has no inherent order — the model doesn't know which patch sat top-left — so you add a positional embedding to each token to encode its slot. The token that enters the encoder is therefore: flatten → linear projection → + positional embedding.
patch (16×16×3) → flatten → 768-vector
→ linear projection W·x → D-dim token
→ + positional embedding[slot]
→ the token fed to the encoder
A completely standard encoder
Here's the part that surprised me: after patchifying, there is nothing image-specific left. The N+1 tokens go through the exact multi-head self-attention + MLP blocks from language — the same transformer encoder, unchanged. And because it's self-attention, every patch attends to every other patch from layer one. A ViT has a global receptive field immediately, where a CNN grows its view slowly through stacked local filters, seeing only a small neighbourhood early on.
The catch is inductive bias
That global reach isn't free. A convolution bakes in two priors: locality (nearby pixels relate) and translation-equivariance (a cat is a cat wherever it appears). ViT throws both away — it must learn those regularities from data. That makes it data-hungry: a plain ViT only beats strong CNNs after pre-training on something like ~300M images (JFT-300M); trained on ImageNet alone it underperforms. DeiT later fixed this with heavy augmentation and distillation, but the lesson stands — a CNN's built-in bias is a head start ViT has to earn.
Patch size is the knob that matters
Self-attention is O(L²) in the token count. So the patch size is the single most important dial: smaller patches → more tokens → finer detail, but a quadratic blow-up in compute. Halve the patch side and you roughly 4× the tokens, which ~16×'s the attention cost. That's the whole tension in one number — ViT-B/16 versus ViT-B/32 is exactly this trade of resolution against cost.
And the payoff of the global receptive field is real: because [CLS] attends to every patch from the first layer, a ViT can bind a distant pair of patches — an ear here, a tail there — in one step, where a CNN needs many stacked layers to grow a window wide enough to see both. That's why ViTs, once they have the data, tend to model long-range relationships more directly than convolutions do.
Building the visualizer, the mental model settled into three moves: patchify (image → grid of tokens, watch the count and O(L²) cost change with patch size), embed (one patch → flatten → projected token → + position), and attend (click a patch and see which others it looks at, plus what [CLS] pools from). Turn an image into a sequence, tell it where each piece sat, and let every piece talk to every other — that's a ViT. Split a live image into patches and watch tokens and attention resolve in the browser:
Top comments (0)