Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
A language model does not know what a pixel is.
Give a transformer a JPEG and, at least conceptually, you cannot simply say:
image.jpg -> LLM
The model expects a sequence of vectors. Language gets this for free: a tokenizer turns text into discrete token IDs, those IDs become embeddings, and the transformer processes the resulting sequence.
Images are different.
A 1024 x 1024 image contains more than a million pixels. Feeding those pixels directly into a transformer would produce a sequence so enormous that ordinary attention becomes absurdly expensive.
So multimodal systems perform a crucial act of engineering:
image
|
v
visual representation
|
v
image tokens / visual tokens
|
v
language model
The interesting part is that "image tokenization" is not one algorithm. It is a whole design space involving patch size, learned visual encoders, compression, projection into the language model's embedding space, and sometimes aggressive token pruning.
This is one of those pieces of multimodal architecture that looks like plumbing until you realize that it largely determines what the model can see, how much it costs to run, and even what kinds of visual reasoning are possible.
1. The core problem: an image is ridiculously large compared with text
Consider a normal RGB image:
1024 x 1024 x 3 = 3,145,728 values
Even before doing anything intelligent, you have roughly 3.1 million scalar values.
Compare that with:
"A dog is sitting on the grass."
~ 7-10 language tokens, depending on tokenizer
Of course, pixels are not equivalent to language tokens. The point is sequence length.
Transformers process sequences. For ordinary dense self-attention, the expensive part scales approximately as:
attention cost ~ O(N^2)
where N is the sequence length.
If we naïvely gave each pixel its own token:
N = 1024 x 1024 = 1,048,576
Then the number of pairwise attention relationships would be approximately:
N^2
= 1,048,576^2
~ 1.1e12
That is over a trillion pairwise interactions per layer.
This is why "just make the pixels tokens" is not a particularly useful strategy.
There is also a deeper issue: neighboring pixels are highly redundant. A 16 x 16 patch of sky contains 256 pixels, but the semantic information in those pixels is nowhere near 256 times greater than the information in a compact representation of the patch.
So the first major trick is:
Compress local visual structure before asking the language model to reason about it.
That observation leads directly to image patches.
2. The historical trick: turn a 2D image into a sequence
An important early experiment was OpenAI's iGPT, reported in 2020 by Mark Chen, Alec Radford, Ilya Sutskever and colleagues.
The idea was surprisingly literal: take an image, flatten it into a sequence, and train a GPT-style transformer to predict the next visual symbol. The work demonstrated that a transformer originally designed around sequential prediction could learn meaningful image representations, even without explicitly building in conventional vision machinery. (OpenAI)
But iGPT also exposed the fundamental scaling problem. Full-resolution pixels create enormous sequences.
The paper explicitly describes having to reduce image resolution and compress color information because transformer memory requirements grow quadratically with context length. (OpenAI)
Then came a much more influential simplification.
In 2020, Alexey Dosovitskiy and collaborators at Google Research presented the Vision Transformer, or ViT.
Instead of treating every pixel as a token, ViT divides the image into fixed-size square patches and treats each patch like a token. For the canonical 224 x 224 image with 16 x 16 patches:
224 / 16 = 14
14 x 14 = 196 patches
Suddenly:
1,048,576 pixels -> 196 visual tokens
for a 1024 x 1024 image, if we used the same 16 x 16 patch size we would get:
1024 / 16 = 64
64 x 64 = 4096 tokens
That's still large, but vastly more manageable than a million. ViT showed that a pure transformer operating on image patches could perform extremely well, rather than requiring convolutions to provide the primary visual machinery. (arXiv)
That patch idea became one of the foundations of modern vision-language systems.
3. What exactly is an image token?
A useful mental model is:
image
|
+-- patch 1
+-- patch 2
+-- patch 3
...
+-- patch N
|
v
linear projection
|
v
embedding vector
Suppose the image is:
224 x 224
and the patch size is:
16 x 16
Then there are:
(224 / 16) x (224 / 16)
= 14 x 14
= 196 patches
Each patch contains:
16 x 16 x 3 = 768
raw RGB values.
A simple patch embedding can be represented as:
z_i = x_i W + b
where:
x_i = flattened pixels of patch i
W = learned projection matrix
b = bias
z_i = resulting embedding
If the model's hidden dimension is, say:
D = 768
then:
x_i : 768 values
z_i : 768-dimensional vector
So the transformer no longer sees a little square of raw pixels. It sees a learned representation of that square.
That distinction matters.
Calling the resulting vector an "image token" can be slightly misleading because it is not necessarily equivalent to a language token like:
"cat"
It is closer to:
a learned vector representing some region of the visual input.
And that representation becomes progressively more semantic as it passes through the vision encoder.
Early layers might encode things resembling:
edges
textures
color transitions
Later layers may represent:
eyes
faces
objects
text
spatial relationships
The precise interpretation is distributed rather than cleanly localized, but the hierarchy is useful.
A modern multimodal model therefore often looks something like:
image
|
v
Vision Transformer
|
v
visual feature sequence
|
v
projection / connector
|
v
LLM-compatible embeddings
|
v
language transformer
The LLM never has to directly manipulate millions of pixels.
4. The token budget is where the real engineering begins
Here is the part developers tend to underestimate.
Patch size determines token count.
For an image of width W, height H, and square patch size P:
tokens = (W / P) x (H / P)
ignoring padding and other architecture-specific details.
Because both dimensions scale inversely with P, token count scales roughly as:
tokens ~ 1 / P^2
That square is brutal.
Compare 8 x 8 patches with 16 x 16 patches:
P = 8
tokens = (224 / 8)^2
= 28^2
= 784
versus:
P = 16
tokens = (224 / 16)^2
= 14^2
= 196
So halving the patch size increases the number of tokens by:
784 / 196 = 4x
And dense attention then makes pairwise interactions grow by roughly another square:
4x more tokens
=> ~16x more attention-pair work
This is the central tradeoff in image tokenization:
smaller patches
|
+-- more spatial detail
+-- better access to small objects/text
+-- more tokens
+-- more compute
+-- more KV-cache / activation pressure
Consider a document image.
At 1024 x 1024:
P = 32
tokens = 32 x 32 = 1024
P = 16
tokens = 64 x 64 = 4096
P = 8
tokens = 128 x 128 = 16,384
Now imagine appending the user's textual prompt:
4096 image tokens
+ 100 text tokens
The image dominates the context.
This is why multimodal context windows can be deceptive. A model advertised with a very large context window may still face a practical distinction between:
100,000 text tokens
and:
100,000 multimodal tokens
because the computational and memory characteristics of generating and attending over those visual tokens can be very different.
The economics follows directly from the math.
Suppose one request contains:
4 images
1024 x 1024
16 x 16 patches
Each image contributes:
4096 tokens
So:
4 x 4096 = 16,384 visual tokens
before the model even sees the user's text.
Now imagine serving 100 requests per second:
16,384 x 100
= 1,638,400 visual tokens / second
That is why image preprocessing, token compression, batching strategy, and sequence-length distributions become production concerns rather than academic details.
5. The surprising part: the image tokens usually do not go directly into the LLM
Suppose your vision encoder produces:
N visual tokens
x
D_v dimensions
while your LLM expects embeddings of:
D_l dimensions
Those dimensions probably do not match.
So multimodal architectures need a bridge.
Conceptually:
Vision encoder:
[N, D_v]
|
| projection / connector
v
LLM:
[N', D_l]
There are several ways to construct this bridge.
A simple approach is a learned projection:
z_llm = z_vision W
where:
z_vision : D_v
W : D_v x D_l
z_llm : D_l
The language model can then consume those vectors alongside textual embeddings.
This sounds trivial, but it is one of the most important architectural boundaries in multimodal systems:
vision representation
|
| semantic interface
v
language representation
In other words, the language model does not necessarily need to know how the image encoder works. It needs an interface through which useful visual information can enter its computation.
DeepMind's Flamingo, introduced in 2022 by Jean-Baptiste Alayrac and collaborators, made this interface idea particularly explicit. Rather than simply pretending that visual features are ordinary text tokens, Flamingo used a Perceiver-style mechanism to compress visual information and gated cross-attention layers to let the language model selectively attend to the visual stream. (arXiv)
That distinction matters because there is a fundamental design question:
Should we force the image into the language model's token stream, or should we give the language model a separate visual memory that it can query?
Those lead to somewhat different scaling properties.
A useful conceptual comparison is:
Strategy A: visual tokens become part of the sequence
[text][image][image][image][image][text]
Strategy B: language queries a visual representation
[text] ----\
[text] -----+--> cross-attention --> visual memory
[text] ----/
The second approach can be substantially more flexible when the raw visual representation is huge.
6. Why "more image tokens" is not automatically better
It is tempting to conclude:
more patches = more information = better model
That is not generally true.
Imagine an image containing:
sky
grass
mountain
person
small road sign
The sky might occupy 40% of the image.
Do we really need thousands of equally important tokens describing the sky?
Probably not.
What we actually care about is information density.
One possible strategy is therefore:
easy / redundant region
|
v
few tokens
complex / informative region
|
v
many tokens
This leads to adaptive or hierarchical tokenization.
Conceptually:
coarse grid
|
+-- boring region ------> stop
|
+-- complex region -----> split
|
+-- split again
A document is a perfect example.
Suppose you photograph a page with:
large blank margins
header
two columns of dense text
small diagram
Uniformly allocating the same visual resolution everywhere is wasteful.
You would ideally spend tokens on:
tiny text
fine diagrams
tables
logos
faces
and fewer tokens on:
blank paper
uniform walls
sky
large flat backgrounds
This is increasingly important as multimodal models move from fixed-image benchmarks toward actual applications.
Consider a coding assistant looking at a screenshot.
It may not care about 95% of the pixels.
It cares intensely about:
error message
line number
function name
small icon
button state
Tokenization therefore becomes an information-allocation problem.
You can think of the ideal tokenizer as trying to solve something like:
maximize useful visual information
subject to a token budget
That's a much more interesting problem than simply choosing a patch size.
7. Operations: image tokens are effectively compute currency
Once you look at multimodal inference from an infrastructure perspective, visual tokenization starts looking like a pricing problem.
Suppose your application processes:
1000 requests / second
and each request contains:
2 images
~1000 visual tokens per image
Then the system processes:
2 x 1000 x 1000
= 2,000,000 visual tokens / second
At large scale, that visual token volume becomes one of the main determinants of:
GPU memory
attention FLOPs
prefill latency
batching efficiency
throughput
cost per request
The prefill phase is particularly relevant.
Generating the answer token by token is only half the story. Before generation begins, the model has to process the input context.
If the input contains:
20,000 visual tokens
+ 500 text tokens
then the LLM has to ingest a 20,500-token context before it can start answering.
This produces an important practical asymmetry:
small textual question
+
huge image representation
=
expensive request
even if the eventual answer is only:
"Yes, that's a bar chart."
This is why production multimodal systems often care about things that sound almost mundane:
image resizing
aspect-ratio handling
token pooling
visual feature caching
batching
early compression
dynamic resolution
They are all attempts to control the number and usefulness of visual tokens reaching the expensive language-model computation.
There is also a subtle systems lesson here:
The expensive object is often not the image. It is the image after you've converted it into something the transformer must process.
A 500 KB JPEG and a 5 MB JPEG might become nearly identical computational workloads after preprocessing.
Meanwhile, two visually similar images can have dramatically different costs if one produces:
400 visual tokens
and another produces:
4000 visual tokens
The file size on disk is therefore a poor proxy for multimodal inference cost.
8. The developer mental model
When you encounter a new multimodal model, it is useful to stop thinking:
"How does the LLM look at images?"
and instead ask five more concrete questions:
1. What is the vision encoder?
2. What constitutes one visual token?
3. How many visual tokens does an image produce?
4. How are visual features mapped into the language model?
5. What happens when there are multiple images?
A simplified architecture might be:
IMAGE
|
v
+------------------+
| Vision Encoder |
| |
| patchify |
| self-attention |
| feature building |
+------------------+
|
v
[v1 v2 ... vN]
|
v
+------------------+
| Connector |
| projection / |
| resampler / |
| cross-attention |
+------------------+
|
v
[z1 z2 ... zM]
|
+-------+-------+
| |
TEXT TOKENS VISUAL TOKENS
| |
+-------+-------+
|
v
LLM
|
v
OUTPUT
The important architectural variable is often M, not merely the number of pixels in the original image.
That number tells you how much visual information the language model actually has to reason over.
And this is where a seemingly simple preprocessing choice turns into a first-order model-design decision.
Conclusion: image tokenization is the bottleneck between seeing and reasoning
The path from an image to an LLM is fundamentally a compression pipeline:
millions of pixels
|
v
thousands of patches
|
v
hundreds/thousands of visual embeddings
|
v
compact multimodal representation
|
v
language-model reasoning
The engineering challenge is to compress aggressively enough to make inference affordable without throwing away the visual details that matter.
A tiny patch can preserve the lettering on a road sign but explode the token count.
A huge patch is computationally cheap but may turn that road sign into unreadable noise.
Uniform tokenization is simple, but it wastes capacity on uninteresting regions.
Aggressive compression is cheap, but may destroy spatial detail that the language model needs.
So the real optimization target is not:
minimum number of image tokens
It is closer to:
maximum useful information per visual token
That framing has a surprisingly broad consequence. Better multimodal models may not primarily come from making the LLM larger. They may come from building better ways of deciding which parts of an image deserve representation, at what resolution, and in what form.
That makes image tokenization less like an input-formatting trick and more like the visual equivalent of the tokenizer itself.
The next time you see a model that claims to understand a 4K screenshot, a scanned PDF, or twelve images in one prompt, the interesting question is not merely "How big is its context window?"
It is:
How many visual tokens did it actually have to spend to understand what mattered?
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
Try LiveReview on your codebase:
Top comments (0)