Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
A language model can write Python, explain quantum mechanics, and imitate Shakespeare.
Show it a screenshot of a production dashboard, however, and suddenly the central question becomes:
How does a transformer that was trained on text learn what a pixel means?
The naïve answer is: “Give the image to the LLM.”
That description hides almost all of the interesting engineering.
Modern multimodal systems are usually compositions of several models: a vision encoder turns pixels into vectors, a connector translates those vectors into something the language model understands, and the LLM then reasons over the resulting representation alongside ordinary text tokens.
That architectural trick has turned the transformer from a language architecture into something much closer to a general-purpose interface for heterogeneous data.
The evolution is worth understanding because it reveals a useful engineering pattern: you often do not need to retrain a giant model to give it a new sensory modality. You need a good representation and a sufficiently expressive interface between representations.
1. The basic mental model: pixels become tokens
Start with an ordinary LLM.
Its input looks conceptually like:
"The server returned HTTP 500. What should I check?"
|
v
tokenizer
|
v
[t1, t2, t3, ..., tn]
|
v
Transformer
|
v
answer
Everything is eventually represented as vectors.
Multimodal transformers exploit this fact.
An image is first converted into a sequence of vectors:
image
|
v
vision encoder
|
v
[v1, v2, v3, ..., vm]
|
v
multimodal connector
|
v
[z1, z2, z3, ..., zk]
|
+------ text tokens [t1, t2, ...]
|
v
LLM
|
v
answer
The important conceptual shift is this:
The LLM does not have to understand pixels directly. It only has to understand a representation produced by another model.
This is remarkably similar to a software interface.
The vision encoder is one service.
The LLM is another service.
The connector is the API contract.
That analogy becomes surprisingly literal once you look at actual architectures.
Why not just tokenize pixels?
You could imagine treating every RGB pixel as a token.
A 1024 x 1024 image has:
1024 * 1024 = 1,048,576 pixels
Even before considering RGB channels, that's about one million spatial locations.
Self-attention over one million tokens is absurdly expensive.
The computational cost of full self-attention is approximately:
O(n^2 * d)
where:
n = sequence length
d = hidden dimension
If you increase n by 100x, attention work increases by roughly:
100^2 = 10,000x
So multimodal systems need compression.
The first major engineering problem is therefore:
How do we turn a huge image into a manageable sequence of informative vectors?
2. The first breakthrough: vision itself becomes transformer-friendly
The most important precursor was the realization that transformers did not fundamentally require language.
In 2020, Alexey Dosovitskiy and colleagues at Google Research published the Vision Transformer, or ViT.
The idea was beautifully simple.
Take an image:
224 x 224 pixels
Divide it into patches, say:
16 x 16 pixels
You get:
224 / 16 = 14
224 / 16 = 14
or:
14 * 14 = 196 patches
Each patch becomes a vector.
Now the image has become a sequence:
[p1, p2, p3, ..., p196]
which looks suspiciously like a sequence of words.
The transformer does not care that these vectors originated from little squares of an image rather than pieces of text.
This was a profound simplification.
Instead of inventing a completely different architecture for vision, researchers could increasingly reuse the machinery that had made transformers dominant in NLP.
A useful way to think about ViT
Suppose each patch is represented by a vector:
x_i in R^d
A linear projection maps the flattened patch into the model dimension:
z_i = W x_i + b
The transformer then processes:
[z_1, z_2, ..., z_n]
with positional information added so that it knows that one patch came from the top-left and another from the bottom-right.
The architecture is now almost indistinguishable from a language transformer at the sequence-processing level.
That opened the door to something much bigger.
What if we could align these visual vectors with language?
That was the next step.
3. CLIP: teach images and language to share a coordinate system
In 2021, OpenAI researchers led by Alec Radford demonstrated an extremely influential idea with CLIP.
Instead of asking a model to classify images into a fixed set of labels, they trained image and text encoders to recognize which images and captions belong together.
Consider a batch containing:
Images:
I1, I2, I3, ..., IB
Captions:
C1, C2, C3, ..., CB
An image encoder produces:
v1, v2, ..., vB
and a text encoder produces:
t1, t2, ..., tB
The training objective tries to make:
similar(vi, ti)
large when they correspond, while making mismatched pairs less similar.
Conceptually:
shared embedding space
"a dog running"
*
/
/
*---*---------------- *
dog image "a red car"
The image and the corresponding textual description are pushed toward each other.
This matters because language becomes a way to refer to visual concepts.
You do not necessarily need a classifier saying:
class_id = 347
You can instead ask which textual concept is closest to the image.
CLIP trained on roughly 400 million image-text pairs collected from the internet.
That was an important historical moment because it demonstrated the economic power of weak supervision.
Instead of manually labeling:
dog
cat
airplane
...
humans had already generated huge quantities of text associated with images.
The internet itself became a gigantic, noisy multimodal dataset.
But CLIP still isn't an LLM
This distinction is important.
CLIP is fundamentally an alignment system.
You can use it to estimate:
similarity(image, "a dog")
but it is not naturally a conversational generative model.
There is no straightforward mechanism for:
User: What is unusual about this image?
Image: [photo]
Model: ...
For that, we need to connect visual representations to a generative language model.
And this is where the architecture gets interesting.
4. The multimodal bridge: three architectures that matter
There are several ways to connect vision and language.
The most useful way to understand them is as three increasingly integrated designs.
Architecture A: concatenate representations
Suppose the image produces:
[v1, v2, ..., vm]
and the prompt produces:
[t1, t2, ..., tn]
We can conceptually construct:
[v1, v2, ..., vm, t1, t2, ..., tn]
and feed the combined sequence into a transformer.
This is the simplest possible multimodal architecture.
But there is a problem.
The vision encoder and LLM were usually pretrained independently.
Their vector spaces have different semantics.
Giving the LLM a random-looking sequence of visual embeddings is a little like taking an internal data structure from one program and passing its memory layout directly into another program.
You need an interface.
Architecture B: a learned connector
This is the strategy behind systems such as BLIP-2.
The image encoder is frozen.
The LLM is frozen.
A relatively small learned component sits between them.
BLIP-2 calls this component the Q-Former, a querying transformer that learns to extract useful information from the image representation.
Very roughly:
image
|
v
frozen vision encoder
|
v
many visual features
|
v
Q-Former
|
v
small number of visual features
|
v
frozen LLM
One particularly elegant aspect is compression.
Imagine the vision encoder produces:
576 visual tokens
The Q-Former might learn a small set of query vectors:
32 learned queries
which interact with those visual features and extract the information most useful for language generation.
The LLM therefore doesn't have to process hundreds or thousands of raw visual features.
It receives a compact visual summary.
This is a beautiful example of representation bottleneck engineering.
Architecture C: inject visual information throughout the LLM
DeepMind's Flamingo took another approach.
Rather than simply converting an image into a pseudo-prefix, Flamingo introduced learned cross-attention mechanisms inside the language model.
Conceptually:
text states --------------------+
|
v
cross attention
^
|
image/video features -----------+
The language representation can selectively attend to visual features while generating text.
This becomes particularly useful for sequences such as:
image
text
image
text
image
question
answer
Flamingo was explicitly designed to deal with arbitrarily interleaved visual and textual inputs.
That is a very different interface from:
[one image] + [one prompt]
It starts looking like a genuine multimodal conversation.
Why this distinction matters
When evaluating multimodal architectures, ask:
Where exactly does information cross the modality boundary?
Possible answers include:
image -> prefix embeddings
image -> learned adapter -> LLM
image -> cross-attention inside LLM
image + text -> a single unified transformer
Those are materially different computational systems.
5. The math: multimodal attention is still just attention
Once the representations have been aligned, the underlying math becomes surprisingly familiar.
For ordinary self-attention:
Q = X W_Q
K = X W_K
V = X W_V
Attention(X) =
softmax(Q K^T / sqrt(d_k)) V
The key idea is that each token constructs:
query
key
value
and computes how much attention it should pay to other tokens.
For multimodal cross-attention, the queries can come from text while keys and values come from an image:
Q = X_text W_Q
K = X_image W_K
V = X_image W_V
CrossAttention(text, image) =
softmax(Q K^T / sqrt(d_k)) V
Now the question becomes:
Which parts of the image are relevant to the current textual state?
Suppose the image contains:
[dog] [grass] [tree] [red frisbee]
and the text state is trying to predict:
"The dog is holding a ..."
Its query representation can assign relatively high attention to visual features associated with the frisbee.
Conceptually:
text query
|
+------> dog 0.18
+------> grass 0.04
+------> tree 0.02
+------> frisbee 0.76
The model is not explicitly computing:
"this is a frisbee"
as a symbolic intermediate step.
It is learning a distributed mapping in which visual features that correlate with "frisbee" become useful when the linguistic context calls for them.
This is one reason multimodal models can feel surprisingly flexible.
6. The real engineering problem: token economics
The easiest way to underestimate multimodal inference is to look only at parameter count.
For inference economics, token count can matter enormously.
Suppose an LLM has a context containing:
2,000 text tokens
and an image representation contributes:
576 visual tokens
Then the effective sequence is roughly:
2,576 tokens
That is already a substantial increase.
Now imagine a system processing:
10 images
with 576 visual tokens each:
10 * 576 = 5,760 visual tokens
plus:
2,000 text tokens
giving:
7,760 tokens
The problem gets even worse when you process video.
Suppose:
1 frame every 0.5 seconds
10 minutes
That's:
20 frames/minute * 10 minutes = 200 frames
At 576 visual tokens per frame:
200 * 576 = 115,200 visual tokens
That is obviously not a reasonable way to feed a language model.
So practical multimodal systems aggressively compress.
They may:
sample fewer frames
pool spatial features
use learned queries
reduce image resolution
crop selectively
summarize visual content
process regions hierarchically
This produces an important systems principle:
Multimodal intelligence is often constrained by representation bandwidth rather than raw model size.
A back-of-the-envelope latency calculation
Suppose the language model processes 100 generated tokens/sec.
A response requiring:
1,000 text tokens
takes roughly:
1,000 / 100 = 10 seconds
But suppose adding the image representation effectively adds another 2,000 tokens of computation in a part of the pipeline with comparable throughput.
Now you are dealing with roughly:
3,000 / 100 = 30 seconds
That is an oversimplification because real systems have different prefill and decode costs, batching effects, GPU kernels, KV-cache behavior, and vision-encoder latency.
But the order-of-magnitude intuition is valuable:
Visual context isn't free.
And unlike ordinary text, a single image can silently introduce hundreds or thousands of model-side tokens.
7. Training strategy: don't throw away the LLM
One of the most interesting lessons from BLIP-2 is economic rather than architectural.
Imagine you already possess:
very strong vision encoder
+
very strong LLM
The naïve approach is:
train everything jointly
That can be horrendously expensive.
BLIP-2 instead demonstrated that a relatively lightweight bridge could connect frozen pretrained components.
Its reported result was striking: the system achieved strong vision-language performance while training far fewer parameters than the enormous language model underneath. In one reported comparison, BLIP-2 outperformed Flamingo 80B by 8.7 percentage points on zero-shot VQAv2 while using 54x fewer trainable parameters.
This is a recurring pattern in modern AI engineering:
giant pretrained model
^
|
small learned adapter
^
|
new modality
The adapter becomes the experimentation surface.
That changes the economics of model development.
Instead of repeatedly paying the cost of retraining a 70B-scale model, you can iterate on:
connector architecture
data mixture
alignment objective
instruction tuning
resolution
token compression
while keeping the expensive foundation models fixed.
The same idea appears in many areas of machine learning: reuse a highly capable representation and spend compute where it changes the interface.
8. From research prototype to developer system
This history eventually produced systems such as LLaVA, which paired a vision encoder with an LLM and used visual instruction tuning to make the resulting model behave like an interactive assistant.
The crucial shift here is not merely "the model can classify images."
It is:
image + instruction
|
v
reasoning-oriented language generation
That means developers can expose multimodality through interfaces that look almost exactly like ordinary LLM APIs.
For example:
response = model.generate(
[
{"type": "image", "data": screenshot},
{
"type": "text",
"data": "Why is this Kubernetes deployment failing?"
}
]
)
The interesting part of such a system is not the API.
Internally, the pipeline might look something like:
screenshot
|
+--> resize / crop
|
+--> vision encoder
|
+--> visual embeddings
|
+--> projection / Q-Former / cross-attention
|
v
language-model context
|
v
autoregressive decode
Now consider what the developer actually gets.
The model might interpret:
screenshot
|
v
visual representation
|
+-------+--------+
| |
text reasoning
| |
+-------+--------+
|
v
"Your readiness probe is
querying /healthz on port 8080,
but the container exposes 8081."
This is where multimodal models become particularly interesting for software engineering.
The modality is no longer the product.
It is an additional input channel to the same reasoning engine.
A screenshot can become context.
A PDF page can become context.
A chart can become context.
A whiteboard photo can become context.
A video frame can become context.
The transformer is increasingly functioning as a general context processor, rather than merely a text processor.
9. Where multimodal transformers still struggle
The weaknesses are revealing because they show what the architecture actually learned.
Counting
Vision-language models can often identify objects while being surprisingly unreliable at exact counting.
A model might correctly say:
"There are several apples."
while getting:
7 apples
wrong.
The continuous visual representation is excellent for semantic recognition but does not automatically produce a discrete symbolic counting process.
Spatial reasoning
Consider:
Is the red box left of the blue box?
Many systems can answer this.
Now make it harder:
What object is two positions to the left
and one row below the triangle?
The model suddenly has to preserve precise spatial relationships rather than merely recognize concepts.
OCR and tiny details
Image understanding is heavily resolution-dependent.
Suppose the original screenshot contains:
HTTP 500
but the relevant text occupies only 12 pixels in height.
If preprocessing downsamples it aggressively, the information may simply disappear.
No amount of clever language reasoning can recover information that the visual encoder never encoded.
Hallucination
This is perhaps the most interesting failure.
Suppose the image is ambiguous.
The language model has a powerful prior about what typically appears in such scenes.
It may effectively reason:
visual evidence: weak
language prior: strong
=> confident answer
That produces a fluent hallucination.
The multimodal model therefore inherits a deep problem from generative language modeling:
probability is not the same thing as observation.
This is one reason evaluation of multimodal systems cannot rely only on whether answers sound reasonable.
10. The deeper architectural lesson
There is a broader lesson hiding behind all of this.
A transformer does not fundamentally care whether an input token came from:
a word
an image patch
a spectrogram
a video frame
a sensor
a code execution trace
What it needs is a representation that can participate in the attention computation.
The hard problem is therefore representation interoperability.
You can think of the modern multimodal stack as:
modality-specific worlds
image audio text video
| | | |
v v v v
vision audio tokenizer video
encoder encoder / embedder encoder
| | | |
+------------+------------+------------+
|
v
shared / bridged
representation
|
v
Transformer
|
v
reasoning + generation
This architecture explains why the progress in multimodal AI has been so rapid.
We did not need to invent an entirely new paradigm.
We discovered that transformers could become the common computational substrate, while specialized encoders handled the raw structure of different modalities.
The important research question then became:
What information should each modality expose to the shared model, and how much of it?
That is fundamentally an information bottleneck problem.
Too little information:
image -> "there is a dog"
and useful detail disappears.
Too much information:
image -> 10,000 visual tokens
and computation, memory, latency, and cost explode.
The interesting architectures sit between those extremes.
11. What developers should take away
When evaluating a multimodal model, don't stop at:
"What vision model does it use?"
Ask these instead:
How many visual tokens does one image produce?
How does the connector map visual features into the LLM?
Are visual features inserted once or accessed through cross-attention?
Can the model consume multiple images?
Can it consume interleaved image/text sequences?
How does it handle high-resolution images?
How does it sample video?
Does the visual encoder remain frozen?
Where does most inference compute go?
What happens when the visual evidence is ambiguous?
Those questions tell you far more about the system's real behavior.
And there is an especially useful systems intuition:
multimodal quality
!=
LLM quality + vision quality
The interface between them matters enormously.
A brilliant vision encoder with a poor connector can produce a mediocre multimodal model.
A powerful LLM with insufficient visual bandwidth can miss information.
A large context window can become economically useless if every image expands into thousands of expensive tokens.
And a model that recognizes visual concepts beautifully can still fail when asked to perform exact symbolic reasoning over those concepts.
12. Conclusion: the LLM is becoming the center of a sensor network
CLIP showed that images and language could inhabit a shared semantic space.
Flamingo showed that visual inputs could be woven into a generative language model.
BLIP-2 showed that this connection could be surprisingly efficient by learning a relatively small bridge between powerful frozen models.
LLaVA and related systems pushed the idea toward practical instruction-following assistants.
The trajectory is interesting because it suggests that "multimodal LLM" may eventually sound as redundant as "networked computer."
The language model is increasingly just the reasoning and generation core.
Around it we can attach encoders for:
vision
audio
video
documents
3D geometry
robotic sensors
The transformer becomes the place where these representations interact.
For developers, that means the interesting design space is no longer just prompt engineering or model size. It is also representation engineering: deciding what information crosses each modality boundary, at what resolution, with what compression, and at what computational cost.
And that may be the more important shift.
The future multimodal system might not look like:
LLM + vision
It may look more like:
sensors
|
+-----------+-----------+
| | |
vision audio video
| | |
+-----------+-----------+
|
representation
|
transformer
|
+-----------+-----------+
| | |
reasoning tools actions
At that point, "language model" starts to look like a historical name for a system whose real job is much broader:
take heterogeneous representations, put them into a common computational space, reason over them, and produce useful actions or outputs.
The interesting question for developers is no longer whether LLMs can see.
It is:
What other kinds of information can we teach a transformer to understand by giving it the right interface?
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
HexmosTech
/
LiveReview
Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
The exact math, not a black box
Visualize blast radius at a glance
Every factor that feeds the score
Here's the goal:
- A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
- A 300-line UI change in one file, fully covered by…
Click below to try LiveReview with your codebase:
Top comments (0)