A model that only ever read text now looks at a photo of your fridge and suggests a recipe. Nothing about "predicting the next token" obviously explains how it can see. The bridge is simpler and more elegant than it looks.
Multimodal AI — models that handle images, audio, and text together — feels like a different kind of technology. It isn't. It's the same machinery with one clever addition. Here's the idea.
The core trick: everything becomes the same kind of thing
A language model reasons over embeddings — vectors that capture meaning — derived from text tokens. The entire breakthrough of multimodal models is this: you can turn an image (or audio) into vectors in that same space.
An image gets sliced into patches, and each patch is encoded into a vector, much as a sentence is split into tokens and embedded. Once an image is a sequence of vectors living in a shared space with text, the model's attention mechanism can treat words and image-patches as one combined sequence — letting a word attend to a region of an image and vice versa. "See" isn't a new faculty bolted on; it's the old faculty fed a new kind of input that's been translated into the language the model already speaks: vectors.
That's the whole conceptual leap. Different senses, one representational space. It's the same "meaning as geometry" principle that powers embeddings and semantic search, extended past text — a unifying idea I find genuinely elegant and lean on across the systems I build.
Why a shared space is so powerful
Once images and text live in the same space, capabilities fall out almost for free:
- Ask questions about an image — the model reasons over picture and prompt together.
- Search images with text (and vice versa) — because a photo of a dog and the word "dog" land near each other, cross-modal search is just nearest-neighbor search across modalities.
- Ground language in the visual world — describe, caption, or reason about what's actually in a scene rather than what's merely plausible.
The same recipe extends to audio, video, and beyond. Encode the modality into the shared vector space and the model can reason across it.
Where it gets hard
The elegance hides real difficulty. Aligning the spaces well — so an image and its description truly land near each other — takes enormous, carefully-paired training data. And the failure modes compound: a multimodal model can misread an image and then confidently describe what it thinks it saw, layering a perception error under a hallucination. Trusting the output means accounting for both.
Why it matters
Multimodal is where a lot of the field is heading, because the world isn't made of text. Systems that can perceive and reason across images, sound, and language unlock everything from document understanding to real-world visual tasks. And once you see it as "translate every modality into one shared space of meaning," it stops being mysterious and starts being buildable.
The senses are different; the representation is one. That's the whole idea. More of how I think about it at www.divyakush.com.
Related reading
- Embeddings and semantic search, from the ground up — the shared vector space that makes multimodal possible.
- Tokenization: why an LLM can't count the R's in strawberry — how text becomes tokens, and why it matters.
- Vector databases: how semantic search scales — nearest-neighbor search at scale.
Divyakush Punjabi · Full-Stack & AI Engineer
Portfolio · GitHub · LinkedIn
Top comments (0)