TL;DR — Multimodal models look like a single system in the demo but are actually a pipeline of independently-trained encoders bolted onto a shared decoder. Each encoder carries its own training distribution, and in production the whole system is bottlenecked by whichever modality has the narrowest one — usually vision or audio, not text. Aggregate accuracy hides this because failures are confident and silent, not error-shaped.
Ask a multimodal model to describe a stock photo of a golden retriever on a beach and it nails it every time. Ask the same model to read a cropped screenshot of a spreadsheet, or transcribe a phone call with crosstalk, or tell you what happened between frame 400 and frame 500 of a security video, and the failure modes get strange fast. Not "I don't know" strange — confidently wrong strange. The model doesn't hedge. It just describes something plausible that isn't there.
The instinct is to blame the model. The more useful frame is architectural: a multimodal model isn't one model. It's a set of independently-trained encoders — one per modality — projected into a shared embedding space and handed off to a decoder that was mostly trained on text. Each encoder has its own training distribution, its own capacity, its own blind spots. The fused system inherits all of them, and in production, quality is bottlenecked by whichever encoder has the narrowest distribution. That's almost never the text side.
The one-model illusion
Text models got good because text training data is absurdly broad — decades of the open web, code, books, forums, transcripts. Vision and audio encoders don't have that. The vision towers underneath most multimodal systems trace back to contrastive image-text pretraining on web images paired with alt text and captions: product photos, news images, stock photography, social media posts. That corpus is enormous, but it's also aesthetically and semantically narrow. It's full of things people photograph and caption. It is not full of things people generate as byproducts of work: dense tables, thermal camera frames, low-light warehouse footage, scanned forms, dashboards, X-rays, satellite tiles.
When you feed one of those into the vision encoder, you're not asking the model a hard question. You're asking it a question from outside its support. The projection layer still produces an embedding — it has no mechanism to say "I've never seen anything like this" — and the decoder, trained to always produce fluent text, fills in the gap with something that sounds right. That's the confident-wrong pattern. It isn't a reasoning failure. It's an out-of-distribution failure wearing a reasoning costume.
Where the vision tower breaks
Screenshots are the clearest production example. A UI screenshot is dense, text-heavy, high-frequency, and nothing like a photograph — but most vision encoders were tuned on photographic statistics, and the tokenizer and resolution pipeline built for photos will happily downsample a screenshot until the small text is mush. The model isn't failing to read; it never received a legible image. Same story with dark or low-contrast frames from cameras that weren't optimized for aesthetics, or dense technical diagrams where spatial relationships carry the meaning and a global embedding throws that away.
The fix people reach for first is "use a bigger, better model." That helps at the margin. It doesn't fix the distribution problem, because the scaling laws for vision-language pretraining are still riding on the same web-image-and-caption data sources. A frontier vision encoder is a better-calibrated version of the same narrow prior, not a different prior.
Audio is speech-shaped
Audio encoders have their own version of this. Most were trained overwhelmingly on clean or near-clean speech — podcasts, audiobooks, call center recordings, read speech corpora. That means the model's implicit prior is: audio is one person talking. Feed it two overlapping speakers, a conference room with cross-talk, a phone call compressed through a codec, or — the case that breaks it hardest — audio that isn't speech at all (an alarm, a machine fault sound, ambient noise you actually care about), and the model degrades in a specific, predictable way: it tries to force what it hears into a speech-shaped transcript. Diarization gets confidently wrong. Non-speech events get silently dropped, because there was never a training signal that rewarded noticing them.
Video is frames, not motion
Video is where the illusion is most complete, because "video understanding" in most systems is not a temporal model at all. It's an image encoder applied to a sparse sample of frames, stitched into a sequence, with the actual continuity of motion reconstructed — badly — by the decoder's language priors. Sample one frame per second from a five-minute clip and you've thrown away almost everything that happened between frames. Ask the model to count repetitions in an exercise video, catch the moment an object changes hands, or notice a one-frame anomaly, and you're asking a question the sampling strategy made unanswerable before the model ever ran. The token budget makes this worse: video is by far the most expensive modality per second of content, so the practical pressure is always toward coarser sampling, which quietly narrows what the model can perceive further still. This is a preprocessing decision, not a model limitation, and it's usually made by whoever wired up the ingestion pipeline, not by anyone thinking about model capability.
What to actually do about it
The engineering implication is to stop treating "multimodal accurac
Top comments (0)