This is a cross-post. The original (with diagrams, code, and cheat sheets for every chapter) lives on my blog:
👉 The Evolution of Spatial VLMs
Vision-language models can describe a warehouse photo in fluent prose — and then fail to answer "how many meters is the forklift from the shelf?" They see but don't perceive. Over 2024–2026 a line of research closed that gap step by step, and the arc is remarkably coherent once you read it through a single lens:
The representational mismatch between discrete language tokens and continuous 3D geometry — and the successive attempts to close it.
I wrote a 10-chapter course tracing that arc paper by paper. Here's the map, with a one-line "why this paper had to exist" for each. Full chapter (analogies + mermaid diagrams + runnable code) is linked on each.
The 10-paper arc
BLINK: See but Not Perceive — The diagnosis. Humans 95.7% vs GPT-4V 51.3% on tasks people solve "in the blink of an eye." VLMs read images as a bag of semantic concepts, not geometry.
SpatialVLM: Data Is the Answer — Reframe spatial blindness as a data problem, not an architecture one. Lift 2D web images to metric 3D and synthesize 2B spatial VQA pairs.
SpatialRGPT — Point at a Region, Plant the Depth — Add region grounding + a gracefully-degrading depth plugin. The first VLM to point at "this object" and reason in metric 3D.
SpatialBot — Learning to Read Depth — Make depth queryable: a Depth API the model can call instead of guessing.
Thinking in Space — From single images to video: does the model build a cognitive map and remember space?
OmniSpatial — Draw the map of the ability itself: a comprehensive taxonomy of spatial reasoning.
Perspective-Aware Reasoning (APC) — "If I stood over there…" — mental imagery to break egocentric bias.
Depth-native VLMs — DepthVLM vs DepthLM — What if the VLM is the depth sensor? Two takes on baking depth in natively.
Verifiable RL for Spatial Reasoning — Spatial skill isn't only more labels; a verifiable reward can activate latent ability. SVQA-R1 → Smooth Operator (50K samples rivaling 4.5M supervised).
Grounded 3D-Aware — Reconstruct While You Reason — The synthesis: GR3D promotes grounding into a generation-time mechanism; VLM-3R folds 3D reconstruction into instruction tuning. Plus the six frontiers still open.
Why read it as one thread?
Because each paper is a different answer to the same question: how do you get a token-based model to respect continuous geometry? Data synthesis, depth plugins, cognitive maps, mental rotation, native depth, verifiable RL, and reconstruction-in-the-loop are all attacks on that one mismatch. Seeing them in sequence turns a pile of papers into a story with a direction.
If that framing is useful, the full course — with a visual cheat sheet, concept diagrams, and code for each chapter — is here:
👉 The Evolution of Spatial VLMs — full course
I write these paper-arc courses regularly (VLM post-training, Physical AI, agent architectures) at direcf.github.io.
Top comments (0)