Moving Beyond Image Priors: Why Video Generative Models Are the Next Frontier for Geometry Estimation

The shift toward generative models in computer vision has reached a point where we no longer just ask models to "see" the world, but to "simulate" it. Traditionally, monocular depth estimation—the task of predicting the 3D structure of a scene from a single 2D image—has been dominated by discriminative models. These systems, such as the widely used Depth Anything V2, rely on massive, diverse datasets to learn the correlation between pixels and depth. While effective, they are compute-heavy and require significant labeled data to generalize across different environments.

Recently, research has pivoted toward repurposing generative image models, like Stable Diffusion, for these tasks. Models like Marigold demonstrated that the visual priors captured during image generation training could be "distilled" into high-fidelity depth maps. However, image-based models have a fundamental limitation: they lack the inherent understanding of physical consistency and temporal flow that a video model must possess.

A new framework called GeoNeXt, detailed in the recent paper Video Generative Models as Geometry Learner, takes a different approach. Instead of adapting an image model, the researchers repurpose pretrained video generative models to solve monocular depth and surface normal estimation. The result is a system that achieves state-of-the-art zero-shot performance using 100 times less training data than the best discriminative models.

The Problem: Fragmentation in Generative Geometry

To understand why video models are a better fit, we first have to look at the failings of the current generative crop. Most existing methods treat geometry estimation as an image-conditioned generation task. You provide an image, and the model generates a corresponding "depth image." This has two main drawbacks:

1. Task Isolation: Most models train for depth and surface normal estimation independently. This ignores the physical reality that depth (distance) and surface normals (orientation) are mathematically and physically linked. By splitting the tasks, you lose the opportunity to use one to regularize the other.
2. Data Requirements: Adapting an image backbone to a structural task typically requires either training task-specific "heads" or fine-tuning the entire backbone with substantial labeled data. The model has to relearn how to represent structure from scratch, as the original image-generation weights weren't optimized for metric precision.

The Innovation: Geometry as Next-Frame Prediction

GeoNeXt moves past these issues by reformulating geometry estimation as a next-frame prediction task. In a typical video model, the system is trained to predict the next $N$ frames given an initial frame. GeoNeXt treats the input image as the first frame and defines the geometric targets—depth maps and surface normal maps—as the subsequent "frames" in the sequence.

Formulating structural vision as a temporal sequence is a subtle but powerful shift. Video generative models, such as Stable Video Diffusion, are inherently built to represent spatial consistency and light behavior over time. To generate a coherent video, the model must "understand" how a 3D object rotates, how perspectives change as a camera moves, and how light interacts with surfaces across multiple frames. By framing depth and normals as the "future" of the image, GeoNeXt inherits these powerful structural priors without needing to be explicitly taught the laws of physics.

The authors adapted the video model's architecture to jointly model the relationship between images and geometry. This "unified" approach means the model doesn't just guess where things are; it builds a consistent internal representation where the depth of a pixel and the orientation of the surface at that pixel support each other. Conceptually, the model is using its temporal reasoning to "extrapolate" the physical reality of a static image.

Why Data Efficiency Matters

One of the most impressive results from the GeoNeXt paper is its data efficiency. In machine learning, we often equate performance with dataset size. Discriminative models like Depth Anything often train on millions of unlabeled or weakly-labeled images to reach their peak performance. These models are essentially memorizing the statistical properties of vast amounts of data.

GeoNeXt, by contrast, competes with these SOTA models while using two orders of magnitude less training data. Specifically, it rivals discriminative models trained on over 100x more data. This is possible because the video generative model has already done the "heavy lifting" during its initial pretraining on massive video corpora. It already knows what a car looks like from the side versus the front, and it knows how shadows fall on a wall or how a corridor recedes toward a vanishing point. The fine-tuning process doesn't need to teach the model about the world; it only needs to provide a small "nudge" to map that existing knowledge onto a specific geometric coordinate system.

For developers and researchers, this drastically lowers the barrier to entry. If you need to train a specialized geometry model for a niche domain—say, industrial inspections of solar panels or analyzing medical ultrasound imagery—you no longer need a massive, labeled dataset. A much smaller, higher-quality set of ground-truth data may be sufficient when starting from a video generative prior, making the development of domain-specific vision systems significantly more accessible to smaller teams and startups.

Practical Implications for Developers

For practitioners in computer vision, GeoNeXt signals a change in how we might build "world models" for robotics and autonomous systems.

• Zero-Shot Robustness: Because GeoNeXt relies on the massive, diverse priors of a video generator, it generalizes remarkably well. In tests across diverse datasets—ranging from the indoor-focused NYU Depth V2 to the outdoor driving-centric KITTI—it maintains high accuracy without needing to be retrained for each specific camera setup or environment. This makes it an ideal candidate for "in-the-wild" applications where lighting and weather conditions are unpredictable.
• Unified Output and Mathematical Consistency: Getting both depth and surface normals from a single, consistent pass simplifies the downstream geometry pipeline. In robotics, a depth map tells you if there is an obstacle, but a surface normal map tells you if a surface is "walkable" or if an object can be grasped. By providing both in a unified framework, GeoNeXt ensures that these two signals aren't contradicting each other, which is a common failure mode when using two separate models.
• Resource Trade-offs and Latency: The primary caveat is at the infrastructural level. Diffusion-based models are computationally more expensive to run during inference than standard CNNs or Transformers. If your application needs 60 FPS on a low-power mobile chip for a drone, a lightweight discriminative model is still the better fit. However, for applications where accuracy and structural fidelity are prioritized over raw latency—such as 3D mapping, architectural scanning, or offline high-fidelity scene reconstruction—the generative video approach is now the clear choice. We are seeing a new era where we can trade "wait time" for "world understanding."

Conclusion

The GeoNeXt framework proves that the "intelligence" captured by video generative models goes deeper than just moving pixels. It represents a functional understanding of 3D geometry that can be extracted with surprising efficiency. As these models become more optimized and inference techniques like distillation continue to improve, the gap between "generative" and "real-time" will shrink.

For now, GeoNeXt stands as a testament to the power of repurposing: by seeing geometry as just another "frame" in the world's video, we can build vision systems that are more efficient, more accurate, and more robust than ever before.

---

Supporting Sources:

• Marigold: Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
• Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
• Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets

Tags: ai, machinelearning, deeplearning, computervision, research