A beginner's guide to the Vggt-1b-Depth model by Vufinder on Replicate

This is a simplified guide to an AI model called Vggt-1b-Depth maintained by Vufinder. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

Overview

vggt-1b-depth is a feed-forward neural network that infers 3D scene attributes from single, multiple, or hundreds of image views, reconstructing scenes in under one second. Built by vufinder and based on research from Meta AI and the University of Oxford's Visual Geometry Group, this model outputs depth maps alongside camera parameters, point clouds, and 3D point tracks. The 1B parameter transformer-based architecture processes images padded to a single aspect ratio and resized to a maximum of 518 pixels, supporting both image and video inputs. The critical distinction before using it: this variant specializes in depth map generation, making it the right choice when depth estimation is your primary need, while sibling models focus on other 3D attributes.

Best use cases

Robotics and autonomous systems perception: Depth estimation from monocular or multi-view imagery powers navigation, obstacle avoidance, and grasp planning. This model produces depth maps with confidence scores in under one second, enabling real-time perception pipelines where latency matters. The ability to process single frames through to multi-view sequences makes it adaptable to different sensor configurations.

3D scene reconstruction for mixed reality applications: Applications requiring rapid 3D environment mapping benefit from simultaneous depth and camera parameter inference. The model reconstructs geometry without post-processing optimization, which is faster than traditional structure-from-motion pipelines while still outperforming methods requiring bundle adjustment.

Monocular depth for computer vision pipelines: The model demonstrates zero-shot single-view depth estimation despite never being trained explicitly for monocular depth. This makes it useful as a backbone for downstream tasks requiring geometric understanding without labeled depth data in your domain.

Video frame analysis and temporal tracking: By accepting video inputs with configurable sampling rates and processing sequences of frames, the model infers consistent 3D geometry across time. The integrated point tracking capability complements depth maps for applications tracking features across video.

3D data generation for training synthetic datasets: Researchers and engineers creating labeled 3D data can feed raw image collections through this model to automatically generate depth maps, camera poses, and point clouds, then refine or filter results. The model's speed (sub-second per scene) makes batch processing feasible.

Limitations

Depth output limited to 518-pixel maximum dimension: Input images are resized to a maximum of 518 pixels on their longest edge after aspect ratio padding. This resolution constraint affects depth map precision—small objects or fine geometric details may not be recoverable, and depth boundaries blur at downsampled resolutions.

Requires multiple views for optimal results: While the model handles single-view input, multi-view depth estimation is its strength. Single-image reconstruction works but produces less reliable geometry than traditional monocular depth estimators on their specialized benchmarks, as noted in the README.

Processing time for visualization separate from inference: The model reconstructs scenes in under one second, but visualizing the resulting 3D point clouds takes tens of seconds due to rendering overhead. This decoupling matters for real-time applications where you only need depth and pose data, not visualization.

Non-commercial license on base checkpoint: The original VGGT-1B checkpoint is non-commercial. A commercial variant (VGGT-1B-Commercial) exists but requires application approval. The Replicate deployment uses the original checkpoint, restricting commercial deployment unless specific approval is obtained.

Masking required for problematic regions: Reflections, transparent surfaces, water, and sky regions degrade reconstruction. While the README mentions masking helps, it adds preprocessing complexity. Simple bounding box masks work, but precise segmentation isn't straightforward to generate automatically.

No explicit real-time depth streaming: Each invocation processes a complete image set. There is no built-in support for streaming or incremental depth updates as new frames arrive, limiting applicability to live sensor feeds without batching infrastructure.

How it compares

vggt-1b is the general-purpose sibling that outputs all 3D attributes (cameras, depth, point maps, tracks) in one model. Choose vggt-1b-depth when depth is your primary output; choose vggt-1b if you need the full attribute suite or don't know in advance whether you'll need camera poses and point clouds.

vggt-1b-point emphasizes point map output. Both depth-specialized and point-specialized variants exist because depth maps produce more accurate 3D coordinates when unprojected than raw point predictions. Select the depth variant for camera calibration and geometric constraints; select point maps if you prioritize dense point cloud density over geometric consistency.

map-anything is a universal feed-forward 3D reconstruction model also from vufinder. It likely offers broader generalization or different output formats; vggt-1b-depth is specifically tuned for depth prediction quality and carries CVPR 2025 Best Paper validation for its approach.

Technical specifications

The model is a transformer-based feed-forward network with 1 billion parameters. It accepts images and videos, preprocessing them by padding to a single aspect ratio and resizing to a maximum 518-pixel dimension. The architecture includes separate prediction heads for cameras (extrinsic and intrinsic matrices in OpenCV convention), depth maps with confidence scores, point maps with confidence scores, and 3D point tracks.

Inference uses bfloat16 on Ampere-class GPUs (Compute Capability 8.0+) and falls back to float16 on older hardware. The model processes through an aggregator stage producing tokens, then feeds those tokens to specialized heads for each output type. The implementation supports masking unwanted pixels by setting them to 0 or 1 without requiring precise segmentation masks.

Key technical details:

• Parameter count: 1 billion
• Input resolution: Up to 518 pixels maximum dimension after aspect ratio padding
• Output: Depth maps, depth confidence maps, camera extrinsics/intrinsics, point maps, point confidence maps, 3D point tracks
• Inference time: Under 1 second per scene
• Precision: bfloat16 preferred, float16 fallback
• Framework: PyTorch
• License: Non-commercial (original checkpoint)
• Video sampling: Configurable frame sampling rate with first and last frames always included

Model inputs and outputs

Inputs

• inputs (array of strings, required): Image or video file URLs. Accepts JPG, JPEG, PNG, WEBP for images; MP4, AVI, MOV for videos. Images and sampled video frames are padded to single aspect ratio and resized to maximum 518 pixels.
• to_base64 (boolean, default: true): Whether to return arrays in JSON files as base64 strings with shape and dtype metadata.
• return_pcd (boolean, default: true): Whether to return a point cloud file.
• return_depth (boolean, default: true): Whether to return depth images.
• sampling_rate (integer, default: 24): Frame sampling rate for video input (every n-th frame). First and last frames always included.
• keys_to_exclude (string, default: ""): Comma-separated list of output JSON keys to exclude from results.
• alpha_blend_onto (enum, default: "white"): Blend mode for images with alpha channels. Options include "mean" (ImageNet mean RGB) and "keep" (original pixel values).

Outputs

The model returns structured JSON files containing:

• Depth maps (when return_depth is true) as float arrays with shape and dtype
• Point cloud data in standard format (when return_pcd is true)
• Camera extrinsic matrices (4x4, world-to-camera transform following OpenCV convention)
• Camera intrinsic matrices (3x3, focal length and principal point)
• Depth confidence maps indicating prediction reliability per pixel
• Point map confidence scores
• 3D point track coordinates and visibility/confidence scores for queried points
• Additional geometry metadata excluding keys specified in keys_to_exclude

All outputs are available as base64-encoded arrays with metadata or as raw binary files depending on the to_base64 setting.

Getting started

import replicate

# Prepare your image URLs or local paths
input_images = [
    "https://example.com/image1.jpg",
    "https://example.com/image2.jpg",
    "https://example.com/image3.jpg"
]

# Run the model
output = replicate.run(
    "vufinder/vggt-1b-depth:cb9ca7fbf8477f0fdc598bd89a480b0e9fac4a06d7c50494137b1dd8cccd846e",
    input={
        "inputs": input_images,
        "to_base64": True,
        "return_pcd": True,
        "return_depth": True,
        "sampling_rate": 24,
        "keys_to_exclude": "",
        "alpha_blend_onto": "white"
    }
)

# Process outputs
print(output)

For video input, pass a single video URL:

output = replicate.run(
    "vufinder/vggt-1b-depth:cb9ca7fbf8477f0fdc598bd89a480b0e9fac4a06d7c50494137b1dd8cccd846e",
    input={
        "inputs": ["https://example.com/video.mp4"],
        "sampling_rate": 12,  # Every 12th frame
        "return_depth": True,
        "return_pcd": True
    }
)

Frequently asked questions

Q: Can I use the base VGGT-1B model on Replicate for commercial applications?

A: No. The checkpoint deployed here uses the original non-commercial license. Meta AI released a VGGT-1B-Commercial variant with commercial rights, but accessing it requires application approval through a system similar to LLaMA's workflow.

Q: How does depth output from this model compare to traditional stereo or structure-from-motion?

A: This model completes reconstruction in under one second without post-processing optimization, while traditional pipelines (COLMAP with bundle adjustment) take minutes to hours. The model outperforms methods requiring geometric optimization on benchmarks, but differs from monocular depth estimators specialized purely on single-image depth—it trades some single-view depth accuracy for multi-view consistency and camera parameter inference.

Q: What happens if I pass a single image instead of multiple views?

A: The model handles single-view input and demonstrates zero-shot monocular depth estimation, though it was never explicitly trained for this task. Results are less reliable than multi-view input; the README notes that single-view performance differs from dedicated monocular depth estimators.

Q: How do I handle images with unwanted regions like reflections or water?

A: Mask unwanted pixels by setting them to 0 or 1 before input. Precise segmentation masks are not necessary—simple bounding boxes work effectively. The model ignores masked regions during reconstruction.

Q: Can I export the depth and camera pose output to other 3D tools?

A: Yes. The model outputs OpenCV-convention extrinsic and intrinsic matrices. The README includes a script (demo_colmap.py) that exports predictions to COLMAP format, which integrates directly with Gaussian splatting libraries (gsplat) and other NeRF tooling.

Q: What video formats and sampling options does the model support?

A: Accepts MP4, AVI, and MOV. Sampling rate defaults to 24 (every 24th frame) but is configurable. The first and last frames are always included regardless of sampling rate.

Q: Is the model still actively maintained?

A: Yes. The repository shows recent updates (May 2026) fixing memory efficiency, allowing 2–3x more input frames on the same GPU budget. Training code became available in July 2025, and the model won CVPR 2025 Best Paper Award.

Q: What GPU memory do I need to run this model?

A: The README does not specify exact GPU memory requirements, but notes that recent optimizations reduce redundant tensor retention, improving memory efficiency. Testing on your hardware is recommended before production deployment.

Click here to read the full guide to Vggt-1b-Depth