This is a simplified guide to an AI model called Vggt-1b-Point maintained by Vufinder. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.
Overview
vggt-1b-point is a feed-forward neural network that directly infers all key 3D attributes of a scene—including extrinsic and intrinsic camera parameters, point maps, depth maps, and 3D point tracks—from one, a few, or hundreds of image views within seconds. Built by vufinder, this model represents the Replicate deployment of VGGT (Visual Geometry Grounded Transformer), a CVPR 2025 Best Paper winner developed by Meta AI Research and the University of Oxford's Visual Geometry Group. The model processes images up to a maximum dimension of 518 pixels and accepts JPG, JPEG, PNG, and WEBP formats, as well as video inputs (MP4, AVI, MOV). Unlike traditional approaches that require post-processing with visual geometry optimization, this single feed-forward pass achieves state-of-the-art results across multiple 3D vision tasks while maintaining sub-second inference speed on typical hardware.
Best use cases
3D reconstruction from multi-view image collections — This model excels at reconstructing complete 3D scenes from unordered image sets. Given a folder of photos of a building, sculpture, or small object taken from different angles, it recovers full camera poses and depth-consistent point clouds in under one second, outperforming traditional SfM pipelines that require hours of optimization. The model naturally handles varying numbers of input images, from single-view reconstruction (with surprisingly good results despite no single-view training) to hundreds of frames without architectural changes.
Dense point cloud generation for robotics and spatial mapping — Applications requiring dense 3D point clouds—robot manipulation planning, AR spatial understanding, autonomous navigation mapping—benefit from the model's ability to output both depth maps and direct point map predictions. The depth-to-point unprojection capability (using recovered intrinsic and extrinsic parameters) produces geometry-consistent reconstructions suitable for downstream geometric reasoning and collision detection.
Camera parameter estimation for video analysis — When you need to recover intrinsic camera calibration and frame-by-frame extrinsic poses from video, this model extracts both in one pass. This enables video-based camera intrinsic inference without traditional checkerboard calibration or vanishing point detection, useful for analyzing archival footage, user-generated video content, or uncalibrated camera sequences where ground truth parameters are unavailable.
3D point tracking across video frames — The model's track head predicts dense correspondence of selected point locations across multiple frames. Given query points in image coordinates, it outputs tracking predictions with visibility and confidence scores, enabling motion capture, optical flow, deformation tracking, and video action understanding without frame-by-frame optical flow computation.
Integration with downstream 3D tasks — The pretrained VGGT backbone significantly enhances downstream 3D tasks including non-rigid point tracking and feed-forward novel view synthesis. If your application requires a 3D feature backbone, using this model's learned representations avoids retraining from scratch and provides stronger priors about 3D geometry than standard 2D vision transformers.
Limitations
Input resolution constraint — Images and video frames are padded to a single aspect ratio and resized to a maximum dimension of 518 pixels. High-resolution details beyond this threshold are lost, limiting applicability for tasks requiring fine-grained feature matching or very large-scale reconstruction. Panoramic or extremely wide aspect ratio images are padded rather than cropped, potentially introducing distortion.
Video frame sampling overhead — Video inputs are sampled at a configurable rate (default 24 fps, sampling every n-th frame). The first and last frames are always included, but intermediate frame selection is regular-interval-based rather than content-aware, potentially missing critical motion or geometric changes in fast-moving scenes.
Limited semantic understanding — The model outputs only geometric attributes (cameras, depth, points, tracks). It does not segment objects, classify scene content, or understand semantic categories. If you need to distinguish foreground from background or identify object types, post-processing with segmentation models is required.
Single-view reconstruction uncertainty — Although the model shows "surprisingly good" zero-shot single-view reconstruction performance despite no single-view training, the paper does not quantify monocular depth error rates. Performance drops compared to multi-view cases, and no absolute depth scale is guaranteed from single images alone.
Memory requirements for many frames — A May 2025 update fixed memory bloat, allowing 2–3x more frames per GPU memory budget, but specific VRAM requirements per frame count are not documented. Very large video sequences or high frame counts may still exceed typical GPU memory.
License restrictions for non-commercial checkpoint — The original VGGT-1B checkpoint is licensed for non-commercial use only. Commercial use requires applying for access to the VGGT-1B-Commercial checkpoint via an approval form similar to LLaMA's workflow. The Replicate deployment uses the commercial checkpoint, but downstream integration of the raw checkpoint requires separate licensing verification.
No guarantees on dynamic or non-rigid scenes — The model was trained on static or nearly-static scenes. Heavily dynamic content, deforming objects, or non-rigid motion may produce inconsistent or inaccurate depth and tracking estimates.
How it compares
vggt-1b (also by vufinder) appears to be an alternative Replicate deployment of the same model family. The "point" variant in this guide's name suggests it prioritizes point map outputs, while vggt-1b may emphasize other attributes like depth or camera parameters. Choose based on which output type (point clouds vs. depth maps) matches your pipeline's expectations.
VGGT-1B hosted on Hugging Face is the original Facebook research checkpoint. Use this if you need fine-grained control over model loading, want to avoid Replicate's API latency, or require direct access to intermediate representations (aggregated tokens, per-branch predictions). The Replicate version offers convenient stateless API calls and automatic output formatting but sacrifices local control and token-level manipulation.
VGGT-1B-Commercial is the licensed-for-commercial-use variant of the Hugging Face checkpoint. If you operate commercially and need to use the raw model weights outside Replicate, this checkpoint is required. Replicate deployments may already use this internally, but verify licensing terms if bundling or redistributing.
vfusion3d is a generative 3D model that synthesizes novel 3D objects and scenes rather than reconstructing from images. Choose VGGT if you have input images and need to extract their geometry; choose VFusion3D if you want to generate entirely new 3D content from text or image prompts. VFusion3D is slower (training-based generation) but supports creative synthesis; VGGT is feed-forward and deterministic.
Technical specifications
The model is a Transformer-based architecture with an aggregator component that fuses multi-view image tokens, followed by specialized prediction heads for cameras, depth, points, and tracks. The base model size is 1 billion parameters. Training used PyTorch and utilized bfloat16 precision on Ampere GPUs (Compute Capability 8.0+), with float16 fallback for older hardware. The model expects input images normalized and padded to a common aspect ratio before resizing to a maximum of 518 pixels in the longest dimension.
Input processing:
- Supported image formats: JPG, JPEG, PNG, WEBP
- Supported video formats: MP4, AVI, MOV
- Maximum image dimension: 518 pixels
- Input images padded to single aspect ratio, resized to fit max dimension
- Video frames sampled at configurable rate (default: every 24th frame, first and last always included)
Output types:
- Extrinsic camera matrices (4×4, world-to-camera, OpenCV convention)
- Intrinsic camera matrices (3×3 per view)
- Depth maps (per-view, floating-point)
- Point maps (per-view XYZ coordinates)
- 3D point tracks with visibility and confidence scores
- Point cloud files (optional, .ply or equivalent)
- JSON files with arrays returned as base64 strings (configurable)
Precision and hardware:
- Supported: bfloat16 on Ampere and newer GPUs (Compute Capability ≥ 8.0)
- Fallback: float16 on older hardware
- Inference time: under 1 second for typical multi-view sets
Geometry utilities:
- Depth-to-point unprojection using intrinsic and extrinsic parameters
- COLMAP export format (cameras.bin, images.bin, points3D.bin)
- Compatible with Gaussian splatting libraries (gsplat 1.3.0+)
- Bundle adjustment optional on COLMAP output
Model inputs and outputs
Inputs
- inputs (array of URIs, required) — Array of image file URLs or video file URLs. Accepts JPG, JPEG, PNG, WEBP for images; MP4, AVI, MOV for video. Each input is processed and padded to a consistent aspect ratio, then resized to maximum 518 pixels.
- sampling_rate (integer, default: 24) — For video inputs only. Sampling interval: every n-th frame is extracted. First and last frames always included regardless of sampling rate.
- to_base64 (boolean, default: true) — Whether to return arrays in JSON output files as base64-encoded strings with shape and dtype metadata, versus raw numeric values.
- return_pcd (boolean, default: true) — Whether to generate and return a point cloud file (.ply or equivalent binary format).
- keys_to_exclude (string, default: empty) — Comma-separated list of output JSON keys to exclude from results. Useful for reducing output size when certain predictions (e.g., confidence scores) are not needed.
- alpha_blend_onto (enum: "white", "mean", "keep", default: "white") — For images with alpha channels: blend onto white background, ImageNet mean RGB, or keep original pixel values.
Outputs
-
predictions (JSON object or files) — Contains nested objects with per-view and per-scene predictions:
- cameras — Extrinsic (4×4 world-to-camera) and intrinsic (3×3) matrices per view
- depth_maps — Per-view floating-point depth predictions with confidence
- point_maps — Per-view XYZ point coordinates with confidence
- tracks — 3D point correspondences across frames with visibility and confidence scores
- metadata — Input resolution, processing details
- point_cloud (binary file) — PLY-format point cloud (if return_pcd=true), containing 3D coordinates and optional RGB from input images
- json_arrays (base64 strings or raw arrays) — Configurable via to_base64; arrays returned as {"data": , "shape": [...], "dtype": "float32"} or raw numeric values
Getting started
import replicate
# Prepare a list of image URLs or local file paths
# For Replicate, use publicly accessible URLs or upload to a file service
image_urls = [
"https://example.com/scene_view_1.jpg",
"https://example.com/scene_view_2.jpg",
"https://example.com/scene_view_3.jpg"
]
# Run the model
output = replicate.run(
"vufinder/vggt-1b-point:9f6a02f93150fe8a7054d5bfed686076c4d88df610211e2e42302675a7e455da",
input={
"inputs": image_urls,
"sampling_rate": 24,
"to_base64": True,
"return_pcd": True,
"keys_to_exclude": "",
"alpha_blend_onto": "white"
}
)
# Output contains predictions dict with cameras, depth_maps, point_maps, tracks
# and optionally a point_cloud file URL
print(output)
Frequently asked questions
Q: Can I use this model for commercial purposes?
A: Yes, the Replicate deployment uses the VGGT-1B-Commercial checkpoint, which is licensed for commercial use (excluding military applications). If you download and use the raw model weights separately, ensure you access the commercial variant; the original non-commercial checkpoint requires a non-commercial license.
Q: What is the difference between point_map and depth-based 3D reconstruction?
A: The model predicts both depth maps and direct point maps. The README notes that depth-based reconstruction (unprojecting the depth map using camera intrinsics and extrinsics) typically produces more accurate 3D points than the point map branch alone, though both are available for flexibility depending on your use case.
Q: How many images do I need for good reconstruction?
A: The model handles one to hundreds of views without architectural changes. Single-view reconstruction works surprisingly well despite no single-view training, but multi-view (3+ images) is expected to produce more accurate geometry due to stronger geometric constraints.
Q: What happens to images with alpha channels or transparency?
A: Transparent regions are blended onto a background specified by the alpha_blend_onto parameter: white (default), ImageNet mean RGB, or the original pixel values. Simple masking of unwanted regions (e.g., reflective surfaces, sky) is possible by setting those pixels to 0 or 1; precise segmentation masks are not required.
Q: Can I track specific points across frames?
A: Yes, the model's track_head accepts query_points (N×2 coordinates per image) and outputs 3D tracks with visibility and confidence scores. This enables dense correspondence tracking without separate optical flow models.
Q: How fast is inference, and what GPU do I need?
A: Inference completes in under 1 second for typical multi-view sets. A 2025 memory optimization allows 2–3x more frames per GPU budget than prior versions, but specific VRAM requirements are not documented. A modern GPU (Ampere or newer for bfloat16 support) is recommended.
Q: Is the model still maintained?
A: Yes. The README notes updates through May 2026, including VGGT-Omega (a next iteration) and memory optimizations. Training code, evaluation code, and integration guides for Gaussian splatting and COLMAP export are actively supported in the official repository.
Q: Can I export the reconstruction for use in other tools?
A: Yes. The model supports direct COLMAP format export (cameras.bin, images.bin, points3D.bin) with optional bundle adjustment, compatible with Gaussian splatting libraries (gsplat 1.3.0+) and other NeRF/3D frameworks. Use the demo_colmap.py script or set return_pcd=true for direct point cloud output.
Top comments (0)