This is a simplified guide to an AI model called Map-Anything maintained by Vufinder. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.
Overview
map-anything is a universal feed-forward metric 3D reconstruction model developed by vufinder, built on research from Meta and Carnegie Mellon University. The model uses a transformer architecture that directly regresses metric 3D scene geometry and camera parameters from one or more images, optionally augmented with camera intrinsics, poses, depth maps, or partial reconstructions. The single most important thing to know is that this model handles 12+ different 3D reconstruction tasks—including uncalibrated structure-from-motion, calibrated multi-view stereo, monocular depth estimation, camera localization, and depth completion—in a single feed-forward pass, eliminating the need to switch between specialist models. It outputs metric 3D coordinates in world space alongside recovered camera parameters, confidence scores, and optional mesh and point cloud reconstructions, all from a single unified interface.
Best use cases
Multi-view 3D reconstruction for robotics and autonomous systems: The model produces metric-scale 3D geometry from multiple images without requiring explicit camera calibration or pose estimation as inputs, making it ideal for robots that need to build 3D scene understanding from onboard cameras. The recovered camera poses and globally consistent 3D points directly feed into SLAM and navigation pipelines without post-processing.
Monocular depth estimation with metric scale recovery: When you have single images but need true metric depth (not just relative depth), this model recovers both pixel-wise depth and the metric scaling factor that converts relative depth to real-world coordinates. This is critical for applications like augmented reality overlays, robotic grasping, or indoor navigation where relative depth alone is insufficient.
Structure-from-motion without calibration: For unstructured image collections where camera parameters are unknown, the model jointly recovers both the 3D scene geometry and the camera intrinsics and extrinsics that explain the images, eliminating the need to run separate bundle adjustment or camera calibration tools.
Depth completion and refinement: When you have sparse or noisy depth maps from sensors (LiDAR, structured light), you can pass them as inputs alongside images, and the model refines them into dense, metrically accurate depth maps while leveraging visual information to fill gaps and correct errors.
Large-scale multi-image datasets with flexible inputs: The model scales to hundreds or thousands of views using memory-efficient inference mode that trades negligible speed for vastly reduced GPU memory. It accepts any combination of images, intrinsics, depths, and poses, allowing you to feed partial information from mixed sensor sources (monocular cameras, stereo rigs, depth sensors, SLAM systems) into a single reconstruction pipeline.
Limitations
The model pads all input images to a single aspect ratio and resizes them to a maximum dimension of 518 pixels, which may lose fine details in high-resolution captures or architecturally complex scenes. While the paper reports strong performance on standard benchmarks, the absolute metric accuracy depends on the quality and diversity of the training data; results may degrade on highly specialized domains not well-represented in training. The model outputs depth in two forms—Z-depth in camera frame and depth-along-ray—which require careful interpretation; incorrect choice between these can cause coordinate system mismatches in downstream applications. Memory-efficient inference mode, while supporting up to 2000 views, trades some speed for memory savings and may be impractical for real-time applications with strict latency budgets. The model requires either internet access to download from Hugging Face or a pre-populated cache, which complicates offline or air-gapped deployments. Camera pose outputs follow OpenCV convention (X-Right, Y-Down, Z-Forward), requiring explicit conversion if your pipeline uses other conventions like ROS or OpenGL. Masking outputs include both learned confidence scores and geometric consistency-based masks, but they may not perfectly filter ambiguous or textureless regions, requiring manual refinement for applications with strict quality thresholds.
How it compares
VGGT-1b is a feed-forward 3D inference model that directly infers scene geometry, but map-anything offers significantly broader task coverage—12+ reconstruction scenarios versus VGGT's narrower focus—and produces globally consistent metric reconstruction across multiple views. Choose map-anything when you need unified handling of mixed input modalities (images with optional poses, depths, intrinsics); choose VGGT-1b if you need a simpler, faster model optimized for a single specific task.
Depth-anything-v3-mono specializes in monocular relative depth estimation and produces fine-grained depth maps, but it returns only relative depth without metric scale or camera parameters. Use map-anything when you need metric-scale depth alongside camera intrinsics and pose recovery; use Depth-Anything-v3-mono when you only need fast, high-quality relative depth and can live without scale information or multi-view integration.
Technical specifications
The model is a transformer-based architecture that factors 3D scene geometry into depth maps, local ray maps, camera poses, and a metric scale factor. It ingests images up to a maximum dimension of 518 pixels (padded to single aspect ratio) and outputs metric 3D points in world coordinates, camera parameters, confidence scores, and optional masks. Inference supports mixed-precision computation using bf16 (or fp16 fallback) and memory-efficient batching modes. The framework supports integration with external 3D reconstruction models including VGGT, DUSt3R, MASt3R, MUSt3R, and Pi3-X through a unified interface.
Key technical details:
- Input resolution: Images resized to maximum dimension of 518 pixels with padding to single aspect ratio
- Output coordinate systems: 3D points in world coordinates and camera-relative coordinates; camera poses in OpenCV cam2world convention (X-Right, Y-Down, Z-Forward)
- Output depth representations: Both Z-depth in camera frame and depth-along-ray in camera frame provided per pixel
- Batch outputs: Per-view predictions include 3D points (B, H, W, 3), depth maps (B, H, W, 1), camera intrinsics (B, 3, 3), poses (B, 4, 4), quaternions, translations, confidence scores (B, H, W), and validity masks
- Inference modes: Standard forward pass or memory-efficient mode supporting up to 2000 views with configurable minibatch sizes
- Precision options: Full precision, automatic mixed precision (AMP) with bf16 (fallback to fp16), or explicit fp16
- Masking outputs: Combined validity mask, non-ambiguous region mask, and mask logits for filtering low-confidence or edge artifact regions
- Optional post-processing: Edge artifact removal via normals and depth, confidence-based filtering with configurable percentiles, and multi-view depth consistency scoring
- Exports: Point cloud (PLY format) and mesh (GLB/OBJ format) reconstruction with optional base64 encoding of array data
- Video support: Samples at configurable rate (default 24 fps) with automatic inclusion of first and last frames
- JSON geometric input: Accepts JSON-encoded camera parameters alongside images for mixed modality reconstruction
Model inputs and outputs
Inputs
- Inputs (array of URIs, required): Image files (JPG, JPEG, PNG, WEBP) or video files (MP4, AVI, MOV) or JSON files containing geometric metadata
- Sampling rate (integer, default 24): Frame sampling interval for video input; first and last frames always included
- Return PCD (boolean, default true): Whether to output a point cloud file
- Return mesh (boolean, default true): Whether to output a mesh file
- To base64 (boolean, default true): Whether to encode array data in JSON outputs as base64 strings with shape and dtype metadata
- Alpha blend onto (enum, default "white"): Blending mode for images with alpha channels; "mean" blends onto ImageNet mean RGB values, "keep" preserves original pixel values
- Keys to exclude (string, default empty): Comma-separated list of keys to omit from output JSON files
Outputs
- 3D geometry arrays: Per-view predictions returned as JSON-encoded arrays or base64-encoded tensors, including world-space 3D points, camera-relative 3D points, Z-depth, depth-along-ray, ray directions
- Camera parameters: Recovered intrinsics (3×3 matrices), extrinsics as 4×4 poses, translation vectors, and quaternion representations in OpenCV convention
- Quality metrics: Per-pixel confidence scores and multiple mask variants (combined validity, non-ambiguous regions, mask logits)
- Point cloud: PLY format file containing 3D points and optional per-point confidence or RGB values
- Mesh: GLB or OBJ format file with vertices, faces, and optional vertex colors derived from confidence or input images
- Metadata: Metric scaling factors, denormalized input images for visualization, and optional geometry export to COLMAP or other standard formats
Getting started
import torch
from mapanything.models import MapAnything
from mapanything.utils.image import load_images
# Initialize model on GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
model = MapAnything.from_pretrained("facebook/map-anything").to(device)
# Load images from a directory or list of paths
images = load_images("path/to/images/") # or ["img1.jpg", "img2.jpg", ...]
# Run metric 3D reconstruction
predictions = model.infer(
images,
memory_efficient_inference=True, # Use for large numbers of views
use_amp=True, # Mixed precision for speed
amp_dtype="bf16", # Use bf16 if supported
apply_mask=True, # Remove invalid regions
mask_edges=True, # Remove edge artifacts
)
# Access metric 3D geometry and camera parameters
for i, pred in enumerate(predictions):
pts3d = pred["pts3d"] # 3D points in world coordinates (H, W, 3)
depth_z = pred["depth_z"] # Z-depth in camera frame (H, W, 1)
intrinsics = pred["intrinsics"] # Camera intrinsics (3, 3)
camera_poses = pred["camera_poses"] # cam2world poses (4, 4)
confidence = pred["conf"] # Per-pixel confidence (H, W)
mask = pred["mask"] # Validity mask (H, W, 1)
print(f"View {i}: {pts3d.shape[0]}x{pts3d.shape[1]} points, "
f"focal length {intrinsics[0, 0]:.1f}")
Frequently asked questions
Q: What formats does the model accept for input images?
A: The model accepts JPG, JPEG, PNG, and WEBP image files, or MP4, AVI, and MOV video files (sampling every N frames). It also accepts JSON files encoding images with geometric metadata like camera intrinsics, depth maps, and poses for multi-modal reconstruction.
Q: Can I input images with known camera calibration to improve reconstruction?
A: Yes. The model accepts optional camera intrinsics, depth maps, camera poses, and depth metrics as JSON or structured inputs. When provided, these constrain the reconstruction and improve metric accuracy; when omitted, the model estimates them from images alone.
Q: What is the difference between the "pts3d" and "pts3d_cam" outputs?
A: pts3d contains 3D points in world coordinates (globally consistent across all views), while pts3d_cam contains the same points transformed to each individual camera's local coordinate frame. Use pts3d for scene reconstruction and mapping; use pts3d_cam for camera-relative geometry.
Q: Should I use Z-depth or depth-along-ray for my application?
A: Z-depth (depth_z) measures distance along the camera's Z-axis (standard in computer vision); depth-along-ray (depth_along_ray) measures distance along the actual ray from the camera center. Z-depth is typically correct for pinhole camera models and 3D reconstruction; depth-along-ray is useful for specialized raycasting or sensor fusion applications.
Q: What does the confidence score represent, and how should I filter results?
A: The conf output is a per-pixel learned confidence score indicating reconstruction reliability. The model also provides confidence_percentile filtering to automatically remove the bottom N% of scores. For safety-critical applications, use strict thresholds; for general purposes, the default masking handles most ambiguous regions.
Q: What license applies, and can I use this commercially?
A: The code is released under the Apache 2.0 license (see the repository LICENSE file). An Apache 2.0 licensed model variant is available as "facebook/map-anything-apache". Check the specific model variant's license before production deployment.
Q: Is memory-efficient inference slower than standard inference?
A: Memory-efficient inference trades negligible speed for vastly reduced GPU memory (supporting up to 2000 views on 140 GB VRAM). The performance penalty is minimal and worthwhile for large batches; use standard inference only if latency is critical and view counts are small.
Q: How does this model differ from running COLMAP for 3D reconstruction?
A: COLMAP is a traditional SfM/MVS pipeline requiring hours for large image sets and manual parameter tuning; map-anything runs in seconds as a single feed-forward pass, outputs metric scale automatically, and handles diverse input modalities. COLMAP may achieve higher accuracy on extremely well-textured scenes with perfect metadata; map-anything is vastly faster and more flexible for real-time and robotic applications.
Top comments (0)