A beginner's guide to the Sam3 model by Vufinder on Replicate

This is a simplified guide to an AI model called Sam3 maintained by Vufinder. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

Overview

sam3 is a unified foundation model for promptable segmentation in images and videos developed by vufinder and based on Meta's SAM 3 research. It contains 848M parameters and can detect, segment, and track objects using text prompts alongside visual prompts like points, boxes, and masks. The critical breakthrough in this model is its ability to exhaustively segment all instances of open-vocabulary concepts specified by text—it achieves 75–80% of human performance on the SA-CO benchmark containing 270K unique concepts, over 50 times more than existing segmentation datasets. The architecture features a DETR-based detector conditioned on text, geometry, and image exemplars, paired with a tracker that inherits the SAM 2 transformer encoder-decoder design. This dual-component design minimizes task interference and scales efficiently with training data, supported by a data engine that automatically annotated over 4 million unique concepts.

Best use cases

Open-vocabulary instance segmentation at scale. Use this model when you need to segment all instances of a concept described in natural language without predefined object categories. For example, segment every occurrence of "a person wearing red" or "a potted plant" in photographs, where the specific visual characteristics matter and the number of instances varies. The model's 270K concept benchmark indicates it handles nuanced distinctions that rule-based or category-locked systems cannot capture.

Interactive refinement workflows. Deploy this model in applications requiring users to iteratively refine segmentation results through additional prompts. The support for positive and negative boxes, positive and negative points, and text prompts together allows users to click, draw, or type corrections. This is ideal for annotation tools, design software, or quality assurance pipelines where precision matters and human feedback guides the system.

Video object tracking with text queries. Use this for video analysis when you need to find and track specific object types across frames using natural language descriptions. The unified architecture supports both image and video inputs, making it suitable for video summarization, content moderation, or automated video editing where identifying "the main subject" or "all cars in the scene" must work across varying lighting, angles, and occlusions.

Concept-based image search and cataloging. Apply this model to automatically index and retrieve images based on open-vocabulary concepts. Instead of manually tagging thousands of images with predefined categories, describe what you want to find ("ceramic vases with blue patterns") and the model segments matching instances across your image library. This scales much better than traditional tag-based systems for large, diverse visual collections.

Prototype computer vision pipelines without retraining. Leverage the model's foundation capabilities to validate vision pipeline ideas before committing to annotation and fine-tuning. Test whether your application concept works by prompting the model with text descriptions, and only invest in custom training if performance gaps emerge. The 75–80% human performance baseline suggests it will handle many tasks out of the box.

Limitations

The model requires at least one prompt input—either text, positive boxes, or positive points—and provides no content without guidance. Text prompts may fail or produce imprecise segmentations for extremely specific visual characteristics (e.g., "the third person from the left"), especially when those characteristics demand understanding of spatial relationships beyond appearance. The confidence threshold ranges from 0.01 to 0.99 with a default of 0.5, and lowering it dramatically increases false positives; there is no guidance on optimal thresholds for different concept types.

Input coordinates for boxes and points must be normalized to [0, 1], which requires coordinate transformation before use and is error-prone if not handled correctly. The schema does not specify maximum image resolution, but the README mentions CUDA 12.6 and GPU memory requirements without stating VRAM limits; extremely large images or high-resolution video frames may exhaust memory. The model does not support batch processing of multiple images in a single request—each image requires a separate inference call.

Video processing is not directly supported through the Replicate API schema shown; the README describes video predictor usage but the Replicate endpoint appears image-focused, creating potential confusion about video capability. The model outputs JSON files containing mask data and optionally PNG visualizations, but the schema does not specify the exact JSON structure, making integration less straightforward than models with clearly documented output formats.

The license is available on GitHub but the Replicate metadata does not explicitly state commercial usage rights; users must verify the Apache 2.0 or equivalent license for their use case. SAM 3 achieves 75–80% human performance on SA-CO, meaning 20–25% of segmentations fall short of human quality—this is acceptable for filtering and acceleration but insufficient for mission-critical applications like medical imaging without human review.

How it compares

sam-3/image by fal-ai provides the same SAM 3 capabilities but through fal-ai's infrastructure instead of Replicate. Choose sam3 on Replicate if you already use Replicate's ecosystem or prefer their billing model; choose fal-ai's version if fal-ai offers lower latency in your region or if you need tighter integration with fal-ai's other tools. The core model is identical, so the tradeoff is infrastructure and ecosystem preference rather than model quality.

sam3-image by mattsays is another Replicate SAM 3 wrapper that may differ in preprocessing, default confidence thresholds, or output formatting. Both run the same foundation model, so choose this version if you need a different maintainer's implementation details or API wrapping choices, but expect minimal quality differences. Check each maintainer's handling of edge cases like extremely small objects or complex negative prompts.

sam-3/image/embed by fal-ai adds embedding functionality to SAM 3, allowing you to extract feature representations of segmented regions for downstream tasks like clustering or similarity search. Choose this if you need embeddings alongside masks; choose the base sam3 if segmentation masks alone suffice, as embeddings add latency and model complexity you may not use.

segment_anything_model by ayumuakagi is the original SAM or SAM 2, lacking the open-vocabulary concept capability and text prompt support that define SAM 3. Choose SAM 3 for text-driven, concept-based segmentation; choose the original SAM models only if you need exclusively point/box/mask prompts and cannot upgrade. SAM 3's data engine and presence token architecture represent a major step forward for open-vocabulary work.

sam-3-1/image by fal-ai is the newer SAM 3.1 with Object Multiplex, a shared-memory approach for joint multi-object tracking that is faster when tracking many objects simultaneously. Choose SAM 3 if you need the current stable version or track few objects; choose SAM 3.1 if you track many objects across frames and speed matters more than model maturity.

Technical specifications

sam3 consists of 848M parameters distributed across a detector and tracker that share a vision encoder. The detector uses a DETR-based architecture conditioned on text, geometry, and image exemplars; the tracker uses a transformer encoder-decoder design derived from SAM 2. The model was trained on a dataset of over 4 million automatically annotated unique concepts, generated through a custom data engine. Performance on the SA-CO benchmark (270K unique concepts) reaches 75–80% of human performance, compared to human baselines of 72.8% on instance segmentation and 74.0% on box detection.

Architecture and training:

• 848M total parameters
• Detector: DETR-based, text-conditioned
• Tracker: Transformer encoder-decoder (SAM 2 lineage)
• Training data: 4 million+ unique concepts via automated annotation pipeline
• SA-CO benchmark: 270K concepts, 75–80% human performance
• Python 3.12+ required
• PyTorch 2.7+ with CUDA 12.6+ for GPU inference
• Optional dependencies: einops, ninja, flash-attn-3, cc_torch for faster inference

Input specifications:

• Image formats: JPG, JPEG, PNG, WEBP
• Coordinates: Normalized to [0, 1] range
• Box format: center_x, center_y, width, height (four floats each)
• Point format: x, y (two floats each)
• Text prompts: Strings describing concepts (e.g., "a person wearing red")
• At least one of text, positive_boxes, or positive_points must be provided per prompt

Output specifications:

• JSON files with mask data, bounding boxes, and confidence scores
• Optional PNG visualizations when visualize=true
• Optional cut-out masks with offset coordinates when offset_masks=true
• Optional backbone features as PT file when backbone_output=true
• Optional per-prompt JSON splitting when split_output=true

Inference parameters:

• confidence_threshold: 0.01–0.99, default 0.5
• visualize: Return PNG predictions (boolean, default false)
• concat_input: Include input prompts in output JSON (boolean, default false)
• offset_masks: Return masks as cut-outs with offset (boolean, default false)
• backbone_output: Return backbone features as PT file (boolean, default false)
• split_output: Separate output JSON per prompt (boolean, default true)

Model inputs and outputs

Inputs

• image (string, URI): Input image file (JPG, JPEG, PNG, or WEBP format)
• prompts (array of strings): JSON-serialized prompt objects, each containing optional text (string), positive_boxes (list of normalized [0, 1] boxes in center_x, center_y, width, height format), negative_boxes (list of normalized boxes), positive_points (list of normalized [x, y] coordinates), and negative_points (list of normalized coordinates). At least one of text, positive_boxes, or positive_points required per prompt.
• confidence_threshold (number): Detection confidence threshold, range 0.01–0.99, default 0.5
• visualize (boolean): Return PNG visualizations of predictions, default false
• concat_input (boolean): Include input prompts in output JSON under input key, default false
• offset_masks (boolean): Return masks as cut-outs with (x, y) offset of upper-left corner, default false
• backbone_output (boolean): Return backbone features as PT file, default false
• split_output (boolean): Separate output JSON files per prompt, default true

Outputs

• JSON files: Mask data, bounding boxes, and confidence scores per prompt
• PNG files (when visualize=true): Visual prediction results per prompt
• PT file (when backbone_output=true): Backbone feature representations
• Mask coordinates (when offset_masks=true): Mask cut-outs with pixel-space offsets for the original image

Getting started

import replicate
import json

# Initialize Replicate client
client = replicate.Replicate()

# Prepare prompts as JSON strings
prompts = [
    json.dumps({
        "text": "a person wearing red",
        "positive_boxes": [],
        "negative_boxes": [],
        "positive_points": [],
        "negative_points": []
    }),
    json.dumps({
        "text": "a potted plant",
        "positive_boxes": [],
        "negative_boxes": [],
        "positive_points": [],
        "negative_points": []
    })
]

# Run inference
output = client.run(
    "vufinder/sam3",
    input={
        "image": "https://example.com/image.jpg",
        "prompts": prompts,
        "confidence_threshold": 0.5,
        "visualize": True,
        "split_output": True
    }
)

# Process output
for item in output:
    print(f"Prompt result: {item}")

Frequently asked questions

Q: Can I segment multiple concepts in a single image request?

A: Yes. Provide multiple JSON-serialized prompt objects in the prompts array, each describing a different concept (e.g., "a person," "a car," "vegetation"). The model processes all prompts against the same image and returns separate results per prompt when split_output=true.

Q: What coordinate system do box and point prompts use?

A: All coordinates are normalized to [0, 1]. For boxes, use center_x, center_y, width, height format. For points, use x, y pairs. You must convert pixel coordinates to normalized coordinates by dividing by image width and height before passing to the API.

Q: What is the difference between positive and negative prompts?

A: Positive boxes and points tell the model "segment objects in this region" or "around this location." Negative boxes and points tell the model "do not segment objects in this region." Use negatives to exclude similar objects or refine oversegmentation when the text prompt alone is ambiguous.

Q: Does this model support video input directly through the Replicate API?

A: The Replicate endpoint shown in the schema is image-focused. The underlying SAM 3 repository includes video predictor capabilities, but the Replicate wrapper does not expose a video input parameter. For video work, you must process frames individually or use the open-source implementation directly.

Q: How do I handle the confidence_threshold parameter?

A: The threshold ranges from 0.01 to 0.99 with default 0.5. Lower values (e.g., 0.2) detect more instances but increase false positives; higher values (e.g., 0.8) filter aggressively, risking missed detections. Start at 0.5 and adjust based on whether you observe undersegmentation or oversegmentation in your test images.

Q: Is this model suitable for production use in medical imaging?

A: SAM 3 achieves 75–80% human performance on general segmentation benchmarks. For medical imaging, this performance gap (20–25% of segmentations below human quality) is problematic without human review. Use it as a preprocessing or acceleration tool, not as a standalone diagnostic system.

Q: Can I export the backbone features for use in downstream models?

A: Yes. Set backbone_output=true to receive backbone features as a PT file alongside masks. These embeddings can be used for clustering, similarity search, or as input to custom classifiers, allowing you to build on SAM 3's learned representations.

Q: What happens if my prompt is too vague or contradictory?

A: Vague prompts (e.g., "stuff") may segment many unrelated objects or nothing, depending on confidence_threshold. Contradictory prompts (e.g., text "red car" with negative_boxes covering all red regions) will likely fail or produce empty segmentations. Use clear, specific language and test negative prompts carefully to avoid unintended exclusions.

Click here to read the full guide to Sam3