How to Auto-Label your Segmentation Dataset with SAM3

Artem Zabarov — Sun, 15 Feb 2026 05:40:57 +0000

How to Auto-Label Your Entire Segmentation Dataset Using SAM 3 Text Prompts

Stop spending hundreds of hours manually drawing polygon masks. Here's how to use SAM 3's text-to-mask capabilities to label thousands of images for YOLO, Mask R-CNN, and other segmentation models — in a fraction of the time.

The Labeling Problem

If you've ever trained a segmentation model, you know the pain. Each image needs pixel-perfect masks drawn around every object of interest. For a single image with three objects, that's 5–10 minutes of careful polygon drawing. Scale that to a dataset of 5,000 images and you're looking at 400+ hours of manual work — or thousands of dollars outsourced to a labeling service.

Traditional tools like LabelMe, CVAT, and Roboflow have made the process faster, but you're still fundamentally drawing shapes by hand.

What if you could just tell the model what to find?

That's exactly what SAM 3's text grounding capability does. You give it an image and a text prompt like "car" or "person holding umbrella", and it returns pixel-perfect segmentation masks — no clicks, no polygons, no points. Just text.

In this guide, I'll walk you through:

How segmentation labeling works (and what format models like YOLO expect)
Setting up SAM 3 locally for text-to-mask inference
Building a batch labeling pipeline to process your entire dataset
Converting the output to YOLO, COCO, and other training formats

A Quick Primer on Segmentation Labels

Before we automate anything, let's understand what we're producing.

Bounding Boxes vs. Instance Masks

Object detection (YOLOv8 detect) only needs bounding boxes — a rectangle defined by [x_center, y_center, width, height] in normalized coordinates. Simple.

Instance segmentation (YOLOv8-seg, Mask R-CNN, etc.) needs the actual outline of each object — a polygon or binary mask that traces the exact boundary.

Label Formats

Different frameworks expect different formats:

YOLO Segmentation — One .txt file per image, each line is:

class_id x1 y1 x2 y2 x3 y3 ... xn yn

Where all coordinates are normalized (0–1) polygon points.

COCO JSON — A single annotations file with:

{
  "annotations": [
    {
      "id": 1,
      "image_id": 1,
      "category_id": 3,
      "segmentation": [[x1, y1, x2, y2, ...]],
      "bbox": [x, y, w, h],
      "area": 15234
    }
  ]
}

Pascal VOC — XML files with bounding boxes (no native mask support; masks stored as separate PNGs).

All of these require the same underlying information: where is the object, and what is its exact shape? SAM 3 gives us both.

What is SAM 3?

SAM 3 is the latest iteration of Meta's Segment Anything Model. What makes SAM 3 different from its predecessors is native text grounding — you can pass a natural language description and the model will find and segment matching objects in the image.

Under the hood, SAM 3 combines a vision encoder with a text encoder. The image is preprocessed to 1008×1008 pixels (with aspect-preserving padding), both encoders run in parallel, and a mask decoder produces per-instance masks, bounding boxes, and confidence scores.

The key components:

Sam3Processor — handles image preprocessing and text tokenization
Sam3Model — the full model (vision encoder + text encoder + mask decoder)
Post-processing — post_process_instance_segmentation() to extract clean masks

Setting Up SAM 3 Locally

Hardware Requirements

GPU: NVIDIA GPU with at least 8 GB VRAM (RTX 3060+ recommended)
RAM: 16 GB system RAM minimum
Storage: ~5 GB for model weights (downloaded automatically on first run)
CUDA: 12.0 or higher

SAM 3 can run on CPU, but expect inference to be 10–50× slower. For batch labeling thousands of images, a GPU is effectively mandatory.

Step 1: Set Up Your Environment

# Create a fresh conda/venv environment
conda create -n sam3-labeling python=3.10 -y
conda activate sam3-labeling

# Install PyTorch with CUDA support
# Visit https://pytorch.org/get-started/locally/ for your specific CUDA version
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

# Install SAM 3 dependencies
pip install transformers huggingface-hub Pillow numpy

Step 2: Verify CUDA Access

import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f} GB")

Step 3: Download and Load the Model

from transformers import Sam3Processor, Sam3Model

model_id = "jetjodh/sam3"

# First run downloads ~3-5 GB of weights to ~/.cache/huggingface/
processor = Sam3Processor.from_pretrained(model_id)
model = Sam3Model.from_pretrained(model_id).to("cuda")

print("Model loaded successfully!")

The first time you run this, it will download the model weights from Hugging Face. Subsequent runs load from cache in seconds.

Your First Text-to-Mask Prediction

Let's verify everything works with a single image:

from PIL import Image
import torch

# Load a test image
image = Image.open("test_image.jpg").convert("RGB")

# Prepare inputs — this is where the magic happens
# We pass BOTH the image and a text prompt
inputs = processor(
    images=image,
    text="car",
    return_tensors="pt",
    do_pad=False
).to("cuda")

# Run inference
with torch.no_grad():
    outputs = model(**inputs)

# Post-process to get instance masks
results = processor.post_process_instance_segmentation(
    outputs,
    threshold=0.5,          # Detection confidence threshold
    mask_threshold=0.5,     # Mask binarization threshold
    target_sizes=[(image.height, image.width)]
)[0]

print(f"Found {len(results['segments_info'])} instances")
for info in results['segments_info']:
    print(f"  Score: {info['score']:.3f}")

If you see "Found N instances" with reasonable scores, you're in business.

Building the Batch Labeling Pipeline

Now let's scale this up. We'll build a script that processes an entire dataset folder and produces labels in your format of choice.

The Complete Pipeline Script

"""
batch_label.py — Auto-label a dataset using SAM 3 text grounding.

Usage:
    python batch_label.py \
        --images ./dataset/images \
        --output ./dataset/labels \
        --prompt "person" \
        --class-id 0 \
        --format yolo \
        --threshold 0.5
"""

import argparse
import json
import os
from pathlib import Path

import numpy as np
import torch
from PIL import Image
from transformers import Sam3Model, Sam3Processor


def load_model(device: str = "cuda"):
    """Load SAM 3 model and processor."""
    model_id = "jetjodh/sam3"
    processor = Sam3Processor.from_pretrained(model_id)
    model = Sam3Model.from_pretrained(model_id).to(device)
    model.eval()
    return processor, model, device


def predict(processor, model, device, image: Image.Image, text: str,
            threshold: float = 0.5, mask_threshold: float = 0.5):
    """Run text-grounded segmentation on a single image."""
    inputs = processor(
        images=image,
        text=text,
        return_tensors="pt",
        do_pad=False,
    ).to(device)

    with torch.no_grad():
        outputs = model(**inputs)

    results = processor.post_process_instance_segmentation(
        outputs,
        threshold=threshold,
        mask_threshold=mask_threshold,
        target_sizes=[(image.height, image.width)],
    )[0]

    return results


def mask_to_polygon(binary_mask: np.ndarray, tolerance: int = 2):
    """Convert a binary mask to a simplified polygon using contour detection."""
    import cv2

    mask_uint8 = (binary_mask * 255).astype(np.uint8)
    contours, _ = cv2.findContours(mask_uint8, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

    if not contours:
        return None

    # Take the largest contour
    contour = max(contours, key=cv2.contourArea)

    # Simplify the polygon to reduce point count
    epsilon = tolerance * cv2.arcLength(contour, True) / 1000
    approx = cv2.approxPolyDP(contour, epsilon, True)

    if len(approx) < 3:
        return None

    # Flatten to [x1, y1, x2, y2, ...]
    polygon = approx.reshape(-1, 2)
    return polygon


def save_yolo_labels(masks, image_size, class_id, output_path):
    """Save masks in YOLO segmentation format (normalized polygon coordinates)."""
    w, h = image_size
    lines = []

    for mask in masks:
        mask_np = mask.cpu().numpy() if torch.is_tensor(mask) else mask
        polygon = mask_to_polygon(mask_np)
        if polygon is None:
            continue

        # Normalize coordinates to 0-1
        normalized = []
        for x, y in polygon:
            normalized.extend([x / w, y / h])

        coords = " ".join(f"{c:.6f}" for c in normalized)
        lines.append(f"{class_id} {coords}")

    with open(output_path, "w") as f:
        f.write("\n".join(lines))


def save_coco_annotation(masks, boxes, scores, image_id, image_size,
                         class_id, annotations_list, ann_id_counter):
    """Append COCO-format annotations to the running list."""
    import cv2

    w, h = image_size

    for i, mask in enumerate(masks):
        mask_np = mask.cpu().numpy() if torch.is_tensor(mask) else mask
        polygon = mask_to_polygon(mask_np)
        if polygon is None:
            continue

        # Flatten polygon for COCO format (absolute pixel coordinates)
        segmentation = polygon.flatten().tolist()

        # Compute bounding box from mask
        ys, xs = np.where(mask_np > 0)
        if len(xs) == 0:
            continue
        bbox = [int(xs.min()), int(ys.min()),
                int(xs.max() - xs.min()), int(ys.max() - ys.min())]

        annotation = {
            "id": ann_id_counter,
            "image_id": image_id,
            "category_id": class_id,
            "segmentation": [segmentation],
            "bbox": bbox,
            "area": int(mask_np.sum()),
            "iscrowd": 0,
            "score": float(scores[i]) if i < len(scores) else 1.0,
        }
        annotations_list.append(annotation)
        ann_id_counter += 1

    return ann_id_counter


def process_dataset(args):
    """Process all images in the dataset."""
    print(f"Loading SAM 3 model...")
    device = "cuda" if torch.cuda.is_available() else "cpu"
    processor, model, device = load_model(device)

    image_dir = Path(args.images)
    output_dir = Path(args.output)
    output_dir.mkdir(parents=True, exist_ok=True)

    # Collect image files
    extensions = {".jpg", ".jpeg", ".png", ".bmp", ".webp"}
    image_files = sorted(
        f for f in image_dir.iterdir()
        if f.suffix.lower() in extensions
    )
    print(f"Found {len(image_files)} images in {image_dir}")

    # COCO format state (if needed)
    coco_annotations = []
    coco_images = []
    ann_id = 1

    for idx, img_path in enumerate(image_files):
        print(f"[{idx + 1}/{len(image_files)}] {img_path.name}...", end=" ")

        image = Image.open(img_path).convert("RGB")
        results = predict(
            processor, model, device, image,
            text=args.prompt,
            threshold=args.threshold,
        )

        # Extract masks
        masks = results.get("masks", results.get("pred_masks"))
        if masks is None or len(masks) == 0:
            print("no instances found.")
            # Write empty label file for YOLO (so the image isn't skipped)
            if args.format == "yolo":
                (output_dir / f"{img_path.stem}.txt").write_text("")
            continue

        scores_list = [info["score"] for info in results.get("segments_info", [])]

        if args.format == "yolo":
            out_file = output_dir / f"{img_path.stem}.txt"
            save_yolo_labels(masks, image.size, args.class_id, out_file)
        elif args.format == "coco":
            coco_images.append({
                "id": idx,
                "file_name": img_path.name,
                "width": image.width,
                "height": image.height,
            })
            ann_id = save_coco_annotation(
                masks, None, scores_list, idx, image.size,
                args.class_id, coco_annotations, ann_id,
            )

        n = len(masks)
        print(f"{n} instance{'s' if n != 1 else ''} found.")

    # Save COCO JSON
    if args.format == "coco":
        coco_output = {
            "images": coco_images,
            "annotations": coco_annotations,
            "categories": [{"id": args.class_id, "name": args.prompt}],
        }
        coco_path = output_dir / "annotations.json"
        with open(coco_path, "w") as f:
            json.dump(coco_output, f, indent=2)
        print(f"COCO annotations saved to {coco_path}")

    print(f"\nDone! Processed {len(image_files)} images.")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Auto-label dataset with SAM 3")
    parser.add_argument("--images", required=True, help="Path to image directory")
    parser.add_argument("--output", required=True, help="Path to output label directory")
    parser.add_argument("--prompt", required=True, help="Text prompt (e.g. 'person', 'car')")
    parser.add_argument("--class-id", type=int, default=0, help="Class ID for labels")
    parser.add_argument("--format", choices=["yolo", "coco"], default="yolo",
                        help="Output format")
    parser.add_argument("--threshold", type=float, default=0.5,
                        help="Detection confidence threshold")
    args = parser.parse_args()
    process_dataset(args)

Running It

Label all cars in YOLO format:

python batch_label.py \
    --images ./dataset/images/train \
    --output ./dataset/labels/train \
    --prompt "car" \
    --class-id 0 \
    --format yolo \
    --threshold 0.5

Label people in COCO format:

python batch_label.py \
    --images ./dataset/images \
    --output ./dataset/annotations \
    --prompt "person" \
    --class-id 1 \
    --format coco

Multiple classes? Run the script once per class with different --class-id values, then merge the label files:

python batch_label.py --images ./data --output ./labels --prompt "car" --class-id 0
python batch_label.py --images ./data --output ./labels --prompt "person" --class-id 1
python batch_label.py --images ./data --output ./labels --prompt "bicycle" --class-id 2

For YOLO format, the script appends lines to existing .txt files, so running multiple passes naturally produces multi-class labels.

Tuning for Quality

Adjusting the Threshold

The threshold parameter controls how confident the model needs to be before reporting an instance:

Threshold	Behavior
0.3	More detections, more false positives — good for rare objects
0.5	Balanced (default) — works well for most use cases
0.7	Fewer detections, higher precision — use when false positives are costly

Prompt Engineering

SAM 3's text encoder understands natural language, so your prompts matter:

"car" — finds all cars
"red car" — finds specifically red cars
"person sitting on chair" — finds seated people (not standing ones)
"damaged road surface" — works for abstract/unusual classes too

Tip: Be specific. "dog" will find all dogs; "golden retriever" might give you better results if that's what you need.

Quality Verification

Auto-labeling isn't perfect. Here's a practical QA workflow:

Run the pipeline on your full dataset
Spot-check 50–100 random images visually
Adjust threshold if you see too many false positives or missed instances
Manual cleanup on the 5–10% of labels that need correction

This is still dramatically faster than labeling from scratch. You're correcting a few masks instead of drawing thousands.

Training with Your Auto-Generated Labels

YOLO Example

Once your labels are ready, your dataset structure should look like this:

dataset/
├── images/
│   ├── train/
│   │   ├── img001.jpg
│   │   ├── img002.jpg
│   │   └── ...
│   └── val/
│       └── ...
├── labels/
│   ├── train/
│   │   ├── img001.txt
│   │   ├── img002.txt
│   │   └── ...
│   └── val/
│       └── ...
└── data.yaml

Your data.yaml:

train: ./images/train
val: ./images/val

nc: 3  # number of classes
names: ["car", "person", "bicycle"]

Train:

yolo segment train data=data.yaml model=yolov8m-seg.pt epochs=100 imgsz=640

Mask R-CNN / Detectron2 Example

For COCO format, point Detectron2 at your annotations:

from detectron2.data import DatasetCatalog, MetadataCatalog
from detectron2.data.datasets import register_coco_instances

register_coco_instances(
    "my_dataset_train", {},
    "./dataset/annotations/annotations.json",
    "./dataset/images/train"
)

The Real-World Tradeoff: Local vs. API

Setting up SAM 3 locally works, but let's be honest about what it takes:

Factor	Local Setup	Cloud API
Hardware	Need an 8+ GB NVIDIA GPU	Just an internet connection
Setup time	30–60 min (drivers, CUDA, deps)	5 minutes (get API key)
Cost	Electricity + GPU wear	Pay per request
Speed	~2–3 sec/image (RTX 3090)	~2–3 sec/image
Maintenance	You handle updates, OOM errors, driver issues	Managed for you
Batch processing	Limited by your single GPU	Scales horizontally

For a one-off experiment with 100 images, local is fine. For production labeling pipelines, recurring projects, or teams without GPU infrastructure, an API makes more sense.

Skip the Setup: Use SegmentationAPI

If you'd rather skip the environment setup, driver debugging, and model management — SegmentationAPI.com gives you access to the same SAM 3 text-to-mask capabilities through a simple REST API.

curl -X POST https://segmentationapi.com/v1/ground \
  -H "X-API-Key: YOUR_API_KEY" \
  -F "file=@photo.jpg" \
  -F "text=person" \
  -F "threshold=0.5"

Response:

{
  "masks_b64": ["iVBORw0KGgo..."],
  "boxes": [[102, 150, 398, 612]],
  "scores": [0.94],
  "num_instances": 1
}

It supports both text prompts and interactive point/box prompts, returns masks in the same format we've been working with, and handles all the GPU infrastructure behind the scenes. There's a free playground to test it out before committing.

You can drop it straight into the batch labeling script from this article — just swap the local model call with an API request:

import requests
import base64

def predict_via_api(image_path: str, text: str, threshold: float = 0.5):
    with open(image_path, "rb") as f:
        resp = requests.post(
            "https://segmentationapi.com/v1/ground",
            headers={"X-API-Key": "YOUR_API_KEY"},
            files={"file": f},
            data={"text": text, "threshold": str(threshold)},
        )
    return resp.json()

No GPU, no CUDA, no model downloads. Same masks.

Wrapping Up

Labeling data for segmentation models used to be the bottleneck in every computer vision project. With SAM 3's text grounding, you can go from an unlabeled dataset to training-ready labels in hours instead of weeks.

The key takeaways:

SAM 3 understands text prompts and produces pixel-perfect instance masks
You can run it locally with an 8 GB+ NVIDIA GPU and a few pip installs
The batch pipeline in this article handles YOLO and COCO formats out of the box
Threshold tuning and prompt engineering get you 90%+ of the way to clean labels
Manual QA on a small subset catches the remaining edge cases

Thank you for reading!

DEV Community: Artem Zabarov