Building Synthetic Training Data Pipelines for Robotics

Real-world robot data is expensive to collect and label. Synthetic data pipelines — generating training data directly from simulation, complete with automatic ground-truth labels — let you produce far larger and more diverse datasets than manual collection ever could, for a fraction of the cost. This tutorial covers designing and building a synthetic data pipeline for robotics, from scene generation through to a usable, labeled dataset.

Where synthetic data fits in a robot learning pipeline

Synthetic data pipelines are most valuable for:

• Perception training — object detection, segmentation, pose estimation, and grasp point prediction all benefit enormously from large volumes of perfectly labeled data, which is exactly what simulation provides for free.
• Bootstrapping policies before real demonstrations exist — pretraining on synthetic rollouts, then fine-tuning on a smaller set of real demonstrations (the same pattern discussed in the sim-to-real tutorial).
• Rare or dangerous scenarios — edge cases that are hard or unsafe to reproduce repeatedly on real hardware (near-collisions, extreme object configurations, sensor failure modes) are trivial to generate synthetically.

It's not a full replacement for real data — visual and dynamics gaps mean synthetic-only training rarely matches real-data performance for the hardest tasks — but as a large, cheap, well-labeled complement to a smaller real dataset, it's extremely effective.

Pipeline architecture

Scene Generation -> Domain Randomization -> Rendering/Simulation -> Ground-Truth Extraction -> Dataset Export

Each stage is a distinct, testable component, which matters because synthetic pipelines tend to run at large scale (thousands to millions of samples) — a bug caught late costs far more compute to regenerate than one caught early.

Step 1: Scene generation

Programmatically assemble scenes rather than hand-building each one. A scene generator typically:

• Samples a task-relevant object (or set of objects) from an asset library.
• Places it at a randomized, but physically valid, pose (checking for collisions/overlaps).
• Samples a background/environment configuration.
• Places the robot and camera(s) according to the deployment configuration (with some randomization, per the domain randomization tutorial).

import random

def generate_scene(asset_library, rng):
    obj = rng.choice(asset_library)
    pose = sample_valid_pose(obj, rng)
    background = rng.choice(BACKGROUND_LIBRARY)
    return {
        "object": obj,
        "object_pose": pose,
        "background": background,
    }

def sample_valid_pose(obj, rng, max_attempts=20):
    for _ in range(max_attempts):
        pose = rng.uniform(WORKSPACE_MIN, WORKSPACE_MAX)
        if not check_collision(obj, pose):
            return pose
    raise RuntimeError("Failed to sample a valid pose")

Step 2: Domain randomization as a data augmentation strategy

For synthetic data generation specifically, domain randomization does double duty: it improves sim-to-real transfer (as covered previously) and it's your primary mechanism for dataset diversity, replacing the manual scene variation a human data collector would otherwise need to provide. Apply the same categories covered in the domain randomization tutorial — lighting, textures, camera pose, distractors — but here, treat "diversity of the generated dataset" as the explicit success metric, not just "policy robustness."

Step 3: Rendering and extracting ground truth

This is where synthetic pipelines earn their value: ground-truth labels that would require expensive manual annotation in the real world come for free from the simulator's internal state.

def render_and_label(sim, scene):
    rgb = sim.render_rgb()
    depth = sim.render_depth()
    segmentation = sim.render_instance_segmentation()

    bbox_2d = compute_2d_bbox(segmentation, target_id=scene["object"].id)
    pose_6d = sim.get_object_pose(scene["object"].id)  # ground-truth 6-DOF pose
    grasp_points = compute_grasp_candidates(scene["object"])

    return {
        "rgb": rgb,
        "depth": depth,
        "segmentation": segmentation,
        "bbox_2d": bbox_2d,
        "pose_6d": pose_6d,
        "grasp_points": grasp_points,
    }

Common label types worth extracting depending on your downstream task: 2D/3D bounding boxes, instance/semantic segmentation masks, 6-DOF object pose, depth maps, surface normals, keypoints, and grasp candidate annotations. Since these all come from querying the simulator's internal state rather than from a human annotator, generating additional label types later costs essentially nothing compared to going back and re-annotating a real dataset.

Step 4: Dataset export

Export to a format your training pipeline (and ideally, standard tooling) can consume directly. Common choices:

• COCO format for detection/segmentation tasks — widely supported by existing training frameworks.
• A custom schema mirroring your real-data format — often the better choice, since it lets you mix synthetic and real samples in the same training pipeline without a translation layer.

import json

def export_sample(sample, output_dir, sample_id):
    image_path = f"{output_dir}/images/{sample_id}.png"
    save_image(sample["rgb"], image_path)

    annotation = {
        "image": image_path,
        "bbox_2d": sample["bbox_2d"],
        "pose_6d": sample["pose_6d"].tolist(),
        "grasp_points": sample["grasp_points"],
    }
    with open(f"{output_dir}/annotations/{sample_id}.json", "w") as f:
        json.dump(annotation, f)

Scaling generation

Because each sample is independent, synthetic data generation parallelizes naturally across processes or machines:

from multiprocessing import Pool

def generate_one_sample(args):
    seed, output_dir, idx = args
    rng = np.random.default_rng(seed)
    scene = generate_scene(ASSET_LIBRARY, rng)
    sim = setup_simulation(scene)
    sample = render_and_label(sim, scene)
    export_sample(sample, output_dir, idx)

def generate_dataset(num_samples, output_dir, num_workers=8):
    args = [(i, output_dir, i) for i in range(num_samples)]
    with Pool(num_workers) as pool:
        pool.map(generate_one_sample, args)

At real scale (hundreds of thousands of samples), this typically moves to a distributed job queue (e.g., across a cluster or cloud batch jobs) rather than a single machine's multiprocessing pool, but the underlying per-sample independence is what makes that scaling straightforward in the first place.

Validating the synthetic dataset

Synthetic data pipelines fail silently more often than real data collection does — a bug in your labeling code produces plausible-looking but wrong labels, and nothing about the pipeline will complain. Build validation in from the start:

• Visualize a random sample of generated images with labels overlaid (bounding boxes, segmentation masks, projected 6-DOF pose axes) every time you change the pipeline, not just once at the beginning.
• Check label statistics — distribution of object poses, bounding box sizes, and class balance — for unexpected clustering that suggests a sampling bug rather than genuine diversity.
• Train a quick baseline model on a small synthetic subset and sanity-check its behavior before committing to generating the full dataset at scale.

Mixing synthetic and real data

The most effective real-world pipelines rarely use synthetic data alone. A common and effective pattern:

1. Generate a large, diverse synthetic dataset covering broad variation.
2. Collect a smaller real dataset (using the teleoperation and demonstration capture pipelines covered earlier in this series) covering the specific deployment conditions.
3. Either pretrain on synthetic data and fine-tune on real data, or mix both into a single training set with real data oversampled relative to its natural proportion, since it's typically more valuable per-sample for closing the sim-to-real gap.

Closing the loop

This tutorial completes the simulation side of the pipeline: Isaac Sim and MuJoCo for building and testing policies, sim-to-real and domain randomization techniques for making those policies transfer, and synthetic data generation for scaling training data cheaply. Combined with the teleoperation, demonstration capture, and imitation learning tutorials earlier in this series, you now have both the real and synthetic halves of a complete robot learning pipeline.

---

Useful Links

Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx