Building Synthetic Training Data Pipelines for Robotics
Real robot data collection is slow and expensive — every demonstration costs operator time and robot hours. Synthetic data pipelines let you generate large volumes of labeled training data programmatically, using simulation, domain randomization, and procedural task generation. This final tutorial in the series covers how to build one.
Why Synthetic Data, and Where It Fits
Synthetic data isn't a replacement for real demonstrations — it's a force multiplier. A typical modern pipeline mixes:
- A smaller set of real demonstrations (from the teleoperation and capture pipeline earlier in this series) for grounding
- A large volume of synthetic/simulated data for coverage and scale
- Domain randomization (previous tutorial) to keep synthetic data useful for real-world transfer
[Real Demos] -----\
+--> [Combined Dataset] --> [Training]
[Synthetic Data] --/
Step 1: Procedural Scene Generation
Rather than hand-authoring every scene, generate scene variations programmatically — object counts, positions, and combinations.
import numpy as np
import random
OBJECT_LIBRARY = ["cube", "cylinder", "sphere", "mug", "box"]
def generate_scene_config(n_objects_range=(1, 4), workspace_bounds=((0.3, 0.7), (-0.2, 0.2))):
n_objects = random.randint(*n_objects_range)
objects = []
for _ in range(n_objects):
objects.append({
"type": random.choice(OBJECT_LIBRARY),
"position": [
np.random.uniform(*workspace_bounds[0]),
np.random.uniform(*workspace_bounds[1]),
0.05,
],
"color": np.random.uniform(0, 1, 3).tolist(),
"scale": np.random.uniform(0.8, 1.2),
})
return {"objects": objects}
Step 2: Building Scenes from Configs
Translate the procedural config into your simulator's scene format. For MuJoCo, this means generating MJCF XML dynamically:
def build_mjcf_scene(scene_config, base_scene_path, output_path):
with open(base_scene_path) as f:
base_xml = f.read()
object_xml_blocks = []
for i, obj in enumerate(scene_config["objects"]):
pos = " ".join(map(str, obj["position"]))
rgba = " ".join(map(str, obj["color"])) + " 1"
object_xml_blocks.append(f"""
<body name="obj_{i}" pos="{pos}">
<freejoint/>
<geom type="{obj['type']}" size="0.02 0.02 0.02" rgba="{rgba}" mass="0.1"/>
</body>
""")
full_xml = base_xml.replace("<!-- OBJECTS -->", "\n".join(object_xml_blocks))
with open(output_path, "w") as f:
f.write(full_xml)
Step 3: Generating Labeled Trajectories with a Scripted Policy
For many manipulation tasks (pick-and-place, pushing, stacking), you can generate expert trajectories with a scripted controller rather than a human operator — a huge scalability win.
def scripted_pick_and_place(env, obj_id, target_pos):
trajectory = []
obs = env.reset()
obj_pos = env.data.xpos[obj_id]
waypoints = [
obj_pos + np.array([0, 0, 0.1]), # approach above
obj_pos, # descend to object
obj_pos + np.array([0, 0, 0.1]), # lift
target_pos + np.array([0, 0, 0.1]), # move above target
target_pos, # place
]
for i, wp in enumerate(waypoints):
gripper_cmd = 1.0 if i in (1, 4) else (0.0 if i == 2 else None)
for _ in range(20): # interpolate toward waypoint
current_pose = env.get_ee_pose()
action = interpolate_toward(current_pose, wp, gripper_cmd)
obs, _, done, info = env.step(action)
trajectory.append({"observation": obs, "action": action})
return trajectory, info.get("success", False)
Scripted policies work well for geometrically simple tasks; for anything requiring nuanced contact or force feedback, you'll still want real demonstrations as a seed, potentially combined with reinforcement learning in simulation to refine the scripted behavior.
Step 4: Large-Scale Parallel Generation
Since simulation is cheap, generate data in parallel across many environment instances:
from multiprocessing import Pool
def generate_episode(seed):
np.random.seed(seed)
scene_config = generate_scene_config()
build_mjcf_scene(scene_config, "base_scene.xml", f"/tmp/scene_{seed}.xml")
env = DomainRandomizedEnv(f"/tmp/scene_{seed}.xml")
obj_id = env.model.body("obj_0").id
target_pos = np.array([0.5, 0.3, 0.05])
trajectory, success = scripted_pick_and_place(env, obj_id, target_pos)
return {"trajectory": trajectory, "success": success, "seed": seed}
def generate_dataset_parallel(n_episodes, n_workers=8):
with Pool(n_workers) as pool:
results = pool.map(generate_episode, range(n_episodes))
return results
On a decent workstation, this can produce thousands of episodes in the time a single teleoperation session would take to record a few dozen.
Step 5: Filtering and Balancing the Synthetic Dataset
Not all generated episodes are useful. Filter aggressively and balance the dataset before training:
def filter_and_balance(episodes, min_success_rate_per_bucket=None):
successful = [e for e in episodes if e["success"]]
print(f"Kept {len(successful)}/{len(episodes)} successful episodes")
# Optional: bucket by scene complexity (e.g., number of objects) and balance
buckets = {}
for e in successful:
n_obj = len(e.get("scene_config", {}).get("objects", [1]))
buckets.setdefault(n_obj, []).append(e)
min_bucket_size = min(len(v) for v in buckets.values())
balanced = []
for v in buckets.values():
balanced.extend(random.sample(v, min_bucket_size))
return balanced
Step 6: Mixing Synthetic and Real Data for Training
Combine both sources into a single dataloader, optionally with a weighting factor to control the real/synthetic ratio per batch:
from torch.utils.data import ConcatDataset, WeightedRandomSampler
def build_mixed_dataloader(real_dataset, synthetic_dataset, real_weight=3.0, batch_size=64):
combined = ConcatDataset([real_dataset, synthetic_dataset])
weights = (
[real_weight] * len(real_dataset) +
[1.0] * len(synthetic_dataset)
)
sampler = WeightedRandomSampler(weights, num_samples=len(combined), replacement=True)
return torch.utils.data.DataLoader(combined, batch_size=batch_size, sampler=sampler)
Upweighting real data (even though it's the minority by volume) is a common and effective trick — it keeps the policy grounded in real dynamics and visuals while still benefiting from synthetic data's scale and diversity.
Practical Tips
- Track provenance. Tag every episode with whether it's real, synthetic-scripted, or synthetic-RL-generated — this makes later ablations ("does removing synthetic data hurt?") tractable.
- Watch for synthetic data collapse. If your scripted policy always solves tasks the same way, your "large" synthetic dataset may have far less effective diversity than its episode count suggests. Measure diversity (e.g., variance in trajectories, object configurations), not just volume.
- Re-validate periodically against real data. As you scale synthetic generation, periodically check that a policy trained purely on synthetic data still transfers — this is your canary for sim-to-real drift creeping back in.
Wrapping Up
Across this second batch of tutorials, we've gone from testing policies in MuJoCo, to closing the sim-to-real gap, to using domain randomization for robustness, to scaling training data with synthetic generation. Combined with the teleoperation, demonstration capture, imitation learning, and sensor recording pipelines from the first series, this covers the full data and simulation infrastructure behind a modern physical AI robotics stack.
Useful Links
Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx
Top comments (0)