DEV Community: vmodal_ai

Building a Vision-Language-Action Robot System from Scratch

vmodal_ai — Wed, 02 Sep 2026 22:20:43 +0000

Building a Vision-Language-Action Robot System from Scratch

Vision-Language-Action (VLA) systems connect visual perception, natural-language instructions, and physical robot actions.

Instead of requiring a user to specify low-level commands such as:

Move forward 2 meters.
Turn left 90 degrees.
Open gripper.

a VLA system can accept a higher-level instruction:

"Pick up the red box and place it on the table."

The system then translates that intent into a sequence of robot actions.

VLA Architecture

             User Instruction
                    ↓
             Language Model
                    ↓
             Task Planner
                    ↓
        ┌───────────┴───────────┐
        ↓                       ↓
   Vision System           World Model
        └───────────┬───────────┘
                    ↓
             Action Planner
                    ↓
              Safety Layer
                    ↓
                Control
                    ↓
                Robot
                    ↑
                 Sensors

The most important design principle is that language and vision models should not directly bypass the robot's safety and control layers.

1. Define the Robot Action Space

Start with a small set of deterministic actions:

navigate(location)
look_at(object)
pick(object)
place(location)
open_gripper()
close_gripper()
stop()

This is safer than allowing a model to generate arbitrary motor commands.

2. Capture Visual Input

A camera publishes frames:

/camera/image_raw

A vision pipeline can detect:

red_box
table
person
floor
obstacle

The perception system should transform raw images into structured objects.

Example:

{
  "objects": [
    {
      "name": "red_box",
      "confidence": 0.95,
      "position": [1.2, 0.4, 0.8]
    }
  ]
}

3. Convert Language into a Task

The language model interprets:

"Pick up the red box."

into a structured task:

{
  "goal": "pick",
  "object": "red_box"
}

Avoid passing free-form model output directly to robot hardware.

4. Ground Language in the World

Language references must be connected to visual objects.

"red box"
    ↓
Language Entity
    ↓
Vision Detections
    ↓
Object ID: object_17
    ↓
3D Position

This process is often called grounding.

A robust system should consider ambiguity:

User: "Pick up the box."

Detected:
- box_01
- box_02

The system should request clarification or use an explicit selection policy rather than guessing when the consequences are significant.

5. Build a Task Planner

Break a high-level task into executable steps.

Pick up red box
      ↓
Navigate to box
      ↓
Locate box
      ↓
Approach box
      ↓
Align gripper
      ↓
Close gripper
      ↓
Verify grasp
      ↓
Navigate to table
      ↓
Place box
      ↓
Verify placement

The planner can be implemented using state machines, behavior trees, or another explicit orchestration mechanism.

6. Integrate ROS 2

ROS 2 provides the communication layer between perception, planning, and control components.

Example:

VLA Node
   ↓
/task_goal
   ↓
Task Planner
   ↓
/navigation_goal
   ↓
Navigation
   ↓
/cmd_vel
   ↓
Robot Base

For long-running operations such as navigation, ROS 2 actions provide goal, feedback, result, and cancellation semantics. ROS 2 actions

7. Add a Safety Layer

The VLA model may propose:

{
  "action": "move",
  "velocity": 4.0
}

The safety layer should reject invalid values:

def validate_action(action):
    if action.velocity > MAX_SAFE_SPEED:
        return False

    if emergency_stop_active():
        return False

    return True

The model should never be the final authority over physical safety.

8. Add Action Verification

After every important action, verify the result.

Pick Action
    ↓
Gripper Close
    ↓
Visual Verification
    ↓
Object Lifted?
   ├── Yes → Continue
   └── No  → Retry / Recover

This creates a closed-loop VLA system.

9. Handle Uncertainty

Vision and language models are probabilistic.

Represent confidence explicitly:

Object confidence: 0.94
Language grounding: 0.91
Grasp confidence: 0.72

Use thresholds and fallback behaviors appropriate to the task.

10. Keep the VLA Layer Replaceable

A clean architecture allows the language or vision model to change:

                 ┌───────────────┐
Camera ─────────→│ Vision Model  │
                 └───────┬───────┘
                         ↓
                   World Model
                         ↑
                 ┌───────┴───────┐
User ───────────→│ Language Model│
                 └───────────────┘
                         ↓
                   Task Planner
                         ↓
                   ROS 2 Actions
                         ↓
                     Control

The rest of the robot should not depend on a specific model vendor or model architecture.

11. Example End-to-End Flow

Suppose the user says:

"Take the red box to the charging station."

The pipeline becomes:

1. Speech / Text Input
2. Language Understanding
3. Identify "red box"
4. Vision Detection
5. Ground Language → Object
6. Generate Task Plan
7. Navigate to Object
8. Approach Object
9. Grasp Object
10. Verify Grasp
11. Navigate to Charging Station
12. Place Object
13. Verify Result
14. Report Completion

This is substantially more robust than:

Prompt → LLM → Motor Commands

12. Production Considerations

A real VLA robot should include:

Model versioning
Prompt/version management
Sensor synchronization
Timeouts
Safety watchdogs
Human override
Action validation
Audit logs
Simulation tests
Failure recovery
Offline fallback behaviors

For mobile robots, Nav2 can provide the navigation layer beneath the higher-level task planner. Nav2

13. Build Incrementally

A practical development roadmap is:

Phase 1
Vision → Object Detection

Phase 2
Language → Structured Tasks

Phase 3
Task → Deterministic Robot Actions

Phase 4
Vision + Language Grounding

Phase 5
Closed-Loop Verification

Phase 6
Safety + Recovery

Phase 7
Production Deployment

Do not start with a fully autonomous general-purpose robot. Start with a small, measurable action space and expand it gradually.

Conclusion

A Vision-Language-Action system is best understood as a layered robotics architecture in which AI provides perception and high-level reasoning while deterministic robotics software handles planning, validation, safety, and control.

The result is a robot that can understand natural-language goals, connect those goals to what its sensors observe, execute physical actions, and verify whether those actions actually worked.

Useful Links

Website: www.v-modal.com
SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
SDK Android: https://github.com/v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Integrating Perception, Planning, and Control in Autonomous Robots

vmodal_ai — Wed, 02 Sep 2026 22:20:08 +0000

Integrating Perception, Planning, and Control in Autonomous Robots

Autonomous robots need to continuously answer three questions:

What is around me? — Perception
What should I do? — Planning
How do I physically do it? — Control

These systems become powerful when integrated into a closed feedback loop.

The Autonomy Loop

       ┌───────────────┐
       │    Sensors    │
       └───────┬───────┘
               ↓
       ┌───────────────┐
       │  Perception   │
       └───────┬───────┘
               ↓
       ┌───────────────┐
       │ World / State │
       └───────┬───────┘
               ↓
       ┌───────────────┐
       │   Planning    │
       └───────┬───────┘
               ↓
       ┌───────────────┐
       │   Control     │
       └───────┬───────┘
               ↓
            Robot
               │
               └──── feedback ────→ Sensors

1. Perception

Perception converts sensor measurements into estimates.

For example:

Camera Frame
     ↓
Object Detector
     ↓
Person: 0.96
Box: 0.91
Chair: 0.87

For navigation, perception may generate obstacles:

Obstacle A → x=2.0, y=0.5
Obstacle B → x=3.2, y=-1.1

Always associate detections with timestamps and, where possible, uncertainty.

2. State Estimation

Planning should not consume isolated detections. It needs a coherent state.

Combine:

IMU
Wheel Odometry
GPS
LiDAR
Camera
     ↓
Sensor Fusion
     ↓
Robot Pose + Velocity

The state estimator should continuously update the robot's belief about its position and motion.

3. Planning

Planning transforms the current state and goal into a desired behavior.

State + Goal + Environment
            ↓
         Planner
            ↓
      Desired Path

For navigation systems, Nav2 provides planning and control infrastructure for ROS 2 robots. Nav2 documentation

4. Control

The controller follows the planned trajectory.

For a differential-drive robot:

Desired Path
     ↓
Controller
     ↓
linear velocity
angular velocity
     ↓
Motor Driver

A controller should operate at a predictable frequency and enforce physical limits.

5. Connect Everything with ROS 2

A simplified ROS graph could be:

camera_node
     ↓
/image
     ↓
perception_node
     ↓
/detections
     ↓
world_model_node
     ↓
/world_state
     ↓
planner_node
     ↓
/planned_path
     ↓
controller_node
     ↓
/cmd_vel
     ↓
base_controller

ROS 2 topics are appropriate for continuous streams such as sensor data and robot state. Actions are useful for long-running navigation or other behaviors requiring feedback. ROS 2 interfaces

6. Avoid Blocking the Control Loop

Suppose object detection takes 200 ms.

Do not make the motor control loop wait for inference:

Bad:
Control → wait for AI → Control

Instead:

Perception: asynchronous
Planning: asynchronous
Control: deterministic

The controller should continue operating with the latest valid state.

7. Add Confidence and Freshness

A perception result should have:

value
confidence
timestamp
source

For example:

Detection(
    label="person",
    confidence=0.94,
    timestamp=123456789
)

Reject stale information when it is no longer safe to use.

8. Add Recovery Behaviors

Autonomous systems should expect failures.

Examples:

Path blocked
   ↓
Replan
   ↓
Still blocked?
   ↓
Recovery behavior
   ↓
Stop / Request Help

Recovery behavior is part of autonomy—not an afterthought.

9. Example Control Pipeline

A mobile robot can implement:

def control(state, path):
    target = select_target(path, state.pose)

    linear = compute_linear_velocity(state, target)
    angular = compute_angular_velocity(state, target)

    linear = clamp(linear, -MAX_V, MAX_V)
    angular = clamp(angular, -MAX_W, MAX_W)

    return linear, angular

The controller receives a safe representation of the planned path and produces bounded commands.

10. Measure End-to-End Latency

Measure:

Sensor timestamp
      ↓
Perception timestamp
      ↓
Planning timestamp
      ↓
Control timestamp
      ↓
Actuator timestamp

The important metric is not just model inference time. It is the age of the information when the actuator receives the command.

11. Simulation and Testing

Start with recorded data or simulation:

Recorded Sensors
       ↓
Perception
       ↓
Planning
       ↓
Control
       ↓
Simulation

Then move to:

Simulation → Hardware-in-the-Loop → Real Robot

Test edge cases such as missing detections, moving obstacles, localization loss, and planner failures.

Conclusion

Perception, planning, and control should be treated as a continuous feedback system.

Perception tells the robot what it believes is happening. Planning chooses an appropriate response. Control turns that response into physical motion. The loop repeats as new sensor information arrives.

The key engineering principle is to keep the control layer predictable while allowing AI-heavy perception and reasoning components to evolve independently.

Useful Links

Website: www.v-modal.com
SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
SDK Android: https://github.com/v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Creating a Robot Sensor Data Recording and Replay System

vmodal_ai — Wed, 02 Sep 2026 22:19:44 +0000

Creating a Robot Sensor Data Recording and Replay System

A robust recording-and-replay system is the backbone of any serious robot learning workflow. Recording lets you build datasets and debug incidents after the fact; replay lets you re-run recorded sensor streams through your perception or control stack without needing the physical robot, which massively speeds up development and debugging. This tutorial covers building both halves as a cohesive system.

Why replay matters as much as recording

It's easy to treat recording as "just logging," but a good replay system pays for itself quickly:

Debugging without hardware: reproduce a bug from last week's session without booking robot time.
Regression testing: replay a fixed set of recorded sessions through a new version of your perception pipeline and diff the outputs.
Dataset iteration: re-extract features, re-label, or re-sample previously recorded sessions as your pipeline evolves, without recollecting data.
Simulation grounding: compare simulated sensor output against real recorded sensor output for the same nominal trajectory.

Core design: a session as a set of synchronized streams

Model a recording session as multiple independent streams (camera, joint states, IMU, force-torque, etc.), each with its own timestamped samples, plus a session-level index that lets you query "what did every stream look like at time T."

Session/
  metadata.json          # session info: robot config, sensors present, start time
  streams/
    camera_wrist/
      timestamps.npy
      frames/            # or a video file + frame index
    joint_states/
      timestamps.npy
      data.npy
    gripper/
      timestamps.npy
      data.npy

The recorder

The recorder's job is simple in principle: subscribe to every sensor stream, timestamp each sample, and write it to disk without blocking the control loop.

import threading
import queue
import time

class StreamRecorder:
    def __init__(self, name, writer):
        self.name = name
        self.writer = writer
        self.queue = queue.Queue()
        self.thread = threading.Thread(target=self._worker, daemon=True)
        self.running = False

    def start(self):
        self.running = True
        self.thread.start()

    def record(self, data):
        # Called from the sensor callback thread — must be fast and non-blocking
        self.queue.put((time.time(), data))

    def _worker(self):
        while self.running or not self.queue.empty():
            try:
                timestamp, data = self.queue.get(timeout=0.1)
                self.writer.write(self.name, timestamp, data)
            except queue.Empty:
                continue

    def stop(self):
        self.running = False
        self.thread.join()

The key design point: record() is called from whatever thread the sensor callback lives on, and it must return immediately. All the actual disk I/O happens on a dedicated writer thread per stream. This decoupling is what prevents a slow disk write from stalling your control loop — a very common source of "why does my robot judder every few seconds" bugs.

Handling different sensor rates cleanly

Cameras, joint encoders, and force-torque sensors rarely run at the same rate. Rather than forcing everything onto one clock during recording, record each stream at its native rate with accurate timestamps, and defer alignment to query time:

import numpy as np

def get_nearest_sample(timestamps: np.ndarray, data: np.ndarray, query_time: float):
    idx = np.searchsorted(timestamps, query_time)
    idx = np.clip(idx, 1, len(timestamps) - 1)
    before, after = idx - 1, idx
    if abs(timestamps[before] - query_time) < abs(timestamps[after] - query_time):
        return data[before]
    return data[after]

This "query by nearest timestamp, per stream" pattern is more flexible than forcing a single global sample rate at recording time — it lets you resample to whatever rate a downstream consumer (a training pipeline, a replay tool) actually needs, without having thrown away information at recording time.

Building the replay engine

A replay engine reconstructs a session and plays it back through the same interfaces your live system uses, so your downstream code doesn't need separate "live" and "replay" code paths.

class SessionReplayer:
    def __init__(self, session_path):
        self.streams = self._load_streams(session_path)
        self.start_time = min(s["timestamps"][0] for s in self.streams.values())
        self.end_time = max(s["timestamps"][-1] for s in self.streams.values())

    def _load_streams(self, session_path):
        # Load timestamps + data arrays per stream from disk
        raise NotImplementedError

    def get_state_at(self, t):
        return {
            name: get_nearest_sample(s["timestamps"], s["data"], t)
            for name, s in self.streams.items()
        }

    def play(self, callback, speed=1.0, dt=0.033):
        t = self.start_time
        wall_start = time.time()
        while t <= self.end_time:
            state = self.get_state_at(t)
            callback(state, t)

            t += dt * speed
            target_wall_time = wall_start + (t - self.start_time) / speed
            sleep_time = target_wall_time - time.time()
            if sleep_time > 0:
                time.sleep(sleep_time)

By designing the callback(state, t) interface to match what your live perception/control code expects as input, you can point the exact same downstream code at either live sensors or a replayed session — this is the single biggest productivity win of building a proper replay system rather than ad hoc debug scripts.

Practical tips for a system that scales

Compress images as you go, not after the fact — writing raw uncompressed frames during a live session will exhaust disk bandwidth quickly on any camera above VGA resolution. JPEG or a lightweight video codec (H.264) with a keyframe interval short enough for random access works well.
Write an index file per session (start/end time, sensors present, robot config, any metadata like task label or operator id) so you can filter and search sessions without opening every stream file.
Version your schema. Sensor configurations change over a project's lifetime — added cameras, changed resolutions, new sensors. Store a schema version in each session's metadata so your loader can handle old and new sessions correctly.
Validate on write, not just on read. Catch dropped frames, clock jumps, and sensor disconnects at recording time when you can still restart the session, rather than discovering the corruption weeks later during training.

Where this fits in the bigger picture

This recording-and-replay system is the shared infrastructure underneath every other tutorial in this series — teleoperation sessions, human demonstrations, and imitation learning training data are all, at the storage layer, just sessions recorded and replayed through this same system. Getting this layer right early pays dividends across the entire robot learning pipeline.

Useful Links

Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Using VR Controllers for Robot Teleoperation

vmodal_ai — Wed, 02 Sep 2026 22:19:40 +0000

Using VR Controllers for Robot Teleoperation

VR controllers (Meta Quest, HTC Vive, Valve Index) have become a popular teleoperation input for robot learning, and for good reason: they provide full 6-DOF pose tracking, intuitive hand-based control, and built-in buttons/triggers that map naturally onto gripper control — all in a consumer-grade, relatively cheap package. This tutorial covers how to set up VR controllers as a teleoperation input for a robot arm.

Why VR controllers work well for teleoperation

Compared to a gamepad or keyboard, a VR controller gives you:

Continuous 6-DOF pose (position + orientation) tracked at high frequency, which maps intuitively to end-effector pose control.
Natural hand motion, so operators produce trajectories that look like real human manipulation rather than artificial waypoint sequences.
Built-in analog trigger, perfect for continuous gripper open/close control instead of a binary toggle.
Haptic feedback (on some controllers), useful for signaling contact or constraint violations back to the operator.

Getting controller pose data

Most VR SDKs (OpenXR, Meta's Oculus SDK, SteamVR) expose controller pose as a position + quaternion relative to the headset's tracking origin, updated at high frequency (often 60–120 Hz). A minimal OpenXR-style polling loop looks like:

class VRControllerInput:
    def __init__(self, xr_session, controller_path):
        self.session = xr_session
        self.controller_path = controller_path

    def read(self):
        pose = self.session.get_controller_pose(self.controller_path)
        trigger_value = self.session.get_input_value(self.controller_path, "trigger")
        grip_button = self.session.get_input_value(self.controller_path, "grip")
        return {
            "position": pose.position,       # (x, y, z)
            "orientation": pose.orientation, # quaternion (x, y, z, w)
            "trigger": trigger_value,        # 0.0 - 1.0, maps to gripper
            "grip_button": grip_button,      # engage/disengage teleop
        }

If you're using Unity or Unreal for the VR side and streaming to a Python robot controller, this same pose data typically gets sent over a lightweight transport like ZeroMQ, gRPC, or a WebSocket, since VR engines and robotics stacks (ROS, Python control loops) usually don't live in the same process.

Mapping controller pose to robot end-effector pose

The core challenge is retargeting: the VR controller's coordinate frame and workspace scale rarely match the robot's. A typical approach:

Establish a reference frame. When the operator engages the grip button, record the controller's current pose as the origin.
Compute relative motion. For every subsequent frame, compute the controller's pose relative to that origin, not its absolute pose.
Apply a scale factor. Human arm movement range is often larger than the robot's comfortable workspace (or vice versa), so scale the relative translation before applying it.
Add the relative motion to the robot's pose at engagement time, producing the new target pose.

import numpy as np
from scipy.spatial.transform import Rotation as R

class VRToRobotMapper:
    def __init__(self, position_scale=1.0):
        self.position_scale = position_scale
        self.engaged = False
        self.origin_controller_pos = None
        self.origin_controller_rot = None
        self.origin_robot_pos = None
        self.origin_robot_rot = None

    def engage(self, controller_pos, controller_rot, robot_pos, robot_rot):
        self.engaged = True
        self.origin_controller_pos = np.array(controller_pos)
        self.origin_controller_rot = R.from_quat(controller_rot)
        self.origin_robot_pos = np.array(robot_pos)
        self.origin_robot_rot = R.from_quat(robot_rot)

    def disengage(self):
        self.engaged = False

    def map(self, controller_pos, controller_rot):
        if not self.engaged:
            return None

        delta_pos = (np.array(controller_pos) - self.origin_controller_pos) * self.position_scale
        target_pos = self.origin_robot_pos + delta_pos

        delta_rot = self.origin_controller_rot.inv() * R.from_quat(controller_rot)
        target_rot = self.origin_robot_rot * delta_rot

        return target_pos, target_rot.as_quat()

This "clutch" pattern — engage, move, disengage, reposition your hand, re-engage — mirrors how a mouse works when you lift it and reposition mid-drag. It's essential once the human's comfortable arm range doesn't match the robot's workspace, which is almost always the case.

Gripper control from the trigger

Map the analog trigger value directly to a gripper closure percentage rather than treating it as a binary switch — this preserves fine-grained grasp force control, which is valuable both for the task itself and, later, as a richer training signal for imitation learning:

def trigger_to_gripper_command(trigger_value, min_open=0.0, max_open=1.0):
    # trigger_value: 0.0 (released) to 1.0 (fully pressed)
    closure = trigger_value  # 0 = open, 1 = fully closed
    return min_open + closure * (max_open - min_open)

Handling orientation carefully

A common bug: naively slerp-ing or averaging quaternions without checking for the double-cover property of quaternions (q and -q represent the same rotation) causes sudden 180-degree "flips" in the mapped orientation. Always normalize sign consistency before interpolating, or use rotation matrices/axis-angle deltas if you're not confident in your quaternion math.

Latency and jitter considerations

VR tracking is generally very good, but a few things to watch for:

Filter trigger and pose noise with a light exponential moving average — raw VR tracking data can have small high-frequency jitter that's imperceptible to the eye in a headset but shows up clearly in logged action data.
Account for network latency if the VR engine and robot controller run in separate processes or machines — a delay of even 50–100 ms is noticeable as sluggish control and will show up as lag between the logged input and the resulting robot motion.
Decouple render rate from control rate. The VR headset render loop typically runs faster than your robot control loop needs; sample the controller pose at your fixed control rate rather than trying to match VR frame rate exactly.

Combining with the rest of the pipeline

Everything from the general teleoperation and demonstration-capture tutorials earlier in this series applies directly here — the VR controller is simply a richer, more intuitive input device sitting at the top of the same architecture. Once your mapping and gripper control are solid, this setup tends to produce noticeably smoother, more natural demonstrations than joystick- or keyboard-based teleoperation, which pays off directly in imitation learning policy quality.

Useful Links

Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Building an Imitation Learning Pipeline for Robotic Manipulation

vmodal_ai — Wed, 02 Sep 2026 22:19:36 +0000

Building an Imitation Learning Pipeline for Robotic Manipulation

With a validated demonstration dataset in hand, the next step is building the actual training pipeline: turning (observation, action) pairs into a policy that can control the robot on its own. This tutorial covers the end-to-end pipeline — data loading, model architecture choices, training, and evaluation — for behavior-cloning-style imitation learning on manipulation tasks.

Pipeline overview

Raw Episodes -> Preprocessing -> Dataset/Dataloader -> Policy Model -> Training Loop -> Evaluation

Each stage has decisions that materially affect final policy quality, so it's worth treating this as a real pipeline with clear interfaces, not a single monolithic script.

Step 1: Preprocessing

Before training, normalize your data:

Action normalization: scale actions to roughly [-1, 1] using dataset statistics (mean/std or min/max). Unnormalized actions with wildly different scales across joints slow down and destabilize training.
Image preprocessing: resize to a consistent resolution, normalize pixel values, and optionally apply light augmentation (color jitter, random crop) to improve robustness — but avoid augmentations that break task-relevant geometry, like aggressive rotation for tasks sensitive to orientation.
Action representation: decide between predicting absolute targets vs. delta actions vs. velocity commands. Delta actions relative to the current state are usually easier to learn and more robust to compounding drift at inference time.

import numpy as np

class ActionNormalizer:
    def __init__(self, actions: np.ndarray):
        self.mean = actions.mean(axis=0)
        self.std = actions.std(axis=0) + 1e-6

    def normalize(self, action):
        return (action - self.mean) / self.std

    def denormalize(self, action):
        return action * self.std + self.mean

Step 2: Dataset and dataloader

Structure your dataset so each sample is a short window of context, not just a single frame — most manipulation policies benefit from a few frames of history to disambiguate velocity and resolve partial occlusion.

from torch.utils.data import Dataset

class ManipulationDataset(Dataset):
    def __init__(self, episodes, obs_horizon=2, action_horizon=8):
        self.episodes = episodes
        self.obs_horizon = obs_horizon
        self.action_horizon = action_horizon
        self.index = self._build_index()

    def _build_index(self):
        index = []
        for ep_i, ep in enumerate(self.episodes):
            for t in range(self.obs_horizon, len(ep) - self.action_horizon):
                index.append((ep_i, t))
        return index

    def __len__(self):
        return len(self.index)

    def __getitem__(self, idx):
        ep_i, t = self.index[idx]
        ep = self.episodes[ep_i]
        obs_seq = ep[t - self.obs_horizon : t]
        action_seq = ep[t : t + self.action_horizon]
        return {
            "images": [s["observation"]["image"] for s in obs_seq],
            "joint_state": obs_seq[-1]["observation"]["joint_state"],
            "actions": [s["action"] for s in action_seq],
        }

Predicting a short sequence of future actions (action chunking) rather than a single next action, and executing several before re-planning, tends to produce noticeably smoother behavior than pure single-step prediction — this is one of the more impactful architectural choices in recent manipulation policy designs.

Step 3: Model architecture

A reasonable baseline architecture for image-based manipulation:

A visual encoder (a small CNN or a pretrained ResNet backbone) processes each camera frame into a feature vector.
Visual features are concatenated with proprioceptive state (joint positions/velocities).
A sequence model (transformer or a simple MLP with the obs_horizon flattened) maps the combined features to a predicted action chunk.

import torch
import torch.nn as nn

class ManipulationPolicy(nn.Module):
    def __init__(self, visual_encoder, state_dim, action_dim, action_horizon):
        super().__init__()
        self.visual_encoder = visual_encoder
        feature_dim = visual_encoder.output_dim + state_dim
        self.head = nn.Sequential(
            nn.Linear(feature_dim, 512),
            nn.ReLU(),
            nn.Linear(512, action_dim * action_horizon),
        )
        self.action_dim = action_dim
        self.action_horizon = action_horizon

    def forward(self, image, joint_state):
        visual_feat = self.visual_encoder(image)
        combined = torch.cat([visual_feat, joint_state], dim=-1)
        out = self.head(combined)
        return out.view(-1, self.action_horizon, self.action_dim)

For more advanced setups, diffusion-based action heads (predicting the action chunk via denoising) tend to model multimodal human behavior — where the same task can reasonably be solved multiple ways — better than a plain MLP regression head, at the cost of more complex training and slower inference.

Step 4: Training loop

Standard supervised regression training, with mean squared error (or smooth L1) between predicted and demonstrated actions:

def train_epoch(model, dataloader, optimizer, device):
    model.train()
    total_loss = 0.0
    for batch in dataloader:
        images = batch["images"].to(device)
        joint_state = batch["joint_state"].to(device)
        target_actions = batch["actions"].to(device)

        pred_actions = model(images, joint_state)
        loss = nn.functional.smooth_l1_loss(pred_actions, target_actions)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        total_loss += loss.item()

    return total_loss / len(dataloader)

A few practical tips:

Track validation loss on held-out episodes, not held-out frames — random frame-level splits leak information between train and val since adjacent frames are highly correlated.
Watch for the gap between training loss and real-world performance: low loss doesn't guarantee good closed-loop behavior, since imitation learning is an open-loop training objective applied to a closed-loop control problem (compounding error is the classic failure mode here).
Use early stopping based on a small set of real or simulated rollout evaluations if at all possible, not just loss curves.

Step 5: Evaluation

Loss curves only tell part of the story. Evaluate with actual closed-loop rollouts on the robot (or in simulation) and track:

Task success rate across multiple trials and initial conditions.
Recovery behavior — does the policy handle small perturbations gracefully, or does any deviation from the training distribution cause it to fail catastrophically?
Smoothness — jerky, high-frequency action outputs often indicate the policy hasn't generalized well and is essentially "hunting" between similar training examples.

Where to go from here

This pipeline covers the core behavior-cloning loop. From here, natural extensions include adding more demonstration diversity, experimenting with different action representations, or moving to more advanced input devices like VR controllers to make demonstration collection faster and more natural — which we cover next in this series.

Useful Links

Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Capturing Human Demonstrations for Imitation Learning

vmodal_ai — Wed, 02 Sep 2026 22:19:33 +0000

Capturing Human Demonstrations for Imitation Learning

Once you have a working teleoperation rig, the next challenge is turning raw teleop sessions into a clean, well-structured demonstration dataset. This is where most imitation learning projects quietly succeed or fail — a policy trained on inconsistent or noisy demonstrations will faithfully reproduce that inconsistency.

What counts as a "demonstration"

A demonstration is a single, complete episode of a task: from a defined start state to a defined end state (success, failure, or reset), recorded as a synchronized sequence of observations and actions.

Demonstration = {
  episode_id,
  timestamps[],
  observations[]  (images, joint states, proprioception),
  actions[]        (commanded joint/end-effector targets, gripper state),
  metadata          (task label, success flag, operator id)
}

Treating episodes as discrete, labeled units (rather than one long continuous log) is what makes the dataset usable for supervised learning later.

Designing your data schema before you collect anything

It's tempting to start recording immediately, but a few minutes spent on schema design saves days of reprocessing later. At minimum, decide on:

Observation space: which camera(s), resolution, joint state format (position/velocity/torque), and any additional sensors (force-torque, tactile).
Action space: joint-space targets vs. end-effector pose targets vs. delta actions. This choice has a big effect on how well imitation learning generalizes — delta (relative) actions are often more robust to small state drift than absolute targets.
Sampling rate: pick one rate for the whole pipeline (commonly 10–30 Hz for manipulation) and resample everything to it rather than mixing rates.
File format: HDF5 and per-episode directories with images-as-files plus a metadata file (e.g., a "LeRobot dataset"-style layout) are both common. HDF5 is compact and fast to load; a directory-per-episode layout is easier to inspect and debug.

Recording the episode loop

Extending the teleoperation loop from the previous tutorial, wrap it with clear episode lifecycle events:

class DemoRecorder:
    def __init__(self, teleop_session, dataset_writer):
        self.teleop = teleop_session
        self.writer = dataset_writer
        self.buffer = []

    def start_episode(self, task_label):
        self.buffer = []
        self.episode_meta = {"task": task_label, "start_time": time.time()}

    def record_step(self, obs, action):
        self.buffer.append({
            "timestamp": time.time(),
            "observation": obs,
            "action": action,
        })

    def end_episode(self, success: bool):
        self.episode_meta["success"] = success
        self.episode_meta["end_time"] = time.time()
        self.writer.write_episode(self.buffer, self.episode_meta)
        self.buffer = []

Key practice: let the operator explicitly mark success/failure at the end of each episode rather than inferring it automatically. Automated success detection is useful later for filtering, but early on, an honest human label is more trustworthy than a heuristic.

Data quality practices that matter more than volume

A smaller dataset of clean, diverse demonstrations usually outperforms a large dataset of repetitive or noisy ones. A few concrete practices:

Vary initial conditions. Randomize object position, orientation, and (if applicable) lighting between episodes. A policy trained only on one fixed starting configuration will not generalize.
Collect multiple operators if possible. A single operator's idiosyncratic style can become an artifact the policy overfits to.
Discard aborted or corrected episodes, or label them explicitly as "corrected" rather than silently keeping them — mixing recovery behavior into clean success trajectories confuses the policy about what "correct" execution looks like.
Balance the dataset across task variations. If 80% of your demonstrations are the easy variant of a task, the policy will be biased toward it.
Sanity-check episode length distribution. Extremely short or extremely long episodes (compared to the median) usually indicate a teleop glitch or an operator mistake — review before including them.

Post-hoc validation before training

Before you ever hand this dataset to a training script, run a validation pass:

def validate_episode(episode, expected_hz, image_shape):
    timestamps = [s["timestamp"] for s in episode]
    dt = [t2 - t1 for t1, t2 in zip(timestamps, timestamps[1:])]
    assert max(dt) < 2.0 / expected_hz, "Frame drop detected"

    for step in episode:
        assert step["observation"]["image"].shape == image_shape
        assert not any(map(lambda v: v != v, step["action"])), "NaN in action"

Checks worth automating: dropped frames, NaNs in joint states or actions, camera frames that are all-black or all-white (a common sign of a disconnected or misconfigured camera), and gripper state that never changes across an episode where it clearly should (often a sign of a logging bug, not real behavior).

Visualizing before trusting

Always render a handful of random episodes back as video with overlaid action values before considering a collection session "done." This catches problems no automated check will — an operator drifting off-task, an object that rolled out of frame, or a gripper that visually never closes despite the logged command saying it did.

From raw demos to a training-ready dataset

Once you have validated, well-labeled episodes, the natural next step is assembling them into the full imitation learning pipeline: splitting into train/validation sets, normalizing observations and actions, and feeding them into a policy architecture — which is exactly what we build in the next tutorial in this series.

Useful Links

Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Building a Multi-Layer Robot Software Architecture from Sensors to AI

vmodal_ai — Wed, 02 Sep 2026 22:19:32 +0000

Building a Multi-Layer Robot Software Architecture from Sensors to AI

A modern robot combines hardware drivers, sensor processing, AI models, localization, planning, control, and user applications. Without clear boundaries, these components quickly become difficult to maintain.

This tutorial shows how to organize a robot software stack from the physical sensor layer to AI-driven applications.

The Seven-Layer Model

Layer 7 — Applications / Mission
Layer 6 — AI Reasoning
Layer 5 — Planning
Layer 4 — World Model / Localization
Layer 3 — Perception
Layer 2 — ROS 2 Middleware / Interfaces
Layer 1 — Hardware & Drivers

Each layer should have a clear responsibility.

Layer 1: Hardware and Drivers

Examples include:

RGB cameras
Depth cameras
LiDAR
IMU
GPS
Wheel encoders
Motor controllers

Drivers translate hardware-specific protocols into standardized software interfaces.

Camera SDK
    ↓
Camera Driver
    ↓
ROS 2 Image Message

Layer 2: ROS 2 Communication

ROS 2 nodes communicate through a graph. Topics are useful for continuous streams, services for short request/response operations, and actions for long-running operations with feedback. ROS 2 documentation

Example graph:

camera_node
     ↓
/camera/image_raw
     ↓
perception_node
     ↓
/detections
     ↓
planner_node

This decouples producers from consumers.

Layer 3: Perception

Perception converts raw sensor signals into meaningful information.

Typical components:

Image → Object Detection
Image → Segmentation
Depth → 3D Points
LiDAR → Obstacles
IMU → Motion Information

A perception output could look conceptually like:

{
  "label": "person",
  "confidence": 0.94,
  "position": [2.1, 0.7, 0.0]
}

Do not let application code depend directly on model-specific output. Define a stable perception interface.

Layer 4: World Model and Localization

The robot needs to combine observations into a representation of the world.

Sensors
   ↓
Sensor Fusion
   ↓
Localization
   ↓
World Model

The world model may contain:

Robot pose
Occupancy information
Dynamic objects
Static landmarks
Navigation goals
Velocity
Uncertainty

This layer provides planning with structured information rather than raw sensor streams.

Layer 5: Planning

Planning answers:

What should the robot do next?

For navigation:

Current Pose + Map + Goal
             ↓
          Planner
             ↓
           Path

Nav2 is a ROS 2 navigation framework that provides components for autonomous navigation and is designed to work across different robot configurations. Nav2

Planning can also include:

Task planning
Motion planning
Manipulation planning
Behavior trees
Recovery behaviors

Layer 6: AI Reasoning

AI can operate above traditional robotics algorithms.

For example:

User: "Bring the box from the storage room."

AI Reasoner
    ↓
Task decomposition
    ↓
Navigate → Find box → Pick → Return

The AI should produce structured intentions:

{
  "task": "navigate",
  "destination": "storage_room"
}

A deterministic task executor then validates and executes that intent.

Layer 7: Application and Mission Layer

The highest layer manages user-facing workflows.

Examples:

Warehouse missions
Delivery tasks
Inspection workflows
Human-robot interaction
Fleet management

A mobile dashboard could visualize:

Robot
├── Position
├── Battery
├── Mission
├── Camera
├── Detected Objects
└── Health

This layer should not contain low-level motor control.

Define Interfaces Between Layers

A useful rule is:

Layer N should know the interface of Layer N-1,
not its implementation.

For example:

Planner → Localization API

rather than:

Planner → Specific GPS Driver

This makes components replaceable.

Example Repository Structure

robot_stack/
├── drivers/
├── interfaces/
├── perception/
├── localization/
├── world_model/
├── planning/
├── control/
├── mission/
├── ai/
├── monitoring/
└── deployment/

Testing Strategy

Test every layer independently.

Driver Tests
     ↓
Interface Tests
     ↓
Perception Tests
     ↓
Planning Tests
     ↓
Control Tests
     ↓
Integration Tests
     ↓
Simulation
     ↓
Hardware

For AI components, maintain recorded sensor datasets so model changes can be evaluated against identical inputs.

Performance Considerations

Measure:

Sensor-to-perception latency
Perception-to-planning latency
Planning duration
Control-loop jitter
Message frequency
CPU/GPU utilization
Memory usage

A robot can have highly accurate AI and still fail if decisions arrive too late.

Conclusion

Layering a robot software stack creates clear boundaries between hardware, communication, perception, planning, AI, and applications.

The most valuable benefit is replaceability. You can change a camera, AI model, planner, or user interface without rewriting the entire robot.

Useful Links

Website: www.v-modal.com
SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
SDK Android: https://github.com/v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Building a Robot Teleoperation System for Data Collection

vmodal_ai — Wed, 02 Sep 2026 22:19:29 +0000

Building a Robot Teleoperation System for Data Collection

If you're training robot policies with imitation learning, the quality of your dataset almost entirely depends on how good your teleoperation system is. A clunky, high-latency teleop rig produces jerky, inconsistent demonstrations — and your policy will happily learn those bad habits. This tutorial walks through designing and building a teleoperation system purpose-built for collecting clean, high-frequency demonstration data.

What "good" teleoperation looks like for data collection

Teleoperation for entertainment or remote inspection has different priorities than teleoperation for data collection. For ML data collection, you specifically need:

Low, consistent latency — variable lag introduces noise that looks like intentional motion to a learning algorithm.
High-frequency, synchronized logging — every joint command needs a timestamp that lines up with camera frames and sensor readings.
Smooth, continuous control — discrete or bang-bang inputs (keyboard-style) produce trajectories that don't resemble natural human motion.
Repeatability — the same operator should be able to produce similar trajectories across many trials, which matters for later behavior cloning.

System architecture

A typical teleoperation-for-data-collection stack has four layers:

Input device layer — leader arm, joystick, VR controller, or a haptic device that produces a continuous control signal.
Mapping layer — converts the input device's pose/state into a target pose or joint command for the follower robot (this is where retargeting happens if the input and robot have different kinematics).
Control layer — a low-level controller (impedance, PD, or inverse kinematics solver) that turns target commands into actuator signals safely.
Logging layer — records synchronized streams: joint states, end-effector pose, camera frames, gripper state, and the raw input signal.

[Input Device] -> [Mapping/Retargeting] -> [Robot Controller] -> [Actuators]
                                                    |
                                                    v
                                          [Synchronized Logger]
                                          (joints, images, timestamps)

Choosing your input device

Common choices, roughly in order of setup complexity:

Leader-follower arm pairs (e.g., a low-cost duplicate arm you move by hand) — gives the most natural kinesthetic feel and near 1:1 kinematic mapping.
3D mouse / SpaceMouse — cheap, precise for 6-DOF end-effector control, but less intuitive for beginners.
Gamepad — good for mobile base + simple arm tasks, poor for fine manipulation.
VR controllers — excellent for full 6-DOF pose control with intuitive hand tracking (covered in depth in a separate tutorial in this series).

For manipulation tasks specifically, leader-follower arms and VR controllers tend to produce the cleanest demonstrations because the operator's hand motion maps almost directly onto the desired end-effector motion.

Implementing the control loop

Here's a minimal Python control loop structure using an end-effector pose target and a simple IK-based controller:

import time

class TeleopSession:
    def __init__(self, input_device, robot, logger, control_hz=30):
        self.input_device = input_device
        self.robot = robot
        self.logger = logger
        self.dt = 1.0 / control_hz

    def run(self, duration_s):
        start = time.time()
        while time.time() - start < duration_s:
            loop_start = time.time()

            raw_input = self.input_device.read()
            target_pose, gripper_cmd = self.map_input(raw_input)

            joint_cmd = self.robot.inverse_kinematics(target_pose)
            self.robot.send_joint_command(joint_cmd, gripper_cmd)

            self.logger.log({
                "timestamp": loop_start,
                "target_pose": target_pose,
                "joint_state": self.robot.get_joint_state(),
                "gripper_state": gripper_cmd,
                "image": self.robot.get_camera_frame(),
            })

            elapsed = time.time() - loop_start
            time.sleep(max(0.0, self.dt - elapsed))

    def map_input(self, raw_input):
        # Retargeting logic goes here: scale, filter, clamp
        raise NotImplementedError

A few practical notes on this loop:

Run it at a fixed control rate (20–50 Hz is typical) and log the actual elapsed time per iteration, not just the nominal timestamp — drift matters for imitation learning.
Apply a low-pass filter to the raw input signal before mapping it to a target pose. Human hand tremor and sensor noise both show up as high-frequency jitter that hurts downstream policy training.
Always clamp the target pose to safe workspace bounds before sending it to the IK solver, not after — this avoids solver instability near joint limits.

Synchronizing multiple data streams

The hardest part of a teleop data collection system usually isn't the control loop — it's keeping the camera stream, joint state stream, and input stream synchronized. A few approaches:

Single-threaded polling loop (shown above) — simplest, works well if your camera and robot APIs are fast enough to poll synchronously.
Multi-threaded with timestamp alignment — each sensor runs on its own thread/process and pushes timestamped samples into a shared buffer; a separate aligner thread matches samples by nearest timestamp. Necessary once you add multiple cameras or higher-rate sensors like force-torque.
Hardware trigger sync — for research-grade setups, a hardware trigger line pulses all sensors simultaneously. Overkill for most hobbyist or startup projects, but worth knowing about if you're chasing sub-millisecond alignment.

For most imitation learning use cases, multi-threaded polling with timestamp-based nearest-neighbor alignment is a good default — it's robust and doesn't require special hardware.

Safety layers you shouldn't skip

Because a human is now directly driving a robot arm in real time, add these guardrails regardless of how "just for data collection" the setup feels:

Workspace bounding box enforced in software, independent of the IK solver.
A velocity limiter on the mapped target pose to prevent sudden jumps if the input device glitches.
A dead-man's switch (physical button or trigger) that must be held to enable motion — releasing it should freeze or gently stop the robot.
An emergency stop that cuts power at the hardware level, not just a software flag.

Putting it together

Once your loop is running reliably, wrap each demonstration collection session with clear episode boundaries — a "start recording" and "end recording" signal (often a button press) so each demonstration becomes a discrete, labeled trajectory rather than one continuous, ambiguous stream. This episodic structure is exactly what you'll need in the next step of the pipeline: capturing human demonstrations for imitation learning.

Useful Links

Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Building Synthetic Training Data Pipelines for Robotics

vmodal_ai — Wed, 02 Sep 2026 22:19:18 +0000

Building Synthetic Training Data Pipelines for Robotics

Real-world robot data is expensive to collect and label. Synthetic data pipelines — generating training data directly from simulation, complete with automatic ground-truth labels — let you produce far larger and more diverse datasets than manual collection ever could, for a fraction of the cost. This tutorial covers designing and building a synthetic data pipeline for robotics, from scene generation through to a usable, labeled dataset.

Where synthetic data fits in a robot learning pipeline

Synthetic data pipelines are most valuable for:

Perception training — object detection, segmentation, pose estimation, and grasp point prediction all benefit enormously from large volumes of perfectly labeled data, which is exactly what simulation provides for free.
Bootstrapping policies before real demonstrations exist — pretraining on synthetic rollouts, then fine-tuning on a smaller set of real demonstrations (the same pattern discussed in the sim-to-real tutorial).
Rare or dangerous scenarios — edge cases that are hard or unsafe to reproduce repeatedly on real hardware (near-collisions, extreme object configurations, sensor failure modes) are trivial to generate synthetically.

It's not a full replacement for real data — visual and dynamics gaps mean synthetic-only training rarely matches real-data performance for the hardest tasks — but as a large, cheap, well-labeled complement to a smaller real dataset, it's extremely effective.

Pipeline architecture

Scene Generation -> Domain Randomization -> Rendering/Simulation -> Ground-Truth Extraction -> Dataset Export

Each stage is a distinct, testable component, which matters because synthetic pipelines tend to run at large scale (thousands to millions of samples) — a bug caught late costs far more compute to regenerate than one caught early.

Step 1: Scene generation

Programmatically assemble scenes rather than hand-building each one. A scene generator typically:

Samples a task-relevant object (or set of objects) from an asset library.
Places it at a randomized, but physically valid, pose (checking for collisions/overlaps).
Samples a background/environment configuration.
Places the robot and camera(s) according to the deployment configuration (with some randomization, per the domain randomization tutorial).

import random

def generate_scene(asset_library, rng):
    obj = rng.choice(asset_library)
    pose = sample_valid_pose(obj, rng)
    background = rng.choice(BACKGROUND_LIBRARY)
    return {
        "object": obj,
        "object_pose": pose,
        "background": background,
    }

def sample_valid_pose(obj, rng, max_attempts=20):
    for _ in range(max_attempts):
        pose = rng.uniform(WORKSPACE_MIN, WORKSPACE_MAX)
        if not check_collision(obj, pose):
            return pose
    raise RuntimeError("Failed to sample a valid pose")

Step 2: Domain randomization as a data augmentation strategy

For synthetic data generation specifically, domain randomization does double duty: it improves sim-to-real transfer (as covered previously) and it's your primary mechanism for dataset diversity, replacing the manual scene variation a human data collector would otherwise need to provide. Apply the same categories covered in the domain randomization tutorial — lighting, textures, camera pose, distractors — but here, treat "diversity of the generated dataset" as the explicit success metric, not just "policy robustness."

Step 3: Rendering and extracting ground truth

This is where synthetic pipelines earn their value: ground-truth labels that would require expensive manual annotation in the real world come for free from the simulator's internal state.

def render_and_label(sim, scene):
    rgb = sim.render_rgb()
    depth = sim.render_depth()
    segmentation = sim.render_instance_segmentation()

    bbox_2d = compute_2d_bbox(segmentation, target_id=scene["object"].id)
    pose_6d = sim.get_object_pose(scene["object"].id)  # ground-truth 6-DOF pose
    grasp_points = compute_grasp_candidates(scene["object"])

    return {
        "rgb": rgb,
        "depth": depth,
        "segmentation": segmentation,
        "bbox_2d": bbox_2d,
        "pose_6d": pose_6d,
        "grasp_points": grasp_points,
    }

Common label types worth extracting depending on your downstream task: 2D/3D bounding boxes, instance/semantic segmentation masks, 6-DOF object pose, depth maps, surface normals, keypoints, and grasp candidate annotations. Since these all come from querying the simulator's internal state rather than from a human annotator, generating additional label types later costs essentially nothing compared to going back and re-annotating a real dataset.

Step 4: Dataset export

Export to a format your training pipeline (and ideally, standard tooling) can consume directly. Common choices:

COCO format for detection/segmentation tasks — widely supported by existing training frameworks.
A custom schema mirroring your real-data format — often the better choice, since it lets you mix synthetic and real samples in the same training pipeline without a translation layer.

import json

def export_sample(sample, output_dir, sample_id):
    image_path = f"{output_dir}/images/{sample_id}.png"
    save_image(sample["rgb"], image_path)

    annotation = {
        "image": image_path,
        "bbox_2d": sample["bbox_2d"],
        "pose_6d": sample["pose_6d"].tolist(),
        "grasp_points": sample["grasp_points"],
    }
    with open(f"{output_dir}/annotations/{sample_id}.json", "w") as f:
        json.dump(annotation, f)

Scaling generation

Because each sample is independent, synthetic data generation parallelizes naturally across processes or machines:

from multiprocessing import Pool

def generate_one_sample(args):
    seed, output_dir, idx = args
    rng = np.random.default_rng(seed)
    scene = generate_scene(ASSET_LIBRARY, rng)
    sim = setup_simulation(scene)
    sample = render_and_label(sim, scene)
    export_sample(sample, output_dir, idx)

def generate_dataset(num_samples, output_dir, num_workers=8):
    args = [(i, output_dir, i) for i in range(num_samples)]
    with Pool(num_workers) as pool:
        pool.map(generate_one_sample, args)

At real scale (hundreds of thousands of samples), this typically moves to a distributed job queue (e.g., across a cluster or cloud batch jobs) rather than a single machine's multiprocessing pool, but the underlying per-sample independence is what makes that scaling straightforward in the first place.

Validating the synthetic dataset

Synthetic data pipelines fail silently more often than real data collection does — a bug in your labeling code produces plausible-looking but wrong labels, and nothing about the pipeline will complain. Build validation in from the start:

Visualize a random sample of generated images with labels overlaid (bounding boxes, segmentation masks, projected 6-DOF pose axes) every time you change the pipeline, not just once at the beginning.
Check label statistics — distribution of object poses, bounding box sizes, and class balance — for unexpected clustering that suggests a sampling bug rather than genuine diversity.
Train a quick baseline model on a small synthetic subset and sanity-check its behavior before committing to generating the full dataset at scale.

Mixing synthetic and real data

The most effective real-world pipelines rarely use synthetic data alone. A common and effective pattern:

Generate a large, diverse synthetic dataset covering broad variation.
Collect a smaller real dataset (using the teleoperation and demonstration capture pipelines covered earlier in this series) covering the specific deployment conditions.
Either pretrain on synthetic data and fine-tune on real data, or mix both into a single training set with real data oversampled relative to its natural proportion, since it's typically more valuable per-sample for closing the sim-to-real gap.

Closing the loop

This tutorial completes the simulation side of the pipeline: Isaac Sim and MuJoCo for building and testing policies, sim-to-real and domain randomization techniques for making those policies transfer, and synthetic data generation for scaling training data cheaply. Combined with the teleoperation, demonstration capture, and imitation learning tutorials earlier in this series, you now have both the real and synthetic halves of a complete robot learning pipeline.

Useful Links

Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Designing a Production-Ready Physical AI Architecture

vmodal_ai — Wed, 02 Sep 2026 22:18:56 +0000

Designing a Production-Ready Physical AI Architecture

Moving a robotics prototype into production requires much more than improving model accuracy. A production Physical AI system must handle unreliable sensors, changing environments, timing constraints, hardware failures, software updates, observability, and safety.

This tutorial presents a practical architecture for building robust Physical AI applications.

Architecture at a Glance

                    Mission / UX
                         │
                         ▼
                Behavior Orchestration
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
         AI Reasoning           Planning
              │                     │
              └──────────┬──────────┘
                         ▼
                 World / Robot State
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
         Perception             Localization
              │                     │
              └──────────┬──────────┘
                         ▼
                    ROS 2 Graph
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
           Sensors              Actuators

ROS 2 organizes distributed computation as a graph of nodes and supports communication through topics, services, and actions. ROS 2 concepts

1. Separate Fast and Slow Loops

Not every component should run at the same frequency.

For example:

Motor Control       100–1000 Hz
State Estimation     50–200 Hz
Perception           10–60 Hz
Planning             1–20 Hz
Mission Reasoning    Event-driven

Exact frequencies depend on the robot. The architectural principle is to avoid allowing a slow AI operation to block a safety-critical control loop.

2. Establish a Hardware Abstraction Layer

Hardware drivers should expose stable interfaces.

Instead of:

Application → Motor SDK

prefer:

Application
    ↓
Robot Interface
    ↓
Hardware Adapter
    ↓
Motor SDK

This lets you replace a motor controller without changing mission logic.

3. Build a Reliable State Layer

A robot needs a consistent state representation.

Include:

RobotState
├── timestamp
├── pose
├── velocity
├── battery
├── localization_status
├── safety_status
├── active_mission
└── faults[]

All state should be timestamped so downstream components can reason about freshness.

4. Treat AI as a Component, Not the Whole System

A common mistake is:

LLM / Vision Model → Robot

A safer architecture is:

AI Model
   ↓
Intent / Proposal
   ↓
Validation
   ↓
Planner
   ↓
Safety Layer
   ↓
Controller
   ↓
Robot

AI should propose actions while deterministic software enforces constraints.

5. Design Explicit Safety Boundaries

Safety should include:

Maximum velocity
Maximum acceleration
Workspace limits
Collision constraints
Emergency stop
Watchdogs
Sensor health checks
Battery limits
Communication timeouts

For example:

def validate_velocity(vx, wz):
    if abs(vx) > MAX_LINEAR_SPEED:
        return False
    if abs(wz) > MAX_ANGULAR_SPEED:
        return False
    return True

Never assume an upstream AI component will always produce valid output.

6. Use ROS 2 Interfaces Intentionally

Use topics for continuous data such as sensor streams and robot state.

Use services for short request/response operations.

Use actions for long-running robot behaviors that require feedback or cancellation. ROS 2 interface guidance

This semantic separation makes a distributed system easier to understand and debug.

7. Add Fault Handling

Every important subsystem should expose health information.

Camera
  └── HEALTHY

LiDAR
  └── DEGRADED

Localization
  └── HEALTHY

Motor Controller
  └── FAULT

The mission manager can then decide whether to continue, retry, stop, or request human intervention.

8. Design for Graceful Degradation

Suppose the RGB camera fails.

A robust robot might transition:

Full Perception
      ↓ camera failure
Reduced Perception
      ↓
Safe Navigation
      ↓
Return / Stop

Avoid architectures where one optional sensor failure crashes the entire autonomy stack.

9. Make Deployment Reproducible

Use containers where appropriate:

Docker
├── perception
├── planning
├── monitoring
└── application

Version:

Models
Configuration
ROS packages
Dependencies
Hardware firmware
Calibration files

A robot should be reproducible from a known software release.

10. Observability

Production telemetry should answer:

What was the robot doing?
What did it perceive?
What did the planner decide?
Why did the controller stop?
How long did inference take?
Which sensor was unavailable?
What safety rule triggered?

Store structured events rather than relying only on console output.

11. Security

Networked robots should authenticate components and restrict interfaces.

Consider:

Device Identity
      ↓
Authentication
      ↓
Authorization
      ↓
Encrypted Communication
      ↓
ROS 2 / Application Services

Apply least privilege to remote control, diagnostics, and software updates.

12. Validate with Failure Scenarios

Create explicit tests for:

Camera disconnect
LiDAR timeout
Localization loss
Network interruption
GPU failure
Low battery
Stuck actuator
Invalid AI output
Planner timeout
Emergency stop

Production readiness comes from predictable behavior under failure, not just success-case demonstrations.

Conclusion

A production Physical AI architecture should be layered, observable, fault-tolerant, and safety-oriented. AI can provide powerful perception and reasoning, but deterministic components should remain responsible for validation, constraints, control, and emergency behavior.

The goal is not to create a robot that works only when everything is perfect. The goal is to create one that behaves predictably when reality is imperfect.

Useful Links

Website: www.v-modal.com
SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
SDK Android: https://github.com/v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Domain Randomization for Robust Robot Learning

vmodal_ai — Wed, 02 Sep 2026 22:18:49 +0000

Domain Randomization for Robust Robot Learning

Domain randomization is one of the most effective and widely used techniques for training robot policies in simulation that actually work on real hardware. Instead of trying to make simulation match reality as precisely as possible, domain randomization deliberately varies simulation parameters across a wide range during training, so the policy learns to be robust to variation rather than overfit to one specific (and inevitably slightly wrong) simulated world. This tutorial covers what to randomize, how much, and how to structure training around it.

The core idea

If a policy is trained across thousands of simulated variations — different friction coefficients, different lighting, different object textures, different camera positions — the real world simply becomes one more sample from that broad training distribution, rather than an unfamiliar edge case the policy has never encountered. The policy learns to rely on features and strategies that are robust across the whole distribution, rather than any single simulation's specific quirks.

What to randomize: dynamics parameters

For contact-rich and physically demanding tasks, randomizing the physics parameters that most affect real-world dynamics mismatch:

Friction coefficients (ground, object surfaces, gripper pads)
Mass and inertia of manipulated objects and robot links
Joint damping and friction
Actuator strength/gain and torque limits
Contact stiffness/softness parameters

import numpy as np

def randomize_dynamics(model, rng):
    for i in range(model.nbody):
        model.body_mass[i] *= rng.uniform(0.8, 1.2)

    for i in range(model.njnt):
        model.dof_damping[i] *= rng.uniform(0.7, 1.3)

    for i in range(model.ngeom):
        model.geom_friction[i] *= rng.uniform(0.6, 1.4)

A practical starting point is randomizing by a percentage range (e.g., ±20–30%) around your best system-identified estimate, rather than guessing an absolute range — this keeps the distribution centered on plausible reality instead of centered on an arbitrary default.

What to randomize: visual parameters

For vision-based policies, visual domain randomization is often the highest-leverage category, since visual mismatch tends to dominate the sim-to-real gap for camera-driven tasks:

Lighting — position, intensity, color temperature, number of light sources.
Textures — randomize materials on the ground, background, table, and even the robot itself, sampling from large texture datasets rather than a handful of hand-picked options.
Camera parameters — position, orientation, field of view, and even slight lens distortion, within a range consistent with real mounting tolerances.
Distractor objects — randomly placed irrelevant objects in the scene so the policy learns to attend to task-relevant objects specifically, rather than memorizing "the third object in the scene."
Color/appearance of the target object — if applicable, so the policy generalizes to object appearance variation rather than one hardcoded color/texture.

def randomize_visuals(scene, rng):
    scene.light_intensity = rng.uniform(300, 1500)
    scene.light_position = rng.uniform([-2, -2, 2], [2, 2, 4])
    scene.ground_texture = rng.choice(TEXTURE_LIBRARY)
    scene.camera_position += rng.uniform(-0.02, 0.02, size=3)
    scene.camera_fov = rng.uniform(55, 65)

What to randomize: control and sensing

Beyond physics and visuals, randomizing aspects of the control loop itself helps close the latency and noise gap:

Observation noise — add Gaussian noise to joint position/velocity readings and camera images to mimic real sensor imperfection.
Action latency — randomly delay applied actions by a few control steps to simulate real communication and processing latency.
Control frequency jitter — occasionally skip or double a control step to mimic the timing irregularities of a real control loop under load.

def add_sensor_noise(obs, rng, noise_std=0.01):
    return obs + rng.normal(0, noise_std, size=obs.shape)

How much randomization is too much

There's a real tradeoff here: too little randomization and the policy remains brittle to the exact gap sources it wasn't exposed to; too much randomization and the task becomes so variable that the policy struggles to learn anything useful, or learns an overly conservative, low-performance strategy just to survive the worst-case samples.

Practical guidance:

Start narrow, then widen. Begin with a modest randomization range validated against your system-identified real-world parameters, confirm the policy still learns the task well in simulation, then progressively widen the range while monitoring simulated task performance.
Randomize what plausibly varies in the real deployment, not everything imaginable. If your robot always operates under consistent lighting, heavy lighting randomization mostly wastes training capacity; if it operates in varied environments, it's essential.
Track both simulated success rate and (periodically) real-world success rate as you widen randomization ranges — the goal is the real-world number going up, not the simulated one going down as little as possible.

Automatic domain randomization (ADR)

A more advanced approach automatically adjusts randomization ranges during training based on policy performance, rather than using fixed ranges chosen up front:

Start with a narrow randomization range.
Periodically evaluate the policy at the edges of the current range.
If performance at the edges is still good, widen the range slightly; if performance degrades, hold or narrow it.

This produces a curriculum that automatically scales randomization difficulty to match the policy's current capability, rather than requiring the practitioner to guess the right fixed range in advance — but it adds meaningful implementation complexity and is usually only worth it once you've validated the basic fixed-range approach works for your task.

Structuring the training loop

A minimal per-episode randomization hook, integrated into an RL or imitation-learning-with-simulated-rollout training loop:

def reset_randomized_episode(env, rng, randomization_config):
    obs = env.reset()
    randomize_dynamics(env.model, rng)
    randomize_visuals(env.scene, rng)
    return obs

for episode in range(num_episodes):
    obs = reset_randomized_episode(env, rng, randomization_config)
    done = False
    while not done:
        action = policy.act(obs)
        action = maybe_delay_action(action)  # simulate control latency
        obs, reward, done, info = env.step(action)
        obs = add_sensor_noise(obs, rng)

Randomizing every episode (not just occasionally) is important — if only a fraction of training episodes are randomized, the policy can still find shortcuts that work for the common, non-randomized case and fail exactly where robustness matters most.

Measuring whether it's working

The real test of domain randomization isn't simulated performance — it's whether real-world performance improves, and ideally with less real-world fine-tuning data needed than a policy trained without randomization. Track a fixed real-world evaluation suite (as discussed in the sim-to-real tutorial) before and after adding or widening randomization, and treat any randomization change that doesn't measurably help real-world performance as a hyperparameter not worth keeping.

From randomized simulation to structured datasets

Domain randomization is also the foundation of another major use case: generating large-scale synthetic training datasets with rich, automatically-labeled ground truth — which is exactly what we build in the final tutorial of this series.

Useful Links

Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Sim-to-Real Transfer for Physical AI Robots

vmodal_ai — Wed, 02 Sep 2026 22:18:44 +0000

Sim-to-Real Transfer for Physical AI Robots

Training a policy in simulation is fast, safe, and scalable — but it's only useful if that policy also works on the real robot. The gap between simulated and real-world performance, known as the "sim-to-real gap," is one of the central challenges in Physical AI. This tutorial covers where that gap comes from and the concrete techniques used to close it.

Where the sim-to-real gap comes from

The gap typically breaks down into a few distinct sources, and it's worth diagnosing which one is dominant before picking a fix:

Dynamics mismatch — simulated friction, mass, damping, and actuator response never perfectly match the real robot. Contact-rich tasks (grasping, insertion, legged locomotion) are especially sensitive to this.
Visual mismatch — simulated camera images differ from real ones in lighting, texture detail, sensor noise, and lens distortion. This matters enormously for vision-based policies.
Latency and control-loop mismatch — real robots have communication delays, control loop jitter, and actuator lag that a simulated environment with perfect, instantaneous control doesn't capture.
Sensor noise and calibration — real sensors are noisier and imperfectly calibrated compared to their simulated, ground-truth counterparts.
Unmodeled phenomena — cable drag, backlash, thermal effects, and wear are rarely modeled in simulation at all.

Strategy 1: System identification

Before trying to make a policy "robust" to the gap, it's often worth simply making the simulation more accurate. System identification means measuring real robot parameters (joint friction, motor torque constants, link masses) and feeding them back into your simulation model.

A simple approach for a single parameter (e.g., joint friction):

import numpy as np
from scipy.optimize import minimize

def simulate_trajectory(friction_coef, initial_state, commands):
    # Run the sim with a given friction coefficient and return resulting trajectory
    ...

def loss(params, real_trajectory, initial_state, commands):
    friction_coef = params[0]
    sim_trajectory = simulate_trajectory(friction_coef, initial_state, commands)
    return np.mean((np.array(sim_trajectory) - np.array(real_trajectory)) ** 2)

result = minimize(
    loss, x0=[0.1], args=(real_trajectory, initial_state, commands), method="Nelder-Mead"
)
fitted_friction = result.x[0]

The idea generalizes: run the same command sequence on both the real robot and the simulator, then optimize simulation parameters to minimize the discrepancy between simulated and real trajectories. This is worth doing for a handful of high-leverage parameters (joint friction/damping, key link masses, actuator gain) rather than trying to identify everything at once.

Strategy 2: Domain randomization

Rather than trying to match simulation to reality exactly, domain randomization deliberately trains the policy across a wide distribution of simulated conditions, so the real world just looks like "one more sample" from that distribution. This is powerful enough that it gets its own dedicated tutorial next in this series — but the core idea is:

Randomize dynamics parameters (friction, mass, damping, motor strength) within a plausible range every episode.
Randomize visual parameters (lighting, textures, camera pose/intrinsics) for vision-based policies.
Randomize latency and add simulated actuator noise to reduce the policy's reliance on perfect, instantaneous control.

Strategy 3: Reducing reliance on precise dynamics

Some policy and control choices are inherently more robust to dynamics mismatch than others:

Position/impedance control over pure torque control — a well-tuned impedance controller absorbs some dynamics mismatch mechanically, before it ever reaches the learned policy.
Closed-loop, reactive policies over open-loop trajectory replay — a policy that continuously observes and reacts to the current state degrades more gracefully under mismatch than one that blindly executes a pre-planned trajectory.
Action chunking with re-planning (predict a short sequence of actions, execute a few, then re-observe and re-plan) strikes a middle ground — smoother than pure reactive single-step control, but still able to correct for accumulated error periodically.

Strategy 4: Closing the visual gap

For vision-based policies specifically:

Domain randomization of visual appearance — randomize textures, lighting, and distractor objects during simulated training so the policy doesn't overfit to simulation-specific visual artifacts.
Domain adaptation / image translation — train a model (e.g., a CycleGAN-style translator) to map real camera images into the visual style of simulation (or vice versa) before feeding them to the policy, reducing the visual distribution shift the policy has to handle natively.
Using real background/texture datasets as randomization sources — rather than purely synthetic textures, sampling from real-world image datasets during randomization tends to produce policies that generalize better to real cameras.
Matching camera intrinsics and mounting position precisely between simulation and the real robot — this sounds mundane but is one of the most common, easily fixed sources of vision-based sim-to-real failure.

Strategy 5: Fine-tuning on real data

Rather than relying entirely on zero-shot sim-to-real transfer, many practical pipelines use simulation to get most of the way there, then fine-tune on a small amount of real-world data:

Pretrain the policy primarily on simulated (and possibly domain-randomized) data.
Collect a modest set of real demonstrations or real rollout data (often far smaller than what would be needed to train from scratch).
Fine-tune the pretrained policy on this real data, ideally with a lower learning rate to avoid catastrophically forgetting the broad simulated experience.

This "sim pretraining + real fine-tuning" pattern often needs an order of magnitude less real-world data than training purely from real demonstrations, while still closing most of the residual gap that domain randomization alone doesn't fully address.

Evaluating sim-to-real transfer honestly

A few practices for measuring whether your transfer strategy is actually working, rather than just hoping:

Track a fixed real-world evaluation suite — the same set of tasks and initial conditions, run on the real robot every time you update your simulated training pipeline, so you can measure whether changes actually help.
Compare real vs. simulated performance on matched tasks, not just in isolation — a large gap that isn't shrinking over iterations is a signal that your randomization ranges or system identification may be systematically off, not just noisy.
Log failure modes qualitatively, not just success rate. A policy that fails safely and predictably (e.g., gets close but stops) is in a very different state than one that fails chaotically (e.g., slams into the workspace boundary) — success rate alone won't tell you which one you have.

Where domain randomization fits next

Domain randomization is powerful enough, and has enough of its own design decisions (what to randomize, how much, and how to structure training around it), that it deserves its own deep dive — which is exactly the next tutorial in this series.

Useful Links

Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx