Building a Robot Teleoperation System for Data Collection

If you're working on robot learning, you already know the bottleneck isn't the model — it's the data. Simulation only gets you so far, and real-world robot data is expensive to collect. A solid teleoperation system is the fastest way to bootstrap a dataset of real, physically-grounded robot trajectories that you can later use for imitation learning.

In this tutorial, we'll walk through the architecture and implementation of a teleoperation pipeline you can use to collect demonstration data from a robot arm (or mobile manipulator) in real time.

Why Teleoperation for Data Collection?

Before diving into code, it's worth being clear on the goal: teleoperation isn't just about controlling a robot remotely — it's about generating (observation, action) pairs that look exactly like what your policy will see and produce at inference time. That means:

• The action space must match your deployment action space (joint angles, end-effector pose, gripper state, etc.)
• The observation stream (camera frames, proprioception, force/torque) must be synchronized and logged at a consistent rate
• Latency and jitter in the control loop should be minimized so demonstrations reflect smooth, intentional human behavior

System Architecture

A typical teleoperation stack for data collection has four layers:

1. Input device layer — joystick, spacemouse, leader-arm, VR controller, or haptic device
2. Control mapping layer — converts raw input into robot commands (Cartesian velocity, joint velocity, or delta pose)
3. Robot interface layer — sends commands to the robot controller and reads back proprioceptive state
4. Data logging layer — records synchronized observation-action pairs to disk

[Input Device] --> [Control Mapper] --> [Robot Driver] --> [Robot Hardware]
                                              |
                                              v
                                     [State/Camera Streams]
                                              |
                                              v
                                       [Data Logger]

Step 1: Choose Your Input Device

Common choices, roughly in order of fidelity vs. cost:

• 3D mouse / spacemouse — cheap, decent for 6-DOF end-effector control
• Leader-follower arm (a second, low-cost robot arm you move by hand) — very high fidelity, matches kinematics
• Gamepad — simple, low-cost, but coarse control
• VR controllers — intuitive 6-DOF input with good ergonomics (we cover this in a dedicated tutorial)

For this example, we'll use a generic 6-DOF input device abstraction so the code works regardless of hardware.

class InputDevice:
    def read(self) -> dict:
        """Return raw input state.
        Returns:
            dict with keys: position_delta (3,), rotation_delta (3,),
            gripper_signal (float), buttons (dict)
        """
        raise NotImplementedError

Step 2: Map Input to Robot Commands

The control mapper translates raw device input into a command the robot understands. A common approach is delta Cartesian control: the operator's motion is interpreted as an incremental change to the end-effector pose.

import numpy as np

class CartesianDeltaMapper:
    def __init__(self, position_scale=0.5, rotation_scale=0.3):
        self.position_scale = position_scale
        self.rotation_scale = rotation_scale

    def map(self, raw_input: dict, current_pose: np.ndarray) -> np.ndarray:
        delta_pos = raw_input["position_delta"] * self.position_scale
        delta_rot = raw_input["rotation_delta"] * self.rotation_scale

        target_pose = current_pose.copy()
        target_pose[:3] += delta_pos
        target_pose[3:6] += delta_rot
        return target_pose

Keep the scaling factors tunable — this is the single biggest lever for making teleoperation feel natural. Too aggressive and operators overshoot; too conservative and tasks feel sluggish.

Step 3: Robot Interface

Wrap your robot's native SDK (ROS, vendor SDK, or custom driver) behind a minimal interface so the rest of the pipeline doesn't care which robot you're using:

class RobotInterface:
    def get_state(self) -> dict:
        """Returns joint_positions, ee_pose, gripper_state, timestamp."""
        raise NotImplementedError

    def send_command(self, target_pose: np.ndarray, gripper_cmd: float):
        raise NotImplementedError

This abstraction pays off enormously later — swapping robots or simulators becomes a one-file change.

Step 4: Synchronized Data Logging

The most common mistake in DIY teleoperation pipelines is unsynchronized logging — camera frames drift out of alignment with robot state, and your dataset ends up noisy. Use a fixed-rate control loop and timestamp everything.

import time

class Logger:
    def __init__(self, save_path):
        self.buffer = []
        self.save_path = save_path

    def log(self, obs: dict, action: np.ndarray, timestamp: float):
        self.buffer.append({
            "timestamp": timestamp,
            "obs": obs,
            "action": action,
        })

    def flush(self, episode_id: int):
        np.save(f"{self.save_path}/episode_{episode_id}.npy", self.buffer)
        self.buffer = []

Step 5: The Control Loop

Tie it all together in a fixed-frequency loop (10–30 Hz is typical for manipulation tasks):

def teleop_loop(device, mapper, robot, camera, logger, hz=20, episode_id=0):
    dt = 1.0 / hz
    while True:
        loop_start = time.time()

        raw_input = device.read()
        state = robot.get_state()
        target_pose = mapper.map(raw_input, state["ee_pose"])

        robot.send_command(target_pose, raw_input["gripper_signal"])

        obs = {
            "image": camera.get_frame(),
            "joint_positions": state["joint_positions"],
            "ee_pose": state["ee_pose"],
        }
        logger.log(obs, target_pose, time.time())

        if raw_input["buttons"].get("end_episode"):
            logger.flush(episode_id)
            break

        elapsed = time.time() - loop_start
        time.sleep(max(0, dt - elapsed))

Practical Tips

• Add a dead-man's switch. Require a button press to enable motion — this prevents accidental commands and gives operators a clean way to "pause" without breaking the episode boundary.
• Log at the robot's native control rate, not the input device's polling rate, to avoid aliasing.
• Version your schema. Once you have hundreds of episodes, changing your observation format is painful. Lock it down early or add a schema version field.
• Record video, not just state, even if your first policy is state-based — you'll want it for debugging and future vision-based models.

What's Next

With this pipeline running, you can start collecting demonstrations. In the next tutorials, we'll cover how to structure human demonstrations specifically for imitation learning, how VR controllers can improve teleoperation ergonomics, and how to build the full imitation learning pipeline on top of this data.

Useful Links

Website: www.v-modal.com
SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
SDK Android: https://github.com/v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx