Capturing Human Demonstrations for Imitation Learning

Once you have a teleoperation system in place, the next challenge is capturing demonstrations that are actually useful for training a policy. Bad demonstration data — inconsistent, noisy, or poorly labeled — is one of the biggest silent killers of imitation learning projects. This tutorial focuses on the data-quality side: what to capture, how to structure it, and how to avoid the common pitfalls.

What Makes a "Good" Demonstration?

A good demonstration for imitation learning has three properties:

1. Task-consistent — it actually completes the task, without recovery from failures unless recoveries are part of what you want the policy to learn
2. Smooth and low-noise — human tremor and hesitation shows up in the action signal; overly jerky demos make for a harder learning problem
3. Diverse enough to generalize — varying initial object positions, lighting, and grasp approaches, not just repeating the exact same trajectory

Defining Your Data Schema

Before recording anything, lock down what an episode looks like. A minimal but robust schema:

episode = {
    "episode_id": str,
    "task_name": str,
    "timesteps": [
        {
            "timestamp": float,
            "observation": {
                "rgb": np.ndarray,       # (H, W, 3)
                "depth": np.ndarray,     # optional, (H, W)
                "joint_positions": np.ndarray,  # (N,)
                "ee_pose": np.ndarray,   # (6,) or (7,) with quaternion
                "gripper_state": float,
            },
            "action": np.ndarray,       # matches your control space
        }
        for _ in range(episode_length)
    ],
    "success": bool,
    "metadata": {
        "operator_id": str,
        "device": str,
        "notes": str,
    },
}

Two details matter more than people expect:

• Store the action in the same representation your policy will predict. If you're training with delta end-effector actions, don't log absolute joint angles and convert later — convert at capture time and sanity-check it.
• Always log a success flag, ideally with a short free-text note. You will want to filter your dataset later, and re-watching hours of video to find failures is a waste of time you can avoid now.

Recording Workflow

A practical recording session loop looks like this:

def record_episode(env, teleop, logger, task_name, operator_id):
    obs = env.reset()
    timesteps = []

    print("Press START to begin recording...")
    teleop.wait_for_start()

    while not teleop.episode_done():
        action = teleop.get_action(obs)
        next_obs, _, done, info = env.step(action)

        timesteps.append({
            "timestamp": time.time(),
            "observation": obs,
            "action": action,
        })
        obs = next_obs

    success = teleop.ask_success_label()  # operator marks success/failure
    logger.save_episode(timesteps, task_name, operator_id, success)

Prompting the operator for a success label immediately after the episode — while it's fresh — is far more reliable than reviewing footage later.

Handling Multimodal Observations

Most manipulation tasks benefit from more than one camera view (e.g., a wrist camera plus a static third-person view). Keep these as separate, named keys rather than concatenating them:

observation = {
    "wrist_rgb": wrist_cam.get_frame(),
    "front_rgb": front_cam.get_frame(),
    "joint_positions": robot.get_joint_positions(),
    "gripper_state": robot.get_gripper_state(),
}

This keeps your dataset flexible — you can train a wrist-only policy, a multi-view policy, or ablate camera views without re-collecting data.

Data Quality Checklist

Before you consider a batch of demonstrations "done," run through this checklist:

• [ ] Are all episodes at a consistent control frequency?
• [ ] Do timestamps show no large gaps (dropped frames, stalls)?
• [ ] Is the success rate reasonable (not 100%, which usually means the task is too easy or labels are wrong; not near 0%, which means the demos aren't useful)?
• [ ] Is there variation in object pose/starting conditions across episodes?
• [ ] Are failed episodes tagged and separable from successful ones?

Handling Multiple Operators

If more than one person collects data, expect style variance — different grasp approaches, speeds, and hesitation patterns. This isn't necessarily bad (it can improve robustness), but you should:

• Log operator_id on every episode
• Periodically check per-operator success rates
• Consider training/eval splits that hold out an entire operator, to test whether your policy generalizes beyond one person's style

Storage Format

For small-to-medium datasets, per-episode .npz or .hdf5 files work well:

import h5py

def save_episode_hdf5(path, timesteps, metadata):
    with h5py.File(path, "w") as f:
        f.attrs.update(metadata)
        for key in timesteps[0]["observation"]:
            data = np.stack([t["observation"][key] for t in timesteps])
            f.create_dataset(f"obs/{key}", data=data, compression="gzip")
        actions = np.stack([t["action"] for t in timesteps])
        f.create_dataset("actions", data=actions, compression="gzip")

HDF5 gives you compression, partial reads, and metadata attributes in one format — useful once your dataset grows past a few hundred episodes.

What's Next

With a clean, well-structured demonstration dataset, you're ready to actually train a policy. The next tutorial covers building the full imitation learning pipeline — from dataset loading to policy architecture to evaluation.

Useful Links

Website: www.v-modal.com
SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
SDK Android: https://github.com/v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx