Capturing Human Demonstrations for Imitation Learning

Once you have a working teleoperation rig, the next challenge is turning raw teleop sessions into a clean, well-structured demonstration dataset. This is where most imitation learning projects quietly succeed or fail — a policy trained on inconsistent or noisy demonstrations will faithfully reproduce that inconsistency.

What counts as a "demonstration"

A demonstration is a single, complete episode of a task: from a defined start state to a defined end state (success, failure, or reset), recorded as a synchronized sequence of observations and actions.

Demonstration = {
  episode_id,
  timestamps[],
  observations[]  (images, joint states, proprioception),
  actions[]        (commanded joint/end-effector targets, gripper state),
  metadata          (task label, success flag, operator id)
}

Treating episodes as discrete, labeled units (rather than one long continuous log) is what makes the dataset usable for supervised learning later.

Designing your data schema before you collect anything

It's tempting to start recording immediately, but a few minutes spent on schema design saves days of reprocessing later. At minimum, decide on:

• Observation space: which camera(s), resolution, joint state format (position/velocity/torque), and any additional sensors (force-torque, tactile).
• Action space: joint-space targets vs. end-effector pose targets vs. delta actions. This choice has a big effect on how well imitation learning generalizes — delta (relative) actions are often more robust to small state drift than absolute targets.
• Sampling rate: pick one rate for the whole pipeline (commonly 10–30 Hz for manipulation) and resample everything to it rather than mixing rates.
• File format: HDF5 and per-episode directories with images-as-files plus a metadata file (e.g., a "LeRobot dataset"-style layout) are both common. HDF5 is compact and fast to load; a directory-per-episode layout is easier to inspect and debug.

Recording the episode loop

Extending the teleoperation loop from the previous tutorial, wrap it with clear episode lifecycle events:

class DemoRecorder:
    def __init__(self, teleop_session, dataset_writer):
        self.teleop = teleop_session
        self.writer = dataset_writer
        self.buffer = []

    def start_episode(self, task_label):
        self.buffer = []
        self.episode_meta = {"task": task_label, "start_time": time.time()}

    def record_step(self, obs, action):
        self.buffer.append({
            "timestamp": time.time(),
            "observation": obs,
            "action": action,
        })

    def end_episode(self, success: bool):
        self.episode_meta["success"] = success
        self.episode_meta["end_time"] = time.time()
        self.writer.write_episode(self.buffer, self.episode_meta)
        self.buffer = []

Key practice: let the operator explicitly mark success/failure at the end of each episode rather than inferring it automatically. Automated success detection is useful later for filtering, but early on, an honest human label is more trustworthy than a heuristic.

Data quality practices that matter more than volume

A smaller dataset of clean, diverse demonstrations usually outperforms a large dataset of repetitive or noisy ones. A few concrete practices:

• Vary initial conditions. Randomize object position, orientation, and (if applicable) lighting between episodes. A policy trained only on one fixed starting configuration will not generalize.
• Collect multiple operators if possible. A single operator's idiosyncratic style can become an artifact the policy overfits to.
• Discard aborted or corrected episodes, or label them explicitly as "corrected" rather than silently keeping them — mixing recovery behavior into clean success trajectories confuses the policy about what "correct" execution looks like.
• Balance the dataset across task variations. If 80% of your demonstrations are the easy variant of a task, the policy will be biased toward it.
• Sanity-check episode length distribution. Extremely short or extremely long episodes (compared to the median) usually indicate a teleop glitch or an operator mistake — review before including them.

Post-hoc validation before training

Before you ever hand this dataset to a training script, run a validation pass:

def validate_episode(episode, expected_hz, image_shape):
    timestamps = [s["timestamp"] for s in episode]
    dt = [t2 - t1 for t1, t2 in zip(timestamps, timestamps[1:])]
    assert max(dt) < 2.0 / expected_hz, "Frame drop detected"

    for step in episode:
        assert step["observation"]["image"].shape == image_shape
        assert not any(map(lambda v: v != v, step["action"])), "NaN in action"

Checks worth automating: dropped frames, NaNs in joint states or actions, camera frames that are all-black or all-white (a common sign of a disconnected or misconfigured camera), and gripper state that never changes across an episode where it clearly should (often a sign of a logging bug, not real behavior).

Visualizing before trusting

Always render a handful of random episodes back as video with overlaid action values before considering a collection session "done." This catches problems no automated check will — an operator drifting off-task, an object that rolled out of frame, or a gripper that visually never closes despite the logged command saying it did.

From raw demos to a training-ready dataset

Once you have validated, well-labeled episodes, the natural next step is assembling them into the full imitation learning pipeline: splitting into train/validation sets, normalizing observations and actions, and feeding them into a policy architecture — which is exactly what we build in the next tutorial in this series.

---

Useful Links

Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx