Using VR Controllers for Robot Teleoperation

VR controllers (Meta Quest, HTC Vive, Valve Index, etc.) have become a popular input device for robot teleoperation, and for good reason: they provide precise 6-DOF tracking, built-in buttons/triggers for gripper control, and an intuitive mapping between human hand motion and robot end-effector motion. This tutorial covers how to integrate VR controllers into a teleoperation pipeline.

Why VR Controllers?

Compared to a spacemouse or gamepad, VR controllers give you:

• Direct spatial mapping — moving your hand forward moves the end-effector forward, with no mode-switching between axes
• Sub-millimeter to centimeter tracking accuracy, depending on the headset's tracking system
• Built-in trigger and grip buttons, which map naturally to gripper open/close
• Stereo visual feedback if you pair it with a headset display fed by the robot's cameras, enabling immersive teleoperation

The trade-off is added system complexity: you now depend on a VR runtime, and you need to handle coordinate frame conversions between VR space and robot space carefully.

Step 1: Reading Controller Pose and Input

Most VR SDKs (OpenXR, SteamVR, Oculus/Meta SDK) expose controller pose as a position + quaternion, plus button/trigger states. Here's a generic wrapper:

class VRController:
    def __init__(self, sdk_client, hand="right"):
        self.sdk_client = sdk_client
        self.hand = hand

    def read(self) -> dict:
        pose = self.sdk_client.get_controller_pose(self.hand)
        return {
            "position": pose.position,       # (x, y, z) in VR world frame
            "orientation": pose.quaternion,  # (x, y, z, w)
            "trigger": self.sdk_client.get_trigger_value(self.hand),
            "grip": self.sdk_client.get_grip_value(self.hand),
            "buttons": self.sdk_client.get_buttons(self.hand),
        }

Step 2: Coordinate Frame Alignment

This is the step most tutorials skip and where most bugs live. The VR headset's world frame, the controller's local frame, and the robot's base frame are all different, and none of them naturally line up. You need a calibration transform.

A simple and effective approach is a one-time origin alignment: ask the operator to hold the controller at a known reference pose (e.g., directly above the robot's end-effector), then compute the transform between VR space and robot space at that moment.

import numpy as np
from scipy.spatial.transform import Rotation as R

def compute_calibration(vr_pose_at_ref, robot_pose_at_ref):
    """Both poses given as (position, quaternion) at the same physical moment."""
    vr_pos, vr_quat = vr_pose_at_ref
    robot_pos, robot_quat = robot_pose_at_ref

    R_vr = R.from_quat(vr_quat)
    R_robot = R.from_quat(robot_quat)

    R_offset = R_robot * R_vr.inv()
    t_offset = np.array(robot_pos) - R_offset.apply(vr_pos)

    return R_offset, t_offset

def vr_to_robot_frame(vr_pos, vr_quat, R_offset, t_offset):
    R_vr = R.from_quat(vr_quat)
    robot_pos = R_offset.apply(vr_pos) + t_offset
    robot_quat = (R_offset * R_vr).as_quat()
    return robot_pos, robot_quat

Run this calibration at the start of every session — controller tracking origins can shift between headset power cycles.

Step 3: Delta-Pose Teleoperation

Rather than mapping VR pose directly to robot pose 1:1 (which can be jarring if the operator's arm reach doesn't match the robot's workspace), use relative/delta control: press and hold the grip button to "engage" control, and only the change in controller pose since engagement is applied to the robot.

class VRDeltaTeleop:
    def __init__(self, controller, robot, position_scale=1.0):
        self.controller = controller
        self.robot = robot
        self.position_scale = position_scale
        self.engaged = False
        self.ref_vr_pos = None
        self.ref_robot_pose = None

    def step(self):
        state = self.controller.read()
        engage = state["grip"] > 0.5

        if engage and not self.engaged:
            # Just started engaging: capture reference poses
            self.ref_vr_pos = np.array(state["position"])
            self.ref_robot_pose = self.robot.get_state()["ee_pose"]
            self.engaged = True

        elif not engage:
            self.engaged = False
            return None  # no motion when disengaged

        delta = (np.array(state["position"]) - self.ref_vr_pos) * self.position_scale
        target_pose = self.ref_robot_pose.copy()
        target_pose[:3] += delta

        gripper_cmd = state["trigger"]
        return target_pose, gripper_cmd

This "clutch" mechanism (engage/disengage like picking up a mouse to reposition it) is exactly how experienced teleoperators handle limited workspace — and it noticeably reduces fatigue over long collection sessions.

Step 4: Adding Haptic and Visual Feedback

If your VR SDK exposes haptics, use them to signal contact events (e.g., gripper closing on an object, or approaching a force limit read from the robot's F/T sensor):

def send_contact_feedback(sdk_client, hand, force_magnitude):
    intensity = min(force_magnitude / 20.0, 1.0)  # normalize to [0, 1]
    sdk_client.trigger_haptic_pulse(hand, duration_ms=50, amplitude=intensity)

For visual feedback, streaming the robot's wrist camera to the headset display (rather than a fixed monitor) significantly improves depth perception and precision for fine manipulation tasks.

Step 5: Integrating with the Data Collection Pipeline

VR teleoperation slots directly into the teleoperation and logging pipeline from the earlier tutorials — swap in VRDeltaTeleop as the input source:

def vr_teleop_loop(vr_teleop, robot, camera, logger, hz=20):
    dt = 1.0 / hz
    while True:
        start = time.time()
        result = vr_teleop.step()

        if result is not None:
            target_pose, gripper_cmd = result
            robot.send_command(target_pose, gripper_cmd)

            obs = {"image": camera.get_frame(), "ee_pose": robot.get_state()["ee_pose"]}
            logger.log(obs, target_pose, time.time())

        time.sleep(max(0, dt - (time.time() - start)))

Practical Tips

• Filter controller jitter. Raw VR tracking has small-scale noise; a simple exponential moving average on position/orientation smooths it out without adding noticeable lag.
• Cap velocity, not just position. Clamp the per-step delta to a maximum magnitude to prevent sudden large jumps from tracking glitches.
• Test calibration drift. Long sessions (30+ minutes) can accumulate small drift; consider a quick re-calibration checkpoint between recording batches.
• Mind the latency budget. VR-to-robot pipelines add tracking + wireless (if applicable) + rendering latency on top of robot control latency — measure end-to-end latency and keep it under ~100ms for comfortable operation.

What's Next

VR teleoperation is especially powerful when paired with sensor recording and replay for debugging — covered in the next tutorial — since you can visually replay exactly what the operator saw and did.

Useful Links

Website: www.v-modal.com
SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
SDK Android: https://github.com/v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx