Using VR Controllers for Robot Teleoperation

VR controllers (Meta Quest, HTC Vive, Valve Index) have become a popular teleoperation input for robot learning, and for good reason: they provide full 6-DOF pose tracking, intuitive hand-based control, and built-in buttons/triggers that map naturally onto gripper control — all in a consumer-grade, relatively cheap package. This tutorial covers how to set up VR controllers as a teleoperation input for a robot arm.

Why VR controllers work well for teleoperation

Compared to a gamepad or keyboard, a VR controller gives you:

• Continuous 6-DOF pose (position + orientation) tracked at high frequency, which maps intuitively to end-effector pose control.
• Natural hand motion, so operators produce trajectories that look like real human manipulation rather than artificial waypoint sequences.
• Built-in analog trigger, perfect for continuous gripper open/close control instead of a binary toggle.
• Haptic feedback (on some controllers), useful for signaling contact or constraint violations back to the operator.

Getting controller pose data

Most VR SDKs (OpenXR, Meta's Oculus SDK, SteamVR) expose controller pose as a position + quaternion relative to the headset's tracking origin, updated at high frequency (often 60–120 Hz). A minimal OpenXR-style polling loop looks like:

class VRControllerInput:
    def __init__(self, xr_session, controller_path):
        self.session = xr_session
        self.controller_path = controller_path

    def read(self):
        pose = self.session.get_controller_pose(self.controller_path)
        trigger_value = self.session.get_input_value(self.controller_path, "trigger")
        grip_button = self.session.get_input_value(self.controller_path, "grip")
        return {
            "position": pose.position,       # (x, y, z)
            "orientation": pose.orientation, # quaternion (x, y, z, w)
            "trigger": trigger_value,        # 0.0 - 1.0, maps to gripper
            "grip_button": grip_button,      # engage/disengage teleop
        }

If you're using Unity or Unreal for the VR side and streaming to a Python robot controller, this same pose data typically gets sent over a lightweight transport like ZeroMQ, gRPC, or a WebSocket, since VR engines and robotics stacks (ROS, Python control loops) usually don't live in the same process.

Mapping controller pose to robot end-effector pose

The core challenge is retargeting: the VR controller's coordinate frame and workspace scale rarely match the robot's. A typical approach:

1. Establish a reference frame. When the operator engages the grip button, record the controller's current pose as the origin.
2. Compute relative motion. For every subsequent frame, compute the controller's pose relative to that origin, not its absolute pose.
3. Apply a scale factor. Human arm movement range is often larger than the robot's comfortable workspace (or vice versa), so scale the relative translation before applying it.
4. Add the relative motion to the robot's pose at engagement time, producing the new target pose.

import numpy as np
from scipy.spatial.transform import Rotation as R

class VRToRobotMapper:
    def __init__(self, position_scale=1.0):
        self.position_scale = position_scale
        self.engaged = False
        self.origin_controller_pos = None
        self.origin_controller_rot = None
        self.origin_robot_pos = None
        self.origin_robot_rot = None

    def engage(self, controller_pos, controller_rot, robot_pos, robot_rot):
        self.engaged = True
        self.origin_controller_pos = np.array(controller_pos)
        self.origin_controller_rot = R.from_quat(controller_rot)
        self.origin_robot_pos = np.array(robot_pos)
        self.origin_robot_rot = R.from_quat(robot_rot)

    def disengage(self):
        self.engaged = False

    def map(self, controller_pos, controller_rot):
        if not self.engaged:
            return None

        delta_pos = (np.array(controller_pos) - self.origin_controller_pos) * self.position_scale
        target_pos = self.origin_robot_pos + delta_pos

        delta_rot = self.origin_controller_rot.inv() * R.from_quat(controller_rot)
        target_rot = self.origin_robot_rot * delta_rot

        return target_pos, target_rot.as_quat()

This "clutch" pattern — engage, move, disengage, reposition your hand, re-engage — mirrors how a mouse works when you lift it and reposition mid-drag. It's essential once the human's comfortable arm range doesn't match the robot's workspace, which is almost always the case.

Gripper control from the trigger

Map the analog trigger value directly to a gripper closure percentage rather than treating it as a binary switch — this preserves fine-grained grasp force control, which is valuable both for the task itself and, later, as a richer training signal for imitation learning:

def trigger_to_gripper_command(trigger_value, min_open=0.0, max_open=1.0):
    # trigger_value: 0.0 (released) to 1.0 (fully pressed)
    closure = trigger_value  # 0 = open, 1 = fully closed
    return min_open + closure * (max_open - min_open)

Handling orientation carefully

A common bug: naively slerp-ing or averaging quaternions without checking for the double-cover property of quaternions (q and -q represent the same rotation) causes sudden 180-degree "flips" in the mapped orientation. Always normalize sign consistency before interpolating, or use rotation matrices/axis-angle deltas if you're not confident in your quaternion math.

Latency and jitter considerations

VR tracking is generally very good, but a few things to watch for:

• Filter trigger and pose noise with a light exponential moving average — raw VR tracking data can have small high-frequency jitter that's imperceptible to the eye in a headset but shows up clearly in logged action data.
• Account for network latency if the VR engine and robot controller run in separate processes or machines — a delay of even 50–100 ms is noticeable as sluggish control and will show up as lag between the logged input and the resulting robot motion.
• Decouple render rate from control rate. The VR headset render loop typically runs faster than your robot control loop needs; sample the controller pose at your fixed control rate rather than trying to match VR frame rate exactly.

Combining with the rest of the pipeline

Everything from the general teleoperation and demonstration-capture tutorials earlier in this series applies directly here — the VR controller is simply a richer, more intuitive input device sitting at the top of the same architecture. Once your mapping and gripper control are solid, this setup tends to produce noticeably smoother, more natural demonstrations than joystick- or keyboard-based teleoperation, which pays off directly in imitation learning policy quality.

---

Useful Links

Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx