DEV Community

Takeo
Takeo

Posted on

Trying VLA (Part 6): Controlling LeRobot with a SpaceMouse

#ai #hardware #python #robotics

Mapping SpaceMouse Controls to SO-101 Movements

In the previous article, I connected the SpaceMouse to the PC and confirmed that all six types of input could be detected correctly.

https://dev.to/takeofuture/trying-vla-part-5-setting-up-and-testing-a-spacemouse-eio

Forward / Backward
Left / Right
Up / Down
Pitch
Roll
Yaw
Enter fullscreen mode Exit fullscreen mode

During the SpaceMouse test, I confirmed the following input values.

Forward horizontal  = +Y
Backward horizontal = -Y

Left horizontal     = -X
Right horizontal    = +X

Up                  = +Z
Down                = -Z

Forward tilt        = +Pitch
Backward tilt       = -Pitch

Left tilt           = -Roll
Right tilt          = +Roll

Left twist          = -Yaw
Right twist         = +Yaw
Enter fullscreen mode Exit fullscreen mode

An important point here is that these values from the SpaceMouse are not sent directly to individual SO-101 motors.

Conceptually, the flow from the SpaceMouse to the SO-101 looks like this:

SpaceMouse
    ↓
x / y / z / roll / pitch / yaw
    ↓
SpaceMouse Teleoperator
    ↓
target_x / target_y / target_z
target_wx / target_wy / target_wz
    ↓
Inverse Kinematics (IK)
    ↓
SO-101 Joint Positions
    ↓
SO-101
Enter fullscreen mode Exit fullscreen mode

The SpaceMouse plugin treats the 6DoF input from the SpaceMouse as movement of the End Effector in Cartesian coordinates.

The target movement is then converted into the required SO-101 joint angles using IK, or Inverse Kinematics.

In other words, instead of directly specifying something like:

"Move this motor by 5 degrees"
Enter fullscreen mode Exit fullscreen mode

we provide commands such as:

"Move the End Effector slightly forward"
"Move the End Effector slightly upward"
"Rotate the End Effector slightly"
Enter fullscreen mode Exit fullscreen mode

The SpaceMouse provides these commands, and IK calculates how the individual joints need to move.

Mapping Between SpaceMouse and LeRobot Coordinates

There is one thing we need to be careful about here.

The x and y values displayed by the SpaceMouse test do not directly become LeRobot's target_x and target_y.

With the default SpaceMouse plugin configuration, the axes are mapped as follows:

SpaceMouse y     -> target_x
SpaceMouse x     -> target_y
SpaceMouse z     -> target_z

SpaceMouse roll  -> target_wx
SpaceMouse pitch -> target_wy
SpaceMouse yaw   -> target_wz
Enter fullscreen mode Exit fullscreen mode

In addition, the sign of target_y is inverted.

Therefore, based on the SpaceMouse inputs I confirmed in the previous test, the mapping to LeRobot becomes:

SpaceMouse Operation SpaceMouse Value LeRobot Side
Move forward horizontally y = + target_x = +
Move backward horizontally y = - target_x = -
Move left horizontally x = - target_y = +
Move right horizontally x = + target_y = -
Move up z = + target_z = +
Move down z = - target_z = -
Tilt left roll = - target_wx = -
Tilt right roll = + target_wx = +
Tilt forward pitch = + target_wy = +
Tilt backward pitch = - target_wy = -
Twist left yaw = - target_wz = -
Twist right yaw = + target_wz = +

Here,

target_x / target_y / target_z
Enter fullscreen mode Exit fullscreen mode

represent End Effector translation, while

target_wx / target_wy / target_wz
Enter fullscreen mode Exit fullscreen mode

represent rotation around the X, Y, and Z axes.

The default SpaceMouse plugin configuration uses:

x_axis="y"
y_axis="x"
y_sign=-1
Enter fullscreen mode Exit fullscreen mode

which is why the axes are mapped this way.

About 6DoF Input and the SO-101

There is another important point.

The SpaceMouse itself is a full 6DoF input device:

X
Y
Z
Roll
Pitch
Yaw
Enter fullscreen mode Exit fullscreen mode

The SO-101 has six servos:

shoulder_pan
shoulder_lift
elbow_flex
wrist_flex
wrist_roll
gripper
Enter fullscreen mode Exit fullscreen mode

However, the final servo, gripper, is used to open and close the gripper.

This means that the robot arm uses five joints to control the position and orientation of the End Effector.

The SpaceMouse can provide a 6DoF target, but this does not mean that the SO-101 can independently realize every arbitrary 3D position and orientation.

LeRobot's IK converts the requested motion into joint movements that are feasible for the SO-101.

For this reason, during the first teleoperation test, I will move the SpaceMouse one axis at a time and verify which direction the End Effector actually moves.

What Are We Actually Controlling with the SpaceMouse?

Before moving the SO-101, let's clarify what we are actually controlling with the SpaceMouse.

The SpaceMouse does not directly control each SO-101 motor one by one.

Instead, what we mainly control is the 3D position and orientation of the robot arm's End Effector.

Using the SpaceMouse, we can:

  • Move forward/backward, left/right, and up/down
  • Tilt forward/backward and left/right
  • Rotate left/right

In other words, we tell the robot:

"Move the End Effector in this direction and orient it this way."

SpaceMouse
    ↓
Specify End Effector
position and orientation
(X / Y / Z / Roll / Pitch / Yaw)
    ↓
LeRobot + IK
(Inverse Kinematics)
    ↓
Calculate how the joints
need to move together
    ↓
Control the SO-101 motors
Enter fullscreen mode Exit fullscreen mode

The user does not need to think in terms of:

Move shoulder_pan by a certain angle
Move shoulder_lift by a certain angle
Move elbow_flex by a certain angle
Enter fullscreen mode Exit fullscreen mode

Instead, we simply operate the SpaceMouse based on where we want the End Effector to move.

LeRobot's IK calculates how the individual joints should move together.

The only exception is the gripper open/close operation.

That operation is mapped directly to the SpaceMouse buttons rather than being calculated by IK.

SpaceMouse 6DoF Motion
    ↓
Position / Orientation
    ↓
Inverse Kinematics
    ↓
Multiple joints move together

SpaceMouse Button
    ↓
Gripper Open / Close
    ↓
Open or close the gripper
Enter fullscreen mode Exit fullscreen mode

So, for the arm's position and orientation, we specify the desired End Effector motion with the SpaceMouse and let LeRobot determine how the joints should move.

For the gripper itself, the open/close commands are directly mapped to SpaceMouse buttons.

Actually Moving the SO-101 with the SpaceMouse

Now let's actually control the SO-101 using the SpaceMouse.

The SpaceMouse plugin provides an IK profile for the SO-101, so teleoperation can basically be started with a command like this.

First, activate the Python virtual environment.

$ source ~/pyenv/lerobot/bin/activate
Enter fullscreen mode Exit fullscreen mode

Then run the teleoperation command.

(lerobot)$ lerobot-teleoperate \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=takeo_so101 \
  --teleop.type=spacemouse \
  --teleop.adapter.mode=ik \
  --fps=60
Enter fullscreen mode Exit fullscreen mode

Since this is my first time actually controlling the robot with the SpaceMouse, I do not want to start immediately with the normal settings.

Instead, I will begin with more conservative settings.

(lerobot)$ lerobot-teleoperate \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=takeo_so101 \
  --teleop.type=spacemouse \
  --teleop.adapter.mode=ik \
  --teleop.deadzone=0.1 \
  --teleop.input_timeout_s=0.08 \
  --teleop.adapter.translation_step_m=0.001 \
  --teleop.adapter.rotation_step_rad=0.005 \
  --fps=60
Enter fullscreen mode Exit fullscreen mode

Since I am operating the machine through SSH, I am not adding the GUI visualization option:

--display_data=true
Enter fullscreen mode Exit fullscreen mode

If you are running LeRobot directly from a local Ubuntu desktop, you can add it if needed.

What the Important Options Mean

The following settings are particularly important for this test.

--teleop.adapter.mode=ik
Enter fullscreen mode Exit fullscreen mode

This converts the Cartesian input from the SpaceMouse into SO-101 joint angles using Inverse Kinematics.

--teleop.deadzone=0.1
Enter fullscreen mode Exit fullscreen mode

This ignores very small SpaceMouse inputs around the neutral position.

--teleop.input_timeout_s=0.08
Enter fullscreen mode Exit fullscreen mode

If no new input is received from the SpaceMouse for a certain amount of time, the input is treated as zero.

--teleop.adapter.translation_step_m=0.001
Enter fullscreen mode Exit fullscreen mode

This limits the translational movement per frame, even when the SpaceMouse is pushed to its maximum input.

0.001 m is equal to 1 mm.

--teleop.adapter.rotation_step_rad=0.005
Enter fullscreen mode Exit fullscreen mode

This also keeps the rotational movement small during the initial test.

These settings are introduced in the SpaceMouse plugin README as a "Conservative first run" configuration.

Before Actually Moving the Robot

Before running the command, make sure that:

  • The SO-101 is securely fixed to the desk
  • The Waveshare board is connected via USB
  • The 12V power supply for the servos is connected
  • The robot arm is not in an extreme pose
  • The arm is positioned so that it will not hit the clamp or desk
  • The SpaceMouse is in its neutral position

Also, depending on the configuration, torque may be disabled when the SO-101 program exits.

Therefore, when stopping the program with Ctrl+C, be prepared to support the robot arm so that it does not suddenly drop under gravity.

First, Try Only Forward and Backward Motion

Instead of immediately trying all six axes, I will first test only the forward and backward horizontal movement of the SpaceMouse.

After starting the command, do not touch the SpaceMouse at first.

First, confirm that the SO-101 maintains its current pose.

Then:

Push the SpaceMouse slightly forward
          ↓
Immediately return it to center
          ↓
Check the movement of the SO-101 End Effector
Enter fullscreen mode Exit fullscreen mode

At first, do not push the SpaceMouse all the way or hold it for a long period of time.

Repeat the following:

Push slightly
↓
Return to center
↓
Observe the robot
Enter fullscreen mode Exit fullscreen mode

If forward and backward movement work correctly, test the remaining axes one at a time:

Left / Right
Up / Down
Pitch
Roll
Yaw
Enter fullscreen mode Exit fullscreen mode

What Happens Between the SpaceMouse and SO-101?

For example, when I push the SpaceMouse forward, the previous test showed:

SpaceMouse y = +
Enter fullscreen mode Exit fullscreen mode

The plugin converts this to:

target_x = +
Enter fullscreen mode Exit fullscreen mode

Then IK converts the requested target_x movement into the required joint angles:

target_x
    ↓
shoulder_pan
shoulder_lift
elbow_flex
wrist_flex
wrist_roll
Enter fullscreen mode Exit fullscreen mode

In other words, pushing the SpaceMouse forward does not simply move one particular motor.

Multiple joints move together so that the End Effector moves in the requested direction.

This is the major difference from the previous test, where I directly moved individual motors from Python.

Controlling the Gripper with SpaceMouse Buttons

The SpaceMouse plugin can also use the side buttons on the SpaceMouse.

By default:

button[1] -> Gripper Open
button[0] -> Gripper Close
Enter fullscreen mode Exit fullscreen mode

However, it is better to verify which physical button on your particular SpaceMouse corresponds to button[0] and button[1].

Before connecting the robot, run:

(lerobot)$ lerobot-teleoperator-spacemouse-test
Enter fullscreen mode Exit fullscreen mode

With no buttons pressed, you should see:

buttons=[0, 0]
Enter fullscreen mode Exit fullscreen mode

Then press the left and right physical buttons one at a time.

You should see either:

buttons=[1, 0]
Enter fullscreen mode Exit fullscreen mode

or:

buttons=[0, 1]
Enter fullscreen mode Exit fullscreen mode

This lets you determine which physical SpaceMouse button corresponds to each button index.

Once this is confirmed, the SpaceMouse can be used to control:

End Effector Translation
+
End Effector Rotation
+
Gripper Open / Close
Enter fullscreen mode Exit fullscreen mode

Overall Flow

The complete flow looks like this:

SpaceMouse
   │
   ├─ Forward / Backward / Left / Right / Up / Down
   ├─ Roll / Pitch / Yaw
   └─ Gripper Buttons
   ↓
LeRobot SpaceMouse Teleoperator
   ↓
Cartesian End-Effector Target
   ↓
Inverse Kinematics
   ↓
SO-101 Joint Position Commands
   ↓
SO-101 Follower Arm
Enter fullscreen mode Exit fullscreen mode

The SpaceMouse is not directly connected to the SO-101.

Both the SpaceMouse and the SO-101 are connected to the PC.

LeRobot running on the PC reads the SpaceMouse input and uses IK to convert that input into joint movements for the SO-101.

Once this setup is working correctly, the next step will be to add cameras and record:

Camera Image
+
Robot State
+
Robot Action
Enter fullscreen mode Exit fullscreen mode

as a LeRobotDataset.

That will allow us to start collecting demonstration data for imitation learning.

Top comments (0)