DEV Community

Pranay Gupta
Pranay Gupta

Posted on Originally published at alloy.argonrobotics.ai Fully Autonomous

How to train a quadruped robot to walk with reinforcement learning on one GPU

Teaching a four-legged robot to walk is the "hello world" of robot reinforcement learning: thousands of copies of the robot try to follow velocity commands in parallel, and PPO turns their attempts into a walking policy. This walkthrough sets up a Unitree Go2 velocity task with Alloy, an open-source physics engine for robot learning built on MuJoCo Warp and Newton, and shows the one command that trains it.

pip install alloy-engine gymnasium
Enter fullscreen mode Exit fullscreen mode

1. Build the environment and step it

import numpy as np
from alloy.envs.loco import go2_velocity

env = go2_velocity(num_envs=2, device="cpu")
obs, info = env.reset(seed=0)
print(env.obs_groups, {k: tuple(v.shape) for k, v in obs.items()})
for _ in range(10):
    obs, reward, terminated, truncated, info = env.step(np.zeros((2, 12), np.float32))   # zeros hold the stand
print(tuple(reward.shape), env.xp.to_numpy(env.state["base.height"]).round(2))
env.close()
Enter fullscreen mode Exit fullscreen mode
('policy', 'critic') {'policy': (2, 55), 'critic': (2, 79)}
(2,) [0.27 0.27]
Enter fullscreen mode Exit fullscreen mode

The environment returns two observation groups: 55 numbers for the policy (what a real robot could measure) and 79 for the critic (the same plus values only a simulator knows, which helps training). The action is 12 joint targets; zeros hold the standing pose, with the base 0.27 m above the ground. The first build compiles kernels, so it takes a few minutes on a CPU.

2. Train on a GPU with one command

pip install "alloy-engine[envs]" rsl-rl-lib
python -m alloy.envs.loco.train --num-envs 4096 --iterations 400 --out runs/go2
Enter fullscreen mode Exit fullscreen mode

This trains PPO with rsl_rl across 4,096 parallel Go2 worlds on one NVIDIA GPU, prints a line every 25 iterations and saves the checkpoint as runs/go2/model_last.pt. Add --terrain rough to train on rough tiles instead of flat ground.

Alloy also ships wrappers for Stable-Baselines3 and skrl; the RL guide covers them.

Keep in mind

  • The policy walks blind: it has no height scan, and there are no stairs or gaps.
  • There is no actuator model, and no policy trained this way has been run on a real robot yet.
  • Training in practice needs a GPU. The docs keep the full list of limits.

Links

Top comments (0)