Teaching a four-legged robot to walk is the "hello world" of robot reinforcement learning: thousands of copies of the robot try to follow velocity commands in parallel, and PPO turns their attempts into a walking policy. This walkthrough sets up a Unitree Go2 velocity task with Alloy, an open-source physics engine for robot learning built on MuJoCo Warp and Newton, and shows the one command that trains it.
pip install alloy-engine gymnasium
1. Build the environment and step it
import numpy as np
from alloy.envs.loco import go2_velocity
env = go2_velocity(num_envs=2, device="cpu")
obs, info = env.reset(seed=0)
print(env.obs_groups, {k: tuple(v.shape) for k, v in obs.items()})
for _ in range(10):
obs, reward, terminated, truncated, info = env.step(np.zeros((2, 12), np.float32)) # zeros hold the stand
print(tuple(reward.shape), env.xp.to_numpy(env.state["base.height"]).round(2))
env.close()
('policy', 'critic') {'policy': (2, 55), 'critic': (2, 79)}
(2,) [0.27 0.27]
The environment returns two observation groups: 55 numbers for the policy (what a real robot could measure) and 79 for the critic (the same plus values only a simulator knows, which helps training). The action is 12 joint targets; zeros hold the standing pose, with the base 0.27 m above the ground. The first build compiles kernels, so it takes a few minutes on a CPU.
2. Train on a GPU with one command
pip install "alloy-engine[envs]" rsl-rl-lib
python -m alloy.envs.loco.train --num-envs 4096 --iterations 400 --out runs/go2
This trains PPO with rsl_rl across 4,096 parallel Go2 worlds on one NVIDIA GPU, prints a line every 25 iterations and saves the checkpoint as runs/go2/model_last.pt. Add --terrain rough to train on rough tiles instead of flat ground.
Alloy also ships wrappers for Stable-Baselines3 and skrl; the RL guide covers them.
Keep in mind
- The policy walks blind: it has no height scan, and there are no stairs or gaps.
- There is no actuator model, and no policy trained this way has been run on a real robot yet.
- Training in practice needs a GPU. The docs keep the full list of limits.
Links
- Docs page this is based on: https://alloy.argonrobotics.ai/docs/how-to/quadruped-rl/
- Code: https://github.com/argonrobotics-ai/alloy
- Paper: https://arxiv.org/abs/2611.16312
Top comments (0)