DEV Community

vmodal_ai
vmodal_ai

Posted on

Building and Testing Robot Policies with MuJoCo

#robotics #ai #simulation #reinforcementlearning

Building and Testing Robot Policies with MuJoCo

MuJoCo (Multi-Joint dynamics with Contact) is a fast, accurate physics engine that has become a standard tool for robot learning research — both for training policies with reinforcement learning and for quickly validating imitation-learned policies before deploying them to real hardware. This tutorial covers setting up a MuJoCo environment, wiring it into a control/policy loop, and using it as a testbed for policy evaluation.

Why MuJoCo specifically

Compared to more visually-oriented simulators like Isaac Sim, MuJoCo optimizes for:

  • Simulation speed — thousands of physics steps per second on a single CPU core, which matters enormously for RL algorithms that need millions of environment interactions.
  • Contact and constraint accuracy — MuJoCo's contact model is specifically tuned for articulated robot dynamics, which is why it's a default choice for legged locomotion and manipulation research.
  • A simple, well-documented XML scene format (MJCF) that's easy to hand-edit, script-generate, and version control.
  • A clean Python API (mujoco package) with minimal overhead, making it easy to embed inside custom training loops.

The tradeoff is that MuJoCo's rendering is much simpler than Isaac Sim's RTX pipeline, so it's typically the right tool for physics-heavy policy training and fast iteration, while Isaac Sim (or a rendering layer on top of MuJoCo) is a better fit when photorealistic vision is central to the task.

Installing MuJoCo 

pip install mujoco
Enter fullscreen mode Exit fullscreen mode

The modern mujoco Python package bundles the simulator itself — no separate license or binary download is required, unlike older MuJoCo versions. Verify the install:

import mujoco
print(mujoco.__version__)
Enter fullscreen mode Exit fullscreen mode

Anatomy of an MJCF model

A minimal MJCF file describing a single-joint pendulum-like arm:

<mujoco model="simple_arm">
  <worldbody>
    <light diffuse=".8 .8 .8" pos="0 0 3"/>
    <geom type="plane" size="2 2 0.1" rgba="0.3 0.3 0.3 1"/>
    <body name="link1" pos="0 0 0.5">
      <joint name="joint1" type="hinge" axis="0 1 0" range="-90 90"/>
      <geom type="capsule" fromto="0 0 0  0.3 0 0" size="0.03" rgba="0.8 0.2 0.2 1"/>
      <body name="link2" pos="0.3 0 0">
        <joint name="joint2" type="hinge" axis="0 1 0" range="-90 90"/>
        <geom type="capsule" fromto="0 0 0  0.3 0 0" size="0.03" rgba="0.2 0.2 0.8 1"/>
      </body>
    </body>
  </worldbody>
  <actuator>
    <motor joint="joint1" gear="50"/>
    <motor joint="joint2" gear="50"/>
  </actuator>
</mujoco>
Enter fullscreen mode Exit fullscreen mode

Key elements: worldbody holds the scene hierarchy, body/joint/geom define the kinematic chain and its visual/collision shape, and actuator defines how you'll command the joints (motor, position servo, or velocity servo).

For real robots, you typically don't hand-write this — you convert an existing URDF to MJCF (MuJoCo includes a URDF importer) or use a pre-built MJCF model from a robot manufacturer or a public model zoo like MuJoCo Menagerie.

Loading and stepping the simulation 

import mujoco
import mujoco.viewer

model = mujoco.MjModel.from_xml_path("simple_arm.xml")
data = mujoco.MjData(model)

with mujoco.viewer.launch_passive(model, data) as viewer:
    while viewer.is_running():
        data.ctrl[:] = [0.1, -0.1]  # motor commands for joint1, joint2
        mujoco.mj_step(model, data)
        viewer.sync()
Enter fullscreen mode Exit fullscreen mode

mj_step advances the physics by one timestep (defined in the model's <option timestep="...">, typically 0.002–0.01s). data.ctrl holds the actuator commands; data.qpos and data.qvel hold joint positions and velocities if you need to read state back out.

Wiring in a policy

Once the environment is running, hooking in a trained policy (from the imitation learning pipeline covered earlier in this series, or an RL policy) is straightforward — replace the hardcoded data.ctrl assignment with a policy inference call:

def run_policy_rollout(model, data, policy, num_steps=500):
    trajectory = []
    for _ in range(num_steps):
        obs = build_observation(data)
        action = policy.predict(obs)
        data.ctrl[:] = action
        mujoco.mj_step(model, data)
        trajectory.append({"obs": obs, "action": action, "qpos": data.qpos.copy()})
    return trajectory

def build_observation(data):
    return np.concatenate([data.qpos, data.qvel])
Enter fullscreen mode Exit fullscreen mode

This is one of the most valuable uses of MuJoCo in a robot learning pipeline: running hundreds of rollouts of a candidate policy in minutes, across many randomized initial conditions, before ever loading it onto real hardware.

Building a Gym-style environment wrapper

Most RL and evaluation tooling expects a reset()/step() interface. Wrapping your MuJoCo model this way makes it compatible with standard RL libraries:

class MujocoRobotEnv:
    def __init__(self, xml_path, max_steps=500):
        self.model = mujoco.MjModel.from_xml_path(xml_path)
        self.data = mujoco.MjData(self.model)
        self.max_steps = max_steps
        self.step_count = 0

    def reset(self):
        mujoco.mj_resetData(self.model, self.data)
        self.step_count = 0
        return build_observation(self.data)

    def step(self, action):
        self.data.ctrl[:] = action
        mujoco.mj_step(self.model, self.data)
        self.step_count += 1

        obs = build_observation(self.data)
        reward = self._compute_reward()
        done = self.step_count >= self.max_steps
        return obs, reward, done, {}

    def _compute_reward(self):
        raise NotImplementedError
Enter fullscreen mode Exit fullscreen mode

Randomizing initial conditions for robust evaluation

Testing a policy against a single fixed initial state tells you very little. Randomize joint positions, object poses, and even physical parameters (friction, mass) between resets to get a realistic picture of policy robustness:

def randomize_reset(env, rng):
    obs = env.reset()
    env.data.qpos[:] += rng.uniform(-0.05, 0.05, size=env.data.qpos.shape)
    mujoco.mj_forward(env.model, env.data)
    return build_observation(env.data)
Enter fullscreen mode Exit fullscreen mode

This is the same underlying idea behind domain randomization for training robust policies — covered in depth in a later tutorial in this series — but applied here purely for evaluation, to get an honest read on how a policy performs outside the exact conditions it was trained or tested on by default.

From simulation to reality

MuJoCo is excellent for fast iteration, but a policy that works perfectly in simulation is not guaranteed to work on the real robot — differences in contact dynamics, actuator response, sensor noise, and visual appearance all contribute to what's known as the sim-to-real gap. The next tutorial in this series covers strategies specifically for closing that gap.

Useful Links

Website: www.v-modal.com
SDK Flutter: v-modal/vmodal_sdk_flutter
SDK Android: v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Top comments (0)