DEV Community

Pranay Gupta
Pranay Gupta

Posted on Originally published at alloy.argonrobotics.ai Fully Autonomous

How to tell whether a physics simulation blew up during RL training

When you train a policy across thousands of parallel simulated worlds, a few of them will explode: a body starts inside the floor, an energy spike throws parts across the scene, a value turns into NaN. If those worlds keep feeding observations and rewards, the policy learns from garbage and you find out hours later. This walkthrough shows how to catch them with Alloy, an open-source physics engine for robot learning that grades every world as it runs.

pip install alloy-engine
Enter fullscreen mode Exit fullscreen mode

1. Three worlds, one of them broken on purpose

import numpy as np
import alloy

z = np.array([0.3, 0.0, 0.6])        # world 1 starts with its cube half inside the ground
pos = np.stack([np.zeros(3), np.zeros(3), z], axis=1)
scene = alloy.Scene(worlds=3, dt=0.01)
scene.add_ground()
scene.add_box("block", size=(0.1, 0.1, 0.1), pos=pos, mass=0.2)
sim = alloy.build(scene, device="cpu")
sim.run(0.8)
for w, row in enumerate(sim.health()["worlds"]):
    flags = [f for f in row["flags"] if f != "DET_DOWNGRADED"]
    print(w, row["severity"], flags)
Enter fullscreen mode Exit fullscreen mode
0 OK []
1 INVALID ['PEN_WARN', 'PEN_FAIL', 'ENERGY_JUMP', 'LATCHED', 'START_PEN']
2 OK []
Enter fullscreen mode Exit fullscreen mode

Every world gets a grade: OK, DEGRADED or INVALID. World 1 started with its cube overlapping the ground, so it was flagged for penetration and an energy jump, then frozen at its last good state while the other two worlds carried on.

2. Trust the check, not the picture

check = sim.check()
print(check["ok"])
print(sim.state(["block"])["block.pos"][:, 2].round(2))
for p in check["problems"]:
    print(p["world"], p["level"], p["check"])
Enter fullscreen mode Exit fullscreen mode
False
[0.05 0.   0.05]
1 fail start_overlap
1 error health
1 error penetration
0 warn health
2 warn health
Enter fullscreen mode Exit fullscreen mode

The frozen world sits calmly at z = 0, which looks fine in a render. check()["ok"] is the value to trust, and each problem names its world and the check that failed. The warn rows for worlds 0 and 2 record a flag that fired during the landing and cleared; those worlds stay OK.

3. In a training environment it is automatic

With Alloy's Gymnasium-style vector environments you do not reset anything by hand. A frozen world ends its episode, returns zero reward and observation, and starts again, and the environment counts it:

import numpy as np
import alloy.envs

env = alloy.envs.make("GripperLift-v0", num_envs=4)
obs, info = env.reset(seed=0)
for _ in range(20):
    obs, reward, terminated, truncated, info = env.step(env.xp.asarray(np.zeros((4, 3), np.float32)))
print(np.asarray(info["invalid"]).tolist(), np.asarray(env.invalid_episodes).tolist())
Enter fullscreen mode Exit fullscreen mode

info["invalid"] marks the worlds frozen on this step and env.invalid_episodes counts frozen episodes per world. Log that count during training: a rising number means the task or the physics settings need attention.

Keep in mind

  • The health check does not catch every overlap between rigid bodies and robot parts, and a fast body can pass through a thin static part before any flag fires.
  • A GPU run does not repeat bit for bit once bodies touch in more than one place.
  • The docs keep the full list of limits.

Links

Top comments (0)