New framework merges safety constraints with realistic onboard sensors to help robots evade moving objects without perfect information.
Researchers at Caltech have developed a novel approach for teaching humanoid robots to dodge incoming projectiles using only the vision systems they would realistically carry into the real world. The work, described as PAC-MAN in a new arXiv paper, addresses a fundamental challenge in robotics: enabling machines to make split-second evasive maneuvers when operating with imperfect sensor data.
Most prior work on robot safety assumes access to perfect information about the environment. A robot knows exactly where every obstacle is, exactly how fast a ball is traveling, and can calculate ideal escape trajectories accordingly. But real robots have cameras with limited fields of view, processing delays, and occasional failures. PAC-MAN bridges this gap by training policies that work with degraded sensory input while maintaining safety guarantees.
How the System Works
The framework couples two key techniques: control-barrier functions (CBF), which mathematically ensure a robot stays safe by preventing contact with obstacles, and reinforcement learning (RL), which lets the robot discover effective evasion strategies through trial and error. According to arXiv, the team deployed a perception-aware variant that receives only segmented depth imagery from a head-mounted camera, not the complete state information available during training.
During development, the system had access to highly accurate ball position data for every body segment. This allowed engineers to create safety constraints tailored to each link. But at deployment time, the robot relied solely on a camera feed and had to make decisions under uncertainty. The researchers also incorporated adversarial motion priors, essentially training the robot's reflexes against an imaginary opponent trying to make evasion harder.
Real-World Performance
Photo by Pavel Danilyuk on Pexels.
The team benchmarked their approach in controlled conditions with thrown balls, testing both isolated evasion attempts and longer scenarios where the robot walked back to its starting position between throws. The policy achieved performance nearly as good as an oracle system with perfect state information. More importantly, it succeeded on 95% of real-world throws when deployed on a Unitree G1 humanoid robot.
Interestingly, the type of safety constraint used mattered significantly. Joint-level barriers performed best with accurate ball tracking but degraded when restricted to camera input. Link-level barriers, which reason about safety for each body segment individually, proved more robust to imperfect perception and transferred effectively to hardware without additional tuning.
Why This Matters
The work demonstrates that robots can perform dynamic, safety-critical behaviors in the real world without requiring perfect information. This has implications far beyond dodgeball. Factory robots that must respond to moving human colleagues, autonomous vehicles navigating unpredictable environments, and disaster response machines operating under sensor constraints could all benefit from similar techniques.
The research also reveals important insights about designing AI systems for robots: perception limitations should be considered during training, not treated as afterthoughts. By deliberately training under realistic sensory conditions, the team created a policy that actually works when deployed rather than failing when theory met practice.
- Policy achieved 95% success rate on real robot throws
- Performance within a few percentage points of oracle with perfect information
- Link-level safety constraints proved most robust to sensor uncertainty
- Zero-shot transfer from simulation to physical hardware
As humanoid robots move from research labs into human environments, techniques like PAC-MAN will be essential for ensuring they can react safely to unexpected events while relying on the imperfect sensors they actually carry.
This article was originally published on AI Glimpse.
Top comments (0)