SenseNova posted an evaluation of GPT-6-Astra across 12 robotics benchmarks covering tabletop arms, quadrupeds, mobile bases, and dexterous hands (arXiv:2610.22471).
The headline numbers look strong. When given in-context demonstrations and allowed to expand its execution steps, the model solves multi-stage manipulation tasks that used to require dedicated per-task policies.
The useful part of the report is the failure profile on physical hardware. The authors deployed the model on eight physical robot setups, paired with low-level motor controllers like π0.5. The system broke down in two places: in-context demonstrations for custom robot hardware are expensive to collect, and physical environments don't wait for inference tokens to finish streaming.
In simulation, benchmarks pause time while the high-level planner thinks. If a reasoning model takes three seconds to output a spatial trajectory, the simulated arm sits motionless in empty space. In a physical shop or kitchen, momentum and external objects continue moving. If an object slips or an obstacle drifts across the workspace during that three-second pause, the resulting action executes against a state that expired two seconds ago.
The authors propose a simple shift for the next generation of benchmarks: evaluate agents against an unpaused world clock, with zero in-domain demonstrations.
That matches what anyone running autonomous agents locally runs into within an hour. The failure mode in production is rarely that the model lacked the raw spatial logic to compute a grasp vector. The failure is that deliberate System 2 planning costs round-trip latency, and low-level controllers have to buffer stale coordinates while waiting for confirmation.
Bridging that gap won't come from feeding larger trajectory context windows into a high-level API. It requires treating inference latency as a physical constraint inside the control loop.
Top comments (5)
The failure during real-world execution is often in the latency of decision making rather than a lack of logic. Do you think implementing a control loop that accounts for inference latency could help improve real-time responsiveness?
A hierarchical loop handles this well: run high-frequency dead-reckoning or PD control locally on the robot at 50-100Hz to stay balanced, and let the model update trajectory goals asynchronously at its own latency. When the model falls behind, the local loop holds or yields rather than executing blind actions on stale state.
The same shape shows up in software agents, just slower. In our router a cheap model answers first and a check decides whether to escalate, so the bad case costs two round trips, and the agent acts on the world as it was before the first one started. A coding agent's world mostly waits. A robot's doesn't.
On our tool calling bench the median call takes 0.92 s across 800 tasks, but that median hides the escalated ones, which pay twice. Your post is why we're adding an escalated only time next.
Did the paper report how stale the state was when actions landed, or only that success dropped? That staleness number seems like the one a wall clock benchmark should publish next to the score. If you have a view on how to measure it well, I'd like the feedback.
Not trying to plug anything, but the closest thing we've written is about escalation making answers worse: dev.to/tom_jones_230c4659491adcd/e...
The benchmark measured overall task completion rate rather than phase delay or angular deviation at actuation time, so staleness wasn't isolated as an independent metric. Tracking latency-induced drift between planned trajectory and observed pose is the right way to benchmark this.
Thanks, that's the metric I was missing. Planned versus observed at actuation time translates cleanly to software: the gap between the state an agent read and the state it writes into. Time per call only tells you how long you waited, not how far the world moved while you did.