DrivingBench reports that GPT-6 Astra completed a 134.7-metre cone course in a real 2022 Toyota Corolla, becoming the only finisher among four tested systems. The result is a genuine physical-world agent demonstration, but it does not show public-road autonomy or replace the fast local policy, safety driver and vehicle-control stack that made the test safe.
Key facts
- Astra finished its second attempt in 5 minutes 22 seconds using 24 commands.
- The trace reports 6.6 million tokens and $7.74 of model cost.
- Astra was the only finisher; Fable reached 45%, Grok 11%, and Sol 6%.
- Primary source: DrivingBench's official report.
The harness gave the model a deliberately narrow interface: observe, set_motion and stop_now via Model Context Protocol tools. Astra saw the scene roughly every five to six seconds and made high-level commands. A modified openpilot/comma system executed the low-level loop at 100 Hz. The published trace records each action, including the successful route around cones. That separation is decisive. The model was the slow planner, not the reflex system controlling a vehicle's stability moment by moment.
A helpful analogy is a manager giving occasional steering instructions to a skilled driver rather than holding the wheel at highway speed. The system can decide “turn left more” based on a broad observation, while the driver—or here the controller—must make thousands of small corrections that cannot wait for cloud inference. The harness repository documents the engineering around that distinction.
The headline number is vivid but must be read in full. The run covered 134.7 metres in 5:22, at low speed, with an empty lot and a human safety driver. The report says Astra used 100% steering on 20 of 32 commands, not 20 of 24; this was evidently a coarse and sometimes awkward interface. That is why the benchmark is better at asking “can a language-model agent direct a constrained physical task?” than “can this model drive?”
The strongest criticism in the Hacker News discussion is correct: cloud latency and sparse observations make this a poor stand-in for real road driving. It does not test rain, pedestrians, traffic negotiation, sensor failure, emergency braking or the long tail of safety-critical scenarios. The comparison with vision-language-action models is useful: embodiment is a chain of perception, action timing, feedback and safety, not a single benchmark score.
Why publish it anyway? Real-world evaluation often exposes a boundary that simulation hides. DrivingBench has shown that current general models can consume perception, plan around visible obstacles and operate a constrained tool surface in a real vehicle. The so-what is not a robotaxi deadline. It is a design pattern: deploy general reasoning above a high-rate verified controller, then make the boundary between the two explicit and testable.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)