The early version of this rig kept walking into walls. Seven depth rays fanned out across the player's field of view, and a doorway narrow enough to slip between two of them simply did not exist as far as the AI was concerned. It would shuffle forward, hit drywall, back up, and try the same wall again. Andrea Ricci pushed the scan to 51 rays, and the corridors of Doom finally opened up.
What Ricci actually built
The project pairs a SCINTIX P4 system-on-module with Anthropic's Claude Sonnet and hands the model the controls of Doom on real hardware. The SoM is built around Espressif's ESP32-P4, with an ESP32-C6 riding alongside for wireless. Doom runs on the board itself rather than streaming from a PC, drawing to a 7-inch 1024x600 MIPI-DSI panel. The game still renders internally at its original 320x200, so the ESP32-P4's Pixel Processing Accelerator does the upscale in hardware. That single offload cut frame scaling from roughly 24 ms to about 9 ms and held the game at a steady 30 FPS.
How the AI sees the level
The clever restraint here is what Claude is not allowed to know. Instead of reading the game's internal state, the firmware exposes Doom over a lockstep WebSocket. The model issues one action, move, turn, fire, open a door, or swap weapons, the engine advances a handful of frames, then freezes and hands back a fresh observation. That observation is deliberately thin: the 51-ray depth scan for wall distances and openings, a list of enemies and items visible on screen, an ASCII automap covering only the walls already explored, and two hints flagging a closed door ahead or a move that got nowhere. Monsters behind walls stay invisible, the same way they would for you.
Getting there was the usual embedded slog. Past the seven-ray blindness, the real gotcha was sign convention. Reported bearing angles ran opposite to the documentation, so Claude turned left with total confidence when it meant right, right up until every component agreed on one coordinate system.
The takeaway
Claude is a mediocre Doom player. It stalls a second or two every few frames while it works out its next move, and it fights like someone reading the manual mid-firefight. That is fine, because the kill count was never the interesting part. The engineering worth stealing is the perception layer and the lockstep loop, and both port down to cheaper silicon than an ESP32-P4. If you want to try the pattern on hardware you already own, start small: expose a handful of state values over a WebSocket from a plain ESP32 using any of the common Arduino WebSocket libraries, then drive it from a script on your laptop before a model touches it. Ricci's full write-up is on Hackster.
Originally published on blog.circuit.rocks.
Top comments (0)