Press A on an Xbox pad. The game receives B. Nothing logs an error, nothing crashes, and every automated check still passes.
That was the bug, and it cost more time than the rest of the feature combined. We spent a chunk of this cycle forwarding a controller from your own machine into a real Android device in a browser tab. Most of it was straightforward. One part was not, and it is the part worth writing down, because it fails silently and it will bite anyone who tries this.
If you ship a mobile game with controller support you already know the testing story it replaces: a drawer of phones, a drawer of pads, and a Bluetooth pairing dance every time you want to check whether B still does what it did last sprint.
The easy half: input that the device believes
The naive version of remote input is a synthetic event: you take a tap in the browser, ship the coordinates, and inject them at the app layer. That works for a lot of testing and it falls apart for exactly the cases games care about. A synthetic event aimed at a focused view is not what a controller produces, and anything reading the actual input stack — which is most game engines — will not see it.
So the input has to arrive lower down. On Android, a forwarded controller can be presented to the system as a genuine input device: the kernel enumerates it, the input stack routes it, and a native game receives it the same way it would from a pad plugged into the phone. Nothing in the app has to be instrumented, and nothing has to know it is remote.
That distinction matters more than it sounds. It is the difference between "we can test the menus" and "we can test the game."
The iOS caveat, stated plainly. This is an Android claim. On iOS a forwarded pad can be presented to web content, but native apps do not see it. If you build an iOS game, remote controller testing is not solved by this and I am not going to pretend otherwise.
The hard half: an Xbox pad is not an Xbox pad
Android decides how to interpret a controller's report by looking at its vendor and product ids, then loading the matching key layout file. That layout is not cosmetic. It decides which bit in the report becomes which button.
The first version presented one fixed identity for every pad. Plug in an Xbox controller, and it arrived at the device claiming to be a PlayStation one. Everything looked fine — the device enumerated a controller, the buttons produced events, the sticks moved. But the two key layouts are near mirror images of each other. Read off a real device:
| DualShock | Xbox | |
|---|---|---|
| Right stick | Z / Rz | Rx / Ry |
| Triggers | Rx / Ry | Z / Rz |
| Face buttons | rotated | — |
| Digital L2 / R2 | present | absent |
So the right stick arrived as a trigger, and the face buttons were rotated by one. A mismapped button still produces a perfectly valid event, so nothing anywhere reported an error — which is how you get back to pressing A and receiving B, with nothing to flag it but a human watching the screen going "that's not right."
The fix is that an identity is a descriptor and a button map together, never one alone. Present the ids, and you have committed to the layout that goes with them. Detection reads the vendor id the browser reports for the pad and falls back to the product name — and that fallback is not a nicety. A real DualShock 4 over Bluetooth on macOS reports itself to Chrome as Wireless Controller (STANDARD GAMEPAD), with no vendor or product id in the string at all. Match on vendor id alone and that pad never resolves.
Two things follow that are worth stealing if you are building anything similar:
- An unrecognised pad should fall back to a real identity, not a neutral one. A neutral vendor id is what makes Chrome refuse to classify the pad, and then web content mislabels every button it guesses at. Better to be confidently one specific controller than vaguely none.
- Swapping controllers mid-session has to tear down and re-create the device, because the identity is fixed at creation time. You cannot mutate a pad into a different pad.
That table came off a real device, confirmed bit by bit, because this is not an area where reading the documentation is enough.
And then an agent picked up the controller
The part we did not plan for.
The same controller surface is exposed as tool calls, so an AI agent can hold the pad. It sees the screen and sends button and axis state to the device.
The first attempt at this was useless, in an instructive way. An agent calling one tool per button press produces input separated by a full network round trip — seconds, not milliseconds. Every combo renders as a held button. The device is not wrong; the timing is simply gone.
What works is sending the sequence rather than the presses: a timed series of controller frames, played back on the device at thirty a second. The agent decides what the combo is, and the timing is preserved on the far end where it can actually be honoured.
What this is not: an agent with human reflexes. We measured it. A round trip through the tool interface is around six seconds. A purpose-built local loop — frame in, controller out, no agent — came in at 134 ms median, against a pre-registered bar of about 50 ms for anything that could claim reflex play. It missed. The verdict was to not build the reflex surface, and that verdict stands.
So: the agent can play. It cannot out-react you. Anyone claiming otherwise about their own setup should be asked for the number.
Why bother
The honest pitch is narrow. If your game has controller support and you currently test it by keeping physical pads next to physical phones, this removes the physical part: any device, any pad, from a browser, with the input arriving as real hardware.
That is it. It does not test your game for you, and on iOS it does not reach native apps at all.
If you want to try it, the devices are at robotactions.com — there is a free tier, and the tool interface installs with npx @robotactions/mcp@latest init.
And if you have hit the key-layout problem from the other side — writing the game rather than the harness — I would genuinely like to hear how you found it. My guess is "a tester said the buttons were wrong and nobody believed them for a week."
Top comments (0)