I'm developing a belt-scroll beat-'em-up set in Bakumatsu-era Japan — swordsmen, the Shinsengumi, lots of slashing. It's a browser game built with Phaser, and it has hundreds of sprite animations across a growing cast of characters. Every animation change has to be checked: does the character stay grounded, does the blade actually reach, does it look right on a phone?
I can't hand-test every motion after every change. So my AI development partner, Muse, playtests the game for me — in a headless browser, driving the real game through its real input paths. This article is about the most valuable lesson that process taught us, which was mostly a lesson about how not to verify a visual change.
The rig
The test setup loads the actual game, puts it into a practice mode with a training dummy, and plays it programmatically: keyboard input for movement and attacks, touch events dispatched the way a phone would produce them. A harness runs suites of checks — spawn this character, perform this combo, measure the result.
Two hard-won rules are baked into that harness. First, tests must exercise the production input path. Early on, we injected synthetic keyboard events that looked right but never reached the game's input handlers the way real keys do, and "passed" tests for input that was actually broken. Second, time in a headless rig doesn't flow like time on a phone — under load, the game may crawl at a couple of frames per second, so tests wait on game state ("has the thrown enemy landed?") instead of sleeping fixed durations.
The failure: correct poses, identical pixels
The defining failure came with an animation rework. The agent regenerated a set of motions, wired them in, and ran the checks. The tests verified the animation state machine: the character entered the right poses, in the right order, for the right number of frames. Everything passed. The report said the new motions were in.
I opened the game on my phone. The character looked the same as before.
Digging in, the truth was embarrassing: the new frames existed and the transitions were correct, but rendered on screen, the silhouettes were nearly pixel-identical to the old ones. The verification had checked the names of things — pose A follows pose B — and names don't tell you what a player sees.
The fix: measure pixels, not poses
We rebuilt the verification around a simple principle: a visual change is only verified when the rendered pixels measurably change in the intended way.
Concretely, that now means:
- Sprite sheet audits with numeric gates. Before a sheet even enters the game, we measure foot width, character height, and the ground-contact line for every pose, directly from the image data. If a "wider stance" pose doesn't actually measure wider, the sheet is rejected before it wastes a playtest.
- In-game screenshot measurement. The rig captures the running game and measures the sprite's on-screen width and extent per pose. A claim like "this attack now reaches further" has to show up as numbers.
- Guard against empty measurements. A snapshot API in our rig could return a blank buffer, and ratios computed from zero pixels happily report "1.000 — perfect match." Measurements now require a minimum count of actual content pixels before they're allowed to pass anything.
- Real-device conditions. Headless screenshots at desktop density missed problems that appear on an actual phone, where the canvas is scaled up. Verification captures now emulate a phone's resolution and pixel density, including zoomed crops of the result.
Both orientations, always
The other structural rule: every test suite runs in landscape and portrait. This exists because of bugs that only existed in one direction — a UI element whose explanation text was hidden behind another panel in portrait, a button that rendered at text height because a padding value was read from the wrong place. A coverage checker now scans the test suites themselves and fails the whole run if any suite lacks dual-orientation coverage. The tests have tests.
What "done" means now
Today, when the agent reports a visual fix, the report has to include the measurements: before and after, in pixels, on an emulated phone screen, in both orientations. "The pose transitions are correct" is accepted as a statement about plumbing, never as a statement about what the game looks like.
The funny part is that this discipline turned out to be exactly what a good human QA lead would insist on. The agent didn't lack effort or honesty; it lacked a definition of done that matched what a player experiences. Once we wrote that definition down — measure the screen, not the data — the playtesting became something I can actually trust, and my own phone checks turned from "catch the agent's mistakes" back into what they should be: judging whether the game feels right.
Indie game developer in Japan. I build games like Samurai 100 Slash Lite and Acid Rogue Shooting, and write about developing with AI agents. Portfolio: https://portfolio.wabiplant.workers.dev/
Top comments (0)