Post 6 of 8 in the game-factory series.
I ran the ancient Egypt variant through the pipeline for the first time and it passed everything. The Builder produced a clean build. The icons loaded. The themed strings were in the code. Then I ran the Tester, and it told me the login form was broken.
The form wasn't broken.
That failure told me more about where automated testing goes wrong than months of reading could have. It's where this post starts.
What the Tester agent is
The Tester is the QA stage in a pipeline: Designer, Image-Gen, Background-Gen, Builder, Tester, Deployer. Its job is to take the freshly built game, run it in a real browser, and decide whether it works before the Deployer touches anything. The Deployer reads the test report and refuses to proceed if the verdict is fail.
Mechanically: the agent spins up the built game as a local subprocess. It runs npm install, then npm start in the output directory, waits for port 3000 to respond, and launches a Playwright browser session. From there it runs a fixed battery of eleven functional checks: navigate to the app, complete the auth flow, place spins, verify themed strings appear, check for console errors, confirm that every API call carries the right theme ID. Then three visual checks.
The unusual part is how it judges those visual results. After the functional checks, it takes three screenshots — idle page, mid-spin, settled-after-spin — and sends each to a vision-capable model with a rubric:
"Does the title and color palette visibly match the spec? Are the symbols and patterns panel readable? Is the tutorial modal absent?"
The vision model scores the screenshot from 1 to 5 and lists any issues it can see. Playwright can assert on visibility and image dimensions, but my existing assertions didn't cover aesthetic defects — a background that loaded as a flat fallback color, a color palette that was technically present but visually wrong. A model looking at the actual screenshot catches those without writing a specific assertion for each one.
Two models share the work. Haiku handles the main orchestration loop: executing the checklist, clicking buttons, reading console output, filling forms. Sonnet handles the nuanced visual judgment on the idle screenshot, where theme fidelity matters most. The simpler visual checks (mid-spin, post-spin) also use Haiku, because the bar there is lower.
At the end the agent calls a terminal tool, mark_test_complete, with a verdict and a summary. That gets written to .test-report.json:
{
"verdict": "pass",
"functional": { "passed": 11, "total": 11 },
"visual": { "average_score": 4.2, "evaluations": 5 },
"findings": []
}
If verdict is fail, the Deployer reads that and stops. The game doesn't ship.
What it did
On a working build, the Tester runs the full happy path. It registers a fresh smoke-test user with a unique email each run (to avoid conflicts with previous runs), logs in, dismisses the tutorial modal, places a spin, waits for the animation to settle, and checks that a win or no-win message appears. It reads console output for errors and reads network traffic to verify every backend call carries the right theme ID. It runs up to 20 spins watching for themed strings: "Pharaoh's Fortune" for ancient Egypt, "Valhalla Jackpot" for Norse mythology.
The vision evaluation caught things I didn't expect. On one early build, the theme colors were correct but the background image hadn't loaded. The page was technically functional, the DOM had no errors, but the game looked wrong. The vision model flagged it with a score of 2 and the specific note that the background was a flat fallback color. A DOM assertion would have passed that build.
That's what the Tester is genuinely good at: looking at the actual rendered page and deciding whether it looks right.
What went wrong
The failure mode was in the navigation, not the judgment.
To drive the game, the agent had to find and interact with elements: fill the registration form, click the spin button, read result messages. The model did this by guessing selectors. It tried placeholder text, role attributes, visible text labels, CSS class patterns. Most of the time it got close enough. Sometimes it didn't.
The registration form was the worst offender. The form has five fields, and two of them are plain input[type="text"] — one for player name, one for country. There is nothing in the element's type to distinguish them. To fill both correctly, the agent has to know to target input[type="text"] with nth=0 for the first and nth=1 for the second. If it guesses wrong, the wrong field gets filled, the form submits with country empty, the browser's HTML5 validation silently blocks the submit, and nothing happens. No error. No network request. Just silence — which the agent records as a test failure.
Twice the run reported "couldn't find field X" when field X was there and working. Both times the automation had broken on its own selector guess, not on anything wrong in the product.
This is the worst kind of test failure. A real failure tells you something is broken in the product. A false negative tells you something is broken in the test infrastructure. Because the output looks the same, you don't always know which you're looking at immediately. I spent time debugging the auth form before I realized the selector was wrong.
The pattern matters: vision-as-judge was the reliable half; LLM-guessed-selector navigation was the fragile half.
Looking at a screenshot and deciding whether a slot machine matches an ancient Egypt theme is exactly the kind of ambiguous judgment a language model handles well. It can reason about color fidelity, symbol legibility, layout, atmosphere. Reverse-engineering a CSS selector for a button it has never seen the source of is a different task. It's brittle, failure-prone, and not something the model is particularly good at.
The fix is structural. The form fields should have accessible labels — aria-label or associated <label> elements — which also makes the game more accessible. Where a stable user-facing locator doesn't exist, the Builder should stamp data-testid attributes: data-testid="spin-button", data-testid="credit-display". Then the Tester queries by label or role first, falls back to testid, and only guesses as a last resort. That gives the automation stable hooks across every theme variant.
I haven't shipped that yet. It's on the backlog. The evidence for it is sitting in the test logs: two false negatives on builds that were actually working.
What to take from this
Use a vision model as a complementary judge of a rendered UI. It catches aesthetic defects that specific assertions weren't written for: wrong colors, layout that's technically valid but visually broken, a background that loaded as a fallback. It's not a replacement for deterministic assertions — it's probabilistic and occasionally inconsistent. But it covers the gap between "the DOM is correct" and "the game looks right."
But don't make the model navigate by guessing selectors. That's where it breaks. Navigation requires knowing the exact shape of the DOM, and a model with no source access is guessing. Every wrong guess is a potential false negative.
The design that works: give the automation stable, semantic hooks — testid attributes, ARIA roles, predictable form structure — and let the model focus on the judgment calls. Make the thing easy to drive, and reserve the model's cleverness for the evaluation.
For any pipeline where one agent builds an artifact and the next agent tests it, the cleanest version of this is a contract between the two. The Builder promises to produce certain testable hooks. The Tester relies on them. Both agents get simpler. Failures become easier to localize.
Previous: the Builder agent. Next: the Deployer agent.
Top comments (0)