Our Android QA Pipeline Passed. The Screenshots Proved It Was Wrong.

A test can pass for the wrong reason.

That sounds obvious. It is much less obvious when the build is green, the Android emulator launched, the script finished without an error, and a folder full of screenshots is waiting at the end.

This week, while building two Android games in Godot, we had exactly that kind of pass.

The pipeline had done a lot of real work. It built the project, created an Android virtual device, installed the APK, launched the game, injected input, recorded video, and saved screenshots. Every mechanical step completed.

The problem was that the screenshots showed the entry screen instead of gameplay.

The automation had proved that it could interact with something. It had not proved that the player reached the experience we meant to test.

That false pass changed how I think about automated game QA. The hard part is not making an emulator run. The hard part is deciding what counts as evidence.

Why we needed an emulator in the first place

The two projects are Glimmerbook Solitaire and Ithaca Rising. Both are Android games built in Godot 4.

I still test releases on a real Android device. That is where touch feel, screen fit, pacing, motion, and the overall experience have to survive. But development cannot stop every time the physical device is unavailable.

We needed a repeatable loop that could run during development:

1. Build the Android project.
2. Install it on an emulator.
3. Launch from a clean state.
4. Follow a real interaction route.
5. Capture video and screenshots.
6. Fail when the route or visual result is wrong.

The first attempts were too slow and unreliable. The useful path turned out to be the existing Linux CI environment with KVM hardware acceleration enabled, rather than moving the work to a more expensive runner.

KVM made the virtual device practical. ANGLE gave us a workable graphics path for the Godot build. Together, they turned the emulator from an occasional manual tool into part of the repository's QA loop.

Once the pipeline worked, it could build, install, cold-launch, inject real input, and return visual artifacts from the running game.

That was the infrastructure milestone. It was not yet a trustworthy product test.

The first lesson: pixels are part of the assertion

Traditional automated tests are comfortable with values.

Did the process exit successfully? Did a file exist? Did a button respond? Did the current scene name change?

For a game, those checks are necessary but incomplete. A successful tap does not mean the right button was visible. A scene change does not mean the screen was readable. A running process does not mean an Android overlay is not covering the game.

Our emulator runs caught issues that code-level checks would not have explained well: a full-screen Android notice sitting over the game, an unwanted orientation, and an input route that missed a return action and invalidated the later part of the test.

Then came the false pass on Glimmerbook Solitaire.

The harness completed, but the captured pixels showed that it had not reached the intended gameplay route. If we had treated the exit code as the result, we would have reported success. Looking at the screenshots overturned that conclusion.

The visual artifact was not decoration for the test report. It was the decisive assertion.

That led to a simple rule: a gameplay test is not complete until someone, or a reliable visual check, confirms what was actually on screen.

The second lesson: test a route, not a tap sequence

A script that taps coordinates can be useful, but coordinates alone do not describe intent.

A sequence such as "tap here, wait, tap there" may keep running even after the UI takes a different path. Every later action can become meaningless while the script still reaches the end.

We changed the harness to think in story beats.

For the early Glimmerbook experience, the route now has explicit milestones: the early win, the keepsake reveal, the modifier choice, and the next-story tease. Each milestone writes a route marker. The emulator script fails unless all four markers appear.

The difference is important.

A coordinate says where the harness touched. A route marker says what the game reached.

We still keep screen-change checks and a final story-wrap assertion, but the markers make the intended player journey legible. When a run fails, we can see which beat never happened instead of only knowing that a later screenshot was wrong.

This is closer to how I evaluate the game as a product. I do not care that four buttons were technically tappable. I care that a first-time player experienced a coherent sequence and reached the reward, choice, and reason to continue.

The third lesson: visual QA changes product decisions

The emulator is not only a regression tool. It also gives us evidence about the experience.

A recent Glimmerbook run produced a short video covering the full early sequence. Reviewing the actual frames showed that the result screen was denser than the three screens that followed it. The keepsake name and the call to action nearly touched.

That was not a crash. No logic test would flag it. The route was correct.

It was still a product defect because the visual hierarchy broke at the moment the player should understand the reward and choose to continue.

The fix was small in code: move the icon, name, and call to action to create a deliberate gap, then update the harness tap point. The reason for the fix came from watching the pixels, not reading the implementation.

This is where automated capture becomes useful beyond "does it run?" It gives the team a stable object to inspect. We can compare pacing, spacing, and transitions across runs instead of relying on memory or a convenient screenshot.

What the harness proves, and what it does not

The current loop gives us stronger evidence than a green build:

• The APK builds and installs.
• The game cold-launches in an Android environment.
• The scripted input can follow the intended route.
• Required story milestones actually occur.
• Video and screenshots show what appeared on screen.

It still does not prove everything.

An emulator is not the target phone. It cannot settle how haptics feel, how a foldable screen behaves in every posture, or whether the final pacing feels right in a person's hands. Those questions still need device testing and human judgment.

The goal is not to replace that judgment. It is to make every earlier loop more honest.

A build should not wait for a person before anyone knows whether it launches. A route should not be called verified when the screenshots show the wrong screen. A visually crowded reward moment should not hide behind passing logic checks.

The better definition of a pass

I used to think of the emulator pipeline as a way to keep development moving.

It is that, but the more useful outcome is a better definition of done.

A pass is not "the script finished."

For this kind of work, a pass means the build ran, the intended route happened, the required milestones appeared, and the pixels support the claim we are making.

That standard creates more failures in the short term. It is supposed to.

The false pass was not wasted work. Catching it prevented us from turning weak evidence into a confident report. More importantly, it forced the QA system to describe the player experience rather than the automation procedure.

That is the standard I want as these games grow: not more green checks, but fewer ways for a green check to lie.

A note on how this article was made

Instinct prepared this draft from our real project history and did the heavy lifting in turning the Android QA work into a technical narrative. I reviewed the facts, chose what should be public, and remain responsible for every claim published under my name.