Stop letting your AI agent test Android apps with screenshots

I shipped an Android app mostly by pairing with Claude Code. Writing features went fine. Testing them on a real phone did not.

The problem

When an AI coding agent tests an app on a device, the usual loop is: take a screenshot, look at it, decide where to tap, tap, take another screenshot. Every step sends an image to the model. A 20-step login-and-checkout flow can cost thousands of image tokens, fill the context window, and the agent still sometimes taps the wrong thing.

Most of that work doesn't need eyes. "Tap Sign in, wait, check that Home is on screen" is a text problem.

The idea

Android can already describe the screen as text: uiautomator dump returns every visible node with its text and bounds. So instead of the agent driving the phone step by step, it writes the steps once as JSON:

{
  "name": "Login works",
  "app": "com.example.app",
  "steps": [
    {"app": "restart"},
    {"type": ["Email", "[e-mail]"]},    {"type": ["Password", "secret123"]},
    {"tap": "Sign in", "exact": true},
    {"expect": ["Home"], "wait": 3}
  ]
}

and runs it with one command:

$ python scenario.py scenarios/login.json
PASS Login works (5 steps, 12 s)

One line back. If a step fails, the runner prints the failing step, what was on screen, and saves a screenshot, which is usually enough for the model to fix either the test or the app:

  ✗  5 expect: expected ['Home'] (found []); screen: ['Sign in', 'Wrong password', ...]
FAIL Login works step 5/5 (9 s)

The model only opens a screenshot when a visual decision is actually needed (layout, colours).

What real devices taught me

Most of the code is not the happy path; it's the stuff that broke on real phones:

• Stale dumps. When uiautomator dump fails during an animation, the old ui.xml is still on the device and gets pulled again, so the test "sees" the previous screen. The runner deletes the file on both sides before every dump.
• The navigation bar. Text drawn under the translucent system bar is "visible", but tapping it presses Home. Those nodes are ignored.
• Password fields drop characters when adb input text sends the whole string at once. Type character by character.
• Other apps steal the foreground (ads, OEM notification managers). The runner notices and brings the app back.
• Switches on settings rows with a multi-line description sit ~180 px below the title. Matching by distance fails; matching the smallest container that holds both the text and one Switch works.
• Airplane mode lies. Android remembers Wi-Fi you turned on manually while in airplane mode, so an "offline" test can silently run online. The runner turns Wi-Fi off explicitly and waits until wlan0 has no IP.
• Airplane mode kills wireless adb, stranding the phone. It's refused unless you're on USB.

Using it from an agent

Add a few lines to your CLAUDE.md / AGENTS.md:

## Device testing
- Never drive the phone step by step with screenshots. Write a scenario in
  scenarios/<name>.json and run: python scenario.py scenarios/<name>.json | tail -3
- To see the current screen as text: python ui.py dump

I ran ~90 scenarios this way on my own app before extracting the runner into its own repo. It's Python standard library + adb, nothing to install on the phone, MIT licensed:

https://github.com/sinangumuskabak-sys/android-scenario-runner

It isn't a replacement for Espresso or Maestro. It's a single file you can hand to an agent so it tests cheaply. Issues and PRs welcome, especially from people on OEM skins I haven't tried.

Comments

[Brian · AI News]

The shift from screenshot loops to uiautomator text and one JSON scenario should cut both token waste and flaky taps. The stale dump cleanup and explicit Wi Fi checks are the kind of ugly device details that decide whether an agent test runner survives real phones. I also like the fallback that keeps screenshots for actual visual decisions instead of pretending every screen needs vision.