Disclosure: I work on Phonebox, which gives AI agents cloud Android phones. This post is about the testing approach, not the product; the last section shows one way to try it.
"Agentic testing" is on every testing vendor's homepage this year, and the definitions don't quite agree. Here's a practical one, from a mobile point of view.
The definition
Agentic testing means an AI agent runs the test: you state a goal in plain language, and the agent decides each next action from what it observes on screen, acts, checks the result and adapts until the goal passes or fails.
That separates it from two things it often gets confused with:
- Scripted testing: the steps are fixed in advance, in Espresso, UI Automator, Appium or Maestro.
- AI-assisted testing: AI writes or repairs a script (Copilot writing Espresso, self-healing locators), and the script then runs as usual.
Tricentis, Momentic, Katalon and Autify all draw the line roughly here. Forrester calls the category "autonomous testing platforms" and doesn't use the word "agentic" at all.
Three ways to automate a UI test
| Scripted | AI-assisted | Agentic | |
|---|---|---|---|
| Who decides the steps | The engineer, in code | AI writes the code; it runs like any script | The agent, at run time |
| When the UI changes | The test breaks | AI repairs the locator | The agent finds another way |
| Repeatability | Exact | Exact | Different paths to the same goal |
| Mobile examples | Espresso, UI Automator, Appium, Maestro | Copilot writing Espresso, self-healing locators | Claude Code on a cloud phone, Journeys in Android Studio |
Where it falls short
- Non-determinism: two runs can take different paths, so a failure can't always be replayed step for step.
- Cost and speed: every step is a model call, so it's slower and dearer than a script.
- Gestures: Google lists long-press, double tap, multi-finger gestures and rotation among the gaps in its own Journeys preview.
- Judgment: the agent only knows the business rules you write down.
So it isn't a replacement for the suite that gates your merges. It's good at the things scripts are bad at: new screens, exploratory passes and reproducing bug reports.
Trying it on an Android app
The agent needs a device it can read and act on. On a laptop that's an emulator plus a local MCP server like mobile-mcp. If the agent runs somewhere without an emulator, a hosted one works:
claude mcp add --transport http phonebox https://phonebox.dev/mcp \
--header "Authorization: Bearer $PHONEBOX_API_KEY"
Then give it a goal, not steps:
Install the debug APK on my Phonebox phone and open it.
Sign in with the test account from TEST_ACCOUNTS.md, open each tab
in the bottom bar once, then sign out. Read the screen before every
action. Report any crash, error message or empty screen, quoting the
screen text. Park the phone when you finish.
The two habits that matter most: tell it to read the screen before every action, and ask for evidence (the exact screen text, or a recording) so the report can be checked.
Originally published on phonebox.dev. Curious how others are drawing the line between agent runs and scripted suites.
Top comments (0)