From Monkeys to Agents: How Mobile App Testing Architecture Changed

Every mobile testing architecture answers two questions:

1. How does the test find things on the screen? (the contract between test and app)
2. How does the test know something is wrong? (the oracle)

Over the last 15 years both answers have changed several times. Tests went from tapping coordinates, to element IDs, to plain-English intents. Bug detection went from crash-only, to hand-written assertions, to models judging the screen. Following those two questions explains most of what you see in today's tooling, including why "AI testing" is better at some things than others.

The rest of this post goes era by era, with real numbers wherever someone has published them.

---

Era 1: Throw random taps at it

The first automated mobile "tester" shipped with the Android SDK. Monkey is "a program that runs on your emulator or device and generates pseudo-random streams of user events" (Android docs).

adb shell monkey -p com.example.app -v 500

• Contract: none. It sends taps and swipes to coordinates.
• Oracle: the app crashed, or it didn't.

This is crude, and it still earns its keep: it's free, it never goes stale, and crashes are unambiguous. What it can't do is log in, fill in a form properly, or tell you that the checkout total is wrong.

Era 2: Scripts and selectors

Next came the architecture most teams still run in production: a human writes a script that finds elements by ID or accessibility label, acts on them, and asserts on the result.

• Appium started in 2011 as Dan Cuellar's "iOSAuto", was open-sourced in August 2012, and was rewritten in Node.js after Sauce Labs funded it in January 2013 (Wikipedia). It brought the WebDriver model to mobile, so one test could drive both iOS and Android.
• Espresso arrived on 18 October 2013. Google said it already ran in more than 30 of its own apps, including Drive and Maps, and that it "removes the need to think about the complexity of multi-threaded testing" (Google Testing Blog).
• XCUITest shipped with Xcode 7 in 2015. An Apple engineer wrote: "The old UI Automation support in Instruments is deprecated. UI testing in XCTest is the replacement" (Apple Developer Forums).

onView(withId(R.id.login_button)).perform(click())
onView(withId(R.id.welcome_text)).check(matches(withText("Hi, Sam")))

• Contract: element IDs and the view hierarchy.
• Oracle: whatever assertions a human wrote.

This was a big step forward. Tests became deterministic, fast, and part of CI. It also locked in the two costs mobile teams have complained about ever since:

1. Maintenance. Rename an ID or redesign a screen and the test breaks, even though the feature still works.
2. Coverage is capped by imagination. The suite only checks what someone thought to write down.

Era 3: Let the machine explore

Research and big platforms went after the second problem by having software explore the app on its own.

• Firebase Test Lab's Robo test "analyzes the structure of your app's user interface and then explores it systematically", simulating user activity automatically (Firebase docs, translated from German).
• Sapienz (2016) treated testing as multi-objective search: maximise coverage and crashes while keeping reproduction sequences short. Its paper reported 558 previously unknown crashes in 1,000 Google Play apps, and Meta deployed it within 17 months. Meta now says it runs "millions of tests daily" (Meta Engineering).

Exploration scaled well. Its limits showed up when someone measured it against real bugs. The Themis benchmark (FSE 2021) took 52 real crash bugs from 20 open-source apps and ran eight tools against them. Together they missed 18 of the 52, and the best single tool found 24. The authors listed where the tools got stuck: "deep use-case scenarios, specific text inputs, system or app setting changes, specific user interaction patterns, and interactions with external apps" (Su et al.).

Every item on that list needs some understanding of what the app is for, and none of these explorers had any.

Era 4: Make it less flaky

While the research world worked on exploration, practitioners were fighting flakiness. Two ideas spread:

• Tolerance built into the framework. Maestro launched in August 2022 with YAML flows and this philosophy: "Maestro embraces the instability of mobile applications and devices, and counters it under the hood", automatically waiting for content "but no longer than required" (Maestro blog).
• Self-healing locators. When a selector breaks, the tool picks a likely replacement. A 2025 review found 32 of 56 AI testing tools (57%) advertise this. The same review warns that a wrong replacement can let a test pass while clicking the wrong element (Garousi et al.).

Both ideas loosened the contract a little. That trade-off comes back in every era after this one.

Era 5: LLMs join the loop (2023 to 2024)

Once large language models could read a screen description and reply with an action, the missing piece from Era 3 (understanding what the app is for) became available. The first wave of results was large:

• GPTDroid turned GUI testing into a Q&A loop with the LLM. Across 93 apps it got 32% more activity coverage and found 31% more bugs than the best baseline. It also found 53 new bugs on Google Play, 35 of which were confirmed and fixed (ICSE 2024).
• DroidAgent let the LLM set its own testing goals. It reached 61% activity coverage against 51% for the prior state of the art. On a messaging app it decided, unprompted, to create a second account and friend the first one (ICST 2024).
• AdbGPT went the other way: it read a human-written bug report and reproduced it on a device, succeeding 81.3% of the time (ICSE 2024).

The catch was cost. Calling an LLM on every step is slow and expensive, which led to hybrids that use the LLM only when the cheap explorer stalls:

• LLMDroid gained 26% code coverage over classical tools at $0.49 an hour, and a cheaper model kept 94% of that result at $0.03 an hour (FSE 2025).
• LLM-Explorer beat the per-step LLM explorers on coverage at 9x to 148x lower LLM cost (MobiCom 2025).

Era 6: The test becomes an intent

The next shift went back to Era 2's maintenance problem. If an agent can read the screen, the test doesn't need selectors. It can state what the user wants:

Request a ride from the airport to downtown.
Verify a driver is assigned and the fare is shown.

The agent works out the taps fresh on every run.

• Uber's DragonCrawl is one of the most thoroughly documented production examples. Version 1 (2024) ran on a fine-tuned model of about 110M parameters, completed trips in 85 of 89 cities, and reported "99%+ stability" (Uber blog). For version 2 (2026), Uber reports Android pass rates rising from 82% to 92%, onboarding a new flow dropping from 4 days to 2.5 hours, and maintenance falling from 30-40% of engineering time to about 5%, at roughly $200K a year in inference (ASE-Industry 2026).
• WeChat's CAT retrieves similar past tests and calls the LLM only when retrieval isn't confident. It ran 6,300+ test descriptions at 88% completion in production and found 141 bugs (ASE 2024).
• Android Studio Journeys puts the same idea in the IDE: natural-language test steps that Gemini executes and checks "based on what it sees on the device" (Android docs).
• QApilot CoWork brings this idea to its users by taking in the BDD designed by the team and automating the test authoring through its planning and execution agents (QApilot CoWork).

The contract has now moved all the way from coordinates to IDs to intent. Uber's paper also names the cost of that move: "The agent sometimes takes valid but unintended paths, which can mask bugs." A test that adapts around a UI change can just as easily adapt around a regression.

Era 7: Split the doer from the judge (now)

Once navigation was mostly solved, the hard part moved to the oracle. Is this screen actually wrong?

Navigation benchmarks show the gap. On AndroidWorld, the best agent scored 30.6% in 2024 against an 80% human baseline (Rawles et al.). By 2026, several systems are reported at 97% or more, and one multi-agent system claims 116 of 116 tasks (Minitap et al.). Judging is far behind: LLM judges still misclassify 9% to 24% of mobile agent runs (MobileJudgeBench).

So current architectures separate roles:

• Explorer, monitor and detector agents to find non-crash functional bugs. Trident reports 43 new Google Play bugs, 31 of them fixed (ICSE 2026 J1st).
• Planner, player and detector agents using bug-tracker knowledge. The authors were candid about the result: 68 true positives and 52 false positives, against zero false positives from human testers (Su et al.).
• Find, then independently reproduce. Momentic's Mo has a separate agent re-run each functional bug from a clean session before it's reported (Momentic). On Momentic's own web-only benchmark, Mo found 3.1x more labelled bugs than the next-best coding agent, but its recall was still 19.3% (QA Bench).
• QApilot Autonomous Testing takes this idea and supports no-touch sanity testing - the agents involved understand the app, define goals(intent), plan the journey and finish the exploration goals to scan the app like a human, while capturing data related to functional, latency, network and accessibility issues in every screen it sees and every action it performs (QApilot Autonomous Testing).

Meanwhile, the plumbing became standard. Appium ships an official MCP server, so coding agents can drive iOS and Android devices directly (appium-mcp), and mobile-mcp and QApilot MCP do the same for emulators and real devices.

---

The whole story in one table

Era	Contract (how the test finds things)	Oracle (how it knows something is wrong)	Main limit
1. Random input	Coordinates	Crash	No semantics
2. Scripts	IDs / view tree	Human-written assertions	Maintenance; coverage capped by imagination
3. Exploration	View tree + search	Crash	Can't reach deep or meaningful flows
4. Flake reduction	Looser selectors, auto-waits, healing	Assertions	Healing can hide real failures
5. LLM exploration	Screen → LLM → action	Mostly crash	Cost per step
6. Intent execution	Natural-language intent	NL assertions judged by a model	Adaptive paths can mask bugs
7. Multi-agent + verify	Intent + screen + sometimes backend	Independent judge or reproducing agent	False positives; judge reliability

What to take from this

1. Each era loosened the contract. That trade gave up determinism in exchange for lower maintenance and more coverage. It's a good trade as long as you know you're making it.
2. The oracle never kept up. Crashes were easy to detect. "This total is wrong" is hard, and it's where today's tools differ most.
3. The best production systems are hybrids. Uber, WeChat and LLMDroid all wrap the model in deterministic scaffolding: ground-truth correction, retrieval, loop detection, golden datasets. The model brings the flexibility; the scaffolding is why the results can be trusted.
4. Older layers don't go away. Monkey, Espresso and Sapienz-style exploration are all still in use. New architectures sit on top of the old ones.

The next question is where this goes once the judging problem gets solved. That's the follow-up post.

---

Numbers come from the linked papers and engineering blogs. Production figures from Uber, Meta, Tencent and Momentic are self-reported by those companies. The benchmarks measure different things (coverage, crash detection, task completion, labelled-bug recall), so don't compare figures across eras.