DEV Community

Duchan
Duchan

Posted on

Coordinate-based UI tests break. So we read the accessibility tree instead — from inside the simulator.

#webdev #opensource #testing #ios

Every recorded mobile test I have ever inherited died the same way: someone moved a button.

The recording said "tap at (340, 712)". The redesign moved that button up by one row, and the test kept tapping — now on empty space, or whatever happened to land there instead. It didn't fail right away. Three sprints later, it started failing in confusing ways, and by then nobody trusted the suite anymore.

The fix isn't a better recorder. It's recording a different thing: not where you tapped, but what you tapped. That needs an element tree, and for a while we didn't have one.

tapflow is an open-source, self-hosted tool that streams iOS simulators and Android emulators into a browser, so a whole team can test builds without installing anything. Until now, everything it moved was pixels in one direction and taps in the other. This post is about getting an element tree out of a simulator with no window, on both platforms. What we do with that tree — replaying flows that survive a redesign — is the next post in this series.

The automation axis this feeds — the flow runner and the MCP server — is experimental. The manual browser QA path is the mature one.

The constraint: no WebDriverAgent, and no simulator window

tapflow already injects touches into the iOS simulator without WebDriverAgent — it loads CoreSimulator.framework and pushes HID events through SimDeviceLegacyHIDClient (that story is ep.1). Streaming reads the framebuffer IOSurface directly. Neither path needs Simulator.app on screen, and that's deliberate: an agent Mac in a closet running four simulators shouldn't be babysitting four windows.

So whatever we used for the tree had to follow the same rule. No WDA to install and keep in sync with Xcode. No simulator window on screen.

Our first attempt ran into exactly that limitation. macOS exposes an accessibility API (AXUIElement), and Simulator.app publishes its content through it. We wrote a helper around it, and it worked perfectly on a developer's laptop. On the headless path it returned nothing, because the AX bridge only exists while the simulator window is being rendered. A simulator started with simctl boot in current Xcode doesn't open a window at all.

So we ended up with a tree reader that only worked in the one situation where we didn't actually need it.

A resident XCUITest runner, living inside the simulator

XCUITest reads the tree of any app by bundle id — that's what it's designed to do — and it runs inside the simulator, so no window is involved. The catch is that xcodebuild test is built around "run a test, print results, exit," while we need something that can answer queries for hours.

So the test target doesn't actually test anything. It starts an HTTP server and blocks:

// TreeRunner — the UI test that never finishes
func testServeTree() {
    let server = try! TreeServer(port: port)
    server.start()
    // block forever: the process stays resident and keeps serving
    RunLoop.current.run()
}
Enter fullscreen mode Exit fullscreen mode

The server is about 100 lines of Network.framework with two routes: GET /health for readiness, and GET /tree?bundleId=<id>, which returns XCUIApplication(bundleIdentifier:).debugDescription — the full element subtree as text, including role, label, identifier, and frame for everything on screen. It's not a documented API contract, and I'll come back to that in the limits section.

The network binding was the detail I most underestimated. The iOS simulator shares the host's network stack, so a default NWListener bind listens on all interfaces. That would expose every screen of the app under test to the local network with no authentication, which is exactly what a self-hosted setup is supposed to avoid. requiredInterfaceType isn't enough. You have to pin the local endpoint:

params.requiredLocalEndpoint = NWEndpoint.hostPort(
    host: NWEndpoint.Host("127.0.0.1"), port: nwPort
)
Enter fullscreen mode Exit fullscreen mode

A failed bind also exits the process immediately, because a listener quietly sitting in .failed looks alive to the parent process and turns a stale-port collision into a two-minute mystery after a 90-second readiness poll.

On the Node side, XCUITreeReader builds the runner once with build-for-testing (cached, so only the first query pays for it), launches it detached with test-without-building, polls /health, then serves queries from the resident process. It starts lazily, on the first tree query, so manual QA — the primary path, where a designer just wants to check a build — never pays the startup cost for a runner it isn't going to use.

Shutdown kills the detached process group and then runs simctl terminate on the in-simulator host app too, since that process isn't part of the group and would otherwise keep holding the port:

// detached spawn → the child leads its own group; kill the whole group
if (pid) process.kill(-pid, 'SIGTERM')
execFile('xcrun', ['simctl', 'terminate', udid, RUNNER_HOST_BUNDLE], () => {})
Enter fullscreen mode Exit fullscreen mode

One rule matters more than anything else: a tree read never returns an empty array to mean "something went wrong." A garbage response body, a wrong bundle id, or an app that's no longer in the foreground all produce explicit errors. "The screen has no elements" should only ever mean exactly that. If it can also mean the infrastructure failed, every test built on top of it becomes ambiguous.

Android was the easy one

uiautomator dump already gives you an XML hierarchy, and the parser is straightforward. The only thing worth watching for is that uiautomator waits for the window to go idle before dumping, so an app with a continuous animation — a spinner or a looping splash screen — never reaches that state and the dump hangs.

The timeout has to run on the device, not on the host:

adb exec-out timeout 10 uiautomator dump /dev/tty
Enter fullscreen mode Exit fullscreen mode

That timeout comes from Android's toybox. The same rule applies as on iOS: infrastructure failures should never look like valid UI output.

One schema, and why frame normalization matters

Both backends parse into the same element shape:

interface UIElement {
  role: 'button' | 'text' | 'input' | 'cell' | 'image' | ...
  label: string
  identifier: string
  frame: { x: number; y: number; width: number; height: number } // 0–1
  enabled: boolean
}
Enter fullscreen mode Exit fullscreen mode

role normalizes two vocabularies into one: iOS XCUIElementType and Android class names (AppCompatButton, AutoCompleteTextView, RecyclerView), ordered so composite names resolve before their generic substrings — ToggleButton has to match switch before Button.

The biggest simplification comes from normalizing every frame into the same 0–1 coordinate space that the touch path already uses. iOS debugDescription frames arrive in points and are divided by the window frame. Android bounds are divided by the root node, which already spans the display, so landscape mode works without an extra wm size query.

After that, the whole pipeline becomes:

tree query → element.frame center → tap(x, y) → the same HID path a human tap uses
Enter fullscreen mode Exit fullscreen mode

No second connection to the device, no separate driver, and no extra coordinate conversion.

What this enables

A test used to have exactly one way to describe where to tap:

(340, 712)
Enter fullscreen mode Exit fullscreen mode

Now it can simply say:

tapOn: "Sign in"
Enter fullscreen mode Exit fullscreen mode

That resolves against the tree — exact identifier first, then exact label, then partial label — and the matched element's frame center goes through the same tap path shown above.

The tree isn't only for the flow runner, either:

GET /api/v1/sessions/:sessionId/ui-tree     # REST
query_ui_tree                               # MCP tool
Enter fullscreen mode Exit fullscreen mode

An LLM agent driving a session reads exactly the same schema a flow does. ep.4 gave an agent eyes through screenshots. This gives it names for what it's looking at.

Limits worth knowing

  • The iOS tree comes from debugDescription. It's a text format, not a stable API. The parser is a pure function with fixture tests, so an Xcode format change fails a unit test before production — but the format can still change.
  • enabled is approximate on iOS. debugDescription doesn't expose it, so elements default to enabled: true.
  • One resident runner at a time. It listens on a fixed port (xcodebuild doesn't propagate host environment variables into the in-simulator runner, so per-device ports aren't available that way). Concurrent multi-device tree queries are deferred.

Try it 

npm install -g tapflow
tapflow start          # → http://localhost:4000
Enter fullscreen mode Exit fullscreen mode

Next in the series: what you can build once a test can name the thing it taps, and the four places we made the runner refuse to guess.

If you've solved the headless tree problem another way, especially something more stable than debugDescription that doesn't require WebDriverAgent, I'd be interested to hear about it.

Top comments (0)