DEV Community

Krenin
Krenin

Posted on Originally published at krenin88.github.io

Test Automation with agent-device (Part 3)

#claude #ai #flutter #testing

This post covers saving test flows as agent-device .ad scripts and running them again, using a Flutter app. It goes through writing an .ad file, getting the app into the right state before a replay, reading the report when a replay stops and fixing it, and the token cost of agent-driven QA. It is based on agent-device 0.21.31 (2026-10-25).

Installation and the basic loop are in Part 1, and real-device setup is in Part 2.

What an .ad file is

When an agent checks an app, it reads the screen, picks the next thing to tap, and reads the result, over and over. Exploring the same flow this way every time keeps costing time and tokens. An .ad file pins down the result of that exploration as a list of commands, and replay runs the list from top to bottom.

Each line has the form <command> [positional...] [flag...], the same as an agent-device command in the terminal. Quoting follows shell rules too, so a selector containing spaces is wrapped like 'label="..."'. Below is the start of a flow that opens the app fresh, taps a button on the first screen, types into an input, and checks the result screen.

# First-run flow — the app ID comes in via -e APP=
open ${APP} --relaunch
wait 'label="<first-screen button>"' 30000
press 'label="<first-screen button>"'
fill 'id="find-input"' "<text to type>"
press 'label="<search button>" role=button'
wait 'label="<result title>"' 60000
wait text "<text the result should show>" 10000
back
close
Enter fullscreen mode Exit fullscreen mode

A few notes:

  • # comments work. The docs do not say so, so the first line was a comment in a test replay, and it passed.
  • Do not hard-code the app ID. Leave it as ${APP} and pass -e APP=<bundle id> at replay time, and the same file works for builds with different IDs, such as a dev build and a release build.
  • The number after wait is a timeout in milliseconds. Give a long one to steps that wait on the network or on model inference.
  • Device-selection flags stay out of the file. --platform, --serial, --device, and --settle are not part of a script step; they go on the replay command.

There are two ways to make an .ad file. You can start recording with open <bundle id> --relaunch --save-script=<file>.ad, go through the flow, and save with session save-script, or you can write it by hand. The 24-step flow in this post was written by hand, based on a flow the agent had already passed manually.

Getting to a "first launch" state before replaying

replay only runs the commands written down; it does not reset the app. If the flow starts by checking the first-launch consent dialog, an app that still remembers an earlier consent stops right at step 2. So a separate runner script deletes and reinstalls the app before each replay.

# Android emulator
adb -s <serial> uninstall <bundle id>
adb -s <serial> install -r <apk path>
adb -s <serial> shell cmd locale set-app-locales <bundle id> --locales en-US
agent-device replay <file>.ad --platform android --serial <serial> -e APP=<bundle id> --timeout 240000

# iOS simulator
xcrun simctl uninstall <UDID> <bundle id>
xcrun simctl install <UDID> <Runner.app path>
agent-device replay <file>.ad --platform ios --udid <UDID> -e APP=<bundle id> --timeout 300000
Enter fullscreen mode Exit fullscreen mode

On Android, set-app-locales opens just this app in English while leaving the device language alone. On the iOS simulator no per-app language command was used, so the app follows the simulator's language setting. replay has a default work budget of 60 seconds, so raise it with --timeout for longer flows.

Deleting the app also deletes its data. So the runner script accepts only the designated test emulators (checked by name with adb -s <serial> emu avd name) and simulator UDIDs, and stops with exit code 2 if it gets any other device or an app ID outside your own. An emulator serial such as emulator-5554 is assigned in boot order and does not tell you which AVD it is, which is why the check uses the name.

With that in place, the 24-step flow finished with Replayed 24 steps on both runs on each platform: 8.8 and 9.4 seconds on the Android emulator, 13.9 and 14.0 seconds on the iOS simulator.

When a replay stops

When a replay gets stuck partway, it fails with REPLAY_DIVERGENCE. The report includes the step number and its line, the actionable elements on screen at that moment, and selector suggestions for that screen. So you do not have to start over after a fix, it also tells you how to resume with replay --from <step> --plan-digest <value>. When no automatic repair is possible, it says Repair hint: manual.

A first-launch system dialog. The flow that passed on Android stopped at step 2 the first time it ran on the iOS simulator.

Error (REPLAY_DIVERGENCE): Replay failed at step 2 (wait "label=\"<first-screen button>\"" 30000): wait timed out for selector: ...
Current surface: Allow "<app name>" to track your activity across other companies' apps ...
Enter fullscreen mode Exit fullscreen mode

The report's Current surface showed the App Tracking Transparency (ATT) dialog. It is a system dialog that appears on first launch only on iOS, so the flow written against Android had no step for it. After closing it with agent-device alert dismiss and replaying again, the run got past this step. The runner script does not handle it, so on builds that show this dialog, close it before replaying.

Input role names that differ by platform. Next, the replay stopped at the fill in step 5. The first selector was role=text-field, which matched on Android, but the iOS simulator exposed the same input as an other element with a label. To find one selector that works on both platforms, these were tried in order:

  1. editable=true — passed on Android, Selector did not match on iOS
  2. role=textfield || role=text-field — || is the "or" syntax that appeared in the report's selector suggestions. It still did not match on iOS.
  3. Adding Semantics(identifier:) to the Flutter widget — passed on both platforms

The last fix is one line of app code: identifier: 'find-input' added to the existing Semantics around the input, with no visible change on screen. The selector becomes id="find-input". As noted in Part 1, a Flutter Key does not show up in the accessibility tree, so adding an identifier is the reliable way to run the same script on both platforms.

Token cost and the rules for cutting it

Token cost was the reason for using .ad in the first place. Most of the cost of agent-driven QA came not from output but from cache reads. A cache read is the earlier conversation the agent re-reads on every response. The first real-device check of two apps took about 40 minutes and came to 121 responses, 77,985 output tokens, and 30,225,404 cache-read tokens, roughly 250,000 tokens re-read per response.

The QA agent was already on Sonnet, so the model was not the cause. In the same stretch, 11 screenshots were read into the conversation, and as the context grew, every response re-read more. So five rules went into the QA agent's instructions:

  1. Save a flow as .ad the first time it passes, and use replay from then on. When it gets stuck, fix only that step using the divergence report.
  2. Act with --settle and look only at the returned diff. Do not read the full screen tree.
  3. Save screenshots to files only, and read at most one per item into the conversation.
  4. End the session per item: one app, one thing to check. Past 40 responses, it is time to stop.
  5. Read logs and build output with grep or tail, only the lines you need.

The batch after the rules (three emulator checks, an iPhone attempt, an Android real device, about 30 minutes) came to 105 responses, 44,113 output tokens, and 19,171,162 cache-read tokens, about 180,000 cache-read tokens per response. The items checked were different, so the two sets of numbers cannot be compared directly, and the 40-response limit was not met. Tokens after adopting .ad have not been measured separately yet, and neither replay on a real device nor running it in CI has been tried.

References

Top comments (0)