DEV Community: Krenin

Test Automation with agent-device (Part 3)

Krenin — Sun, 01 Nov 2026 13:08:42 +0000

This post covers saving test flows as agent-device .ad scripts and running them again, using a Flutter app. It goes through writing an .ad file, getting the app into the right state before a replay, reading the report when a replay stops and fixing it, and the token cost of agent-driven QA. It is based on agent-device 0.21.31 (2026-10-25).

Installation and the basic loop are in Part 1, and real-device setup is in Part 2.

What an .ad file is

When an agent checks an app, it reads the screen, picks the next thing to tap, and reads the result, over and over. Exploring the same flow this way every time keeps costing time and tokens. An .ad file pins down the result of that exploration as a list of commands, and replay runs the list from top to bottom.

Each line has the form <command> [positional...] [flag...], the same as an agent-device command in the terminal. Quoting follows shell rules too, so a selector containing spaces is wrapped like 'label="..."'. Below is the start of a flow that opens the app fresh, taps a button on the first screen, types into an input, and checks the result screen.

# First-run flow — the app ID comes in via -e APP=
open ${APP} --relaunch
wait 'label="<first-screen button>"' 30000
press 'label="<first-screen button>"'
fill 'id="find-input"' "<text to type>"
press 'label="<search button>" role=button'
wait 'label="<result title>"' 60000
wait text "<text the result should show>" 10000
back
close

A few notes:

# comments work. The docs do not say so, so the first line was a comment in a test replay, and it passed.
Do not hard-code the app ID. Leave it as ${APP} and pass -e APP=<bundle id> at replay time, and the same file works for builds with different IDs, such as a dev build and a release build.
The number after wait is a timeout in milliseconds. Give a long one to steps that wait on the network or on model inference.
Device-selection flags stay out of the file. --platform, --serial, --device, and --settle are not part of a script step; they go on the replay command.

There are two ways to make an .ad file. You can start recording with open <bundle id> --relaunch --save-script=<file>.ad, go through the flow, and save with session save-script, or you can write it by hand. The 24-step flow in this post was written by hand, based on a flow the agent had already passed manually.

Getting to a "first launch" state before replaying

replay only runs the commands written down; it does not reset the app. If the flow starts by checking the first-launch consent dialog, an app that still remembers an earlier consent stops right at step 2. So a separate runner script deletes and reinstalls the app before each replay.

# Android emulator
adb -s <serial> uninstall <bundle id>
adb -s <serial> install -r <apk path>
adb -s <serial> shell cmd locale set-app-locales <bundle id> --locales en-US
agent-device replay <file>.ad --platform android --serial <serial> -e APP=<bundle id> --timeout 240000

# iOS simulator
xcrun simctl uninstall <UDID> <bundle id>
xcrun simctl install <UDID> <Runner.app path>
agent-device replay <file>.ad --platform ios --udid <UDID> -e APP=<bundle id> --timeout 300000

On Android, set-app-locales opens just this app in English while leaving the device language alone. On the iOS simulator no per-app language command was used, so the app follows the simulator's language setting. replay has a default work budget of 60 seconds, so raise it with --timeout for longer flows.

Deleting the app also deletes its data. So the runner script accepts only the designated test emulators (checked by name with adb -s <serial> emu avd name) and simulator UDIDs, and stops with exit code 2 if it gets any other device or an app ID outside your own. An emulator serial such as emulator-5554 is assigned in boot order and does not tell you which AVD it is, which is why the check uses the name.

With that in place, the 24-step flow finished with Replayed 24 steps on both runs on each platform: 8.8 and 9.4 seconds on the Android emulator, 13.9 and 14.0 seconds on the iOS simulator.

When a replay stops

When a replay gets stuck partway, it fails with REPLAY_DIVERGENCE. The report includes the step number and its line, the actionable elements on screen at that moment, and selector suggestions for that screen. So you do not have to start over after a fix, it also tells you how to resume with replay --from <step> --plan-digest <value>. When no automatic repair is possible, it says Repair hint: manual.

A first-launch system dialog. The flow that passed on Android stopped at step 2 the first time it ran on the iOS simulator.

Error (REPLAY_DIVERGENCE): Replay failed at step 2 (wait "label=\"<first-screen button>\"" 30000): wait timed out for selector: ...
Current surface: Allow "<app name>" to track your activity across other companies' apps ...

The report's Current surface showed the App Tracking Transparency (ATT) dialog. It is a system dialog that appears on first launch only on iOS, so the flow written against Android had no step for it. After closing it with agent-device alert dismiss and replaying again, the run got past this step. The runner script does not handle it, so on builds that show this dialog, close it before replaying.

Input role names that differ by platform. Next, the replay stopped at the fill in step 5. The first selector was role=text-field, which matched on Android, but the iOS simulator exposed the same input as an other element with a label. To find one selector that works on both platforms, these were tried in order:

editable=true — passed on Android, Selector did not match on iOS
role=textfield || role=text-field — || is the "or" syntax that appeared in the report's selector suggestions. It still did not match on iOS.
Adding Semantics(identifier:) to the Flutter widget — passed on both platforms

The last fix is one line of app code: identifier: 'find-input' added to the existing Semantics around the input, with no visible change on screen. The selector becomes id="find-input". As noted in Part 1, a Flutter Key does not show up in the accessibility tree, so adding an identifier is the reliable way to run the same script on both platforms.

Token cost and the rules for cutting it

Token cost was the reason for using .ad in the first place. Most of the cost of agent-driven QA came not from output but from cache reads. A cache read is the earlier conversation the agent re-reads on every response. The first real-device check of two apps took about 40 minutes and came to 121 responses, 77,985 output tokens, and 30,225,404 cache-read tokens, roughly 250,000 tokens re-read per response.

The QA agent was already on Sonnet, so the model was not the cause. In the same stretch, 11 screenshots were read into the conversation, and as the context grew, every response re-read more. So five rules went into the QA agent's instructions:

Save a flow as .ad the first time it passes, and use replay from then on. When it gets stuck, fix only that step using the divergence report.
Act with --settle and look only at the returned diff. Do not read the full screen tree.
Save screenshots to files only, and read at most one per item into the conversation.
End the session per item: one app, one thing to check. Past 40 responses, it is time to stop.
Read logs and build output with grep or tail, only the lines you need.

The batch after the rules (three emulator checks, an iPhone attempt, an Android real device, about 30 minutes) came to 105 responses, 44,113 output tokens, and 19,171,162 cache-read tokens, about 180,000 cache-read tokens per response. The items checked were different, so the two sets of numbers cannot be compared directly, and the 40-response limit was not met. Tokens after adopting .ad have not been measured separately yet, and neither replay on a real device nor running it in CI has been tried.

References

agent-device repository: https://github.com/callstack/agent-device
agent-device agent setup docs: https://oss.callstack.com/agent-device/docs/agent-setup
Flutter SemanticsProperties.identifier: https://api.flutter.dev/flutter/semantics/SemanticsProperties/identifier.html

Test Automation with agent-device (Part 2)

Krenin — Tue, 27 Oct 2026 08:26:42 +0000

agent-device drives real devices with the same commands it uses for simulators and emulators. A real iPhone, though, needs a signed test app installed on the device to drive the screen, and a real Android device has to be selected explicitly depending on how it is connected. This post covers why real devices were needed, how to prepare an iPhone and an Android device, and what to watch out for when handling apps on them. It is based on agent-device 0.21.31 (2026-10-24).

Installation and the basic loop are in Part 1.

Why real devices

Simulators and emulators are enough to check app screens, but they cannot reproduce how a build shipped through the store behaves on a real user's device. Three things had to be checked on real devices.

Store-build updates and reinstalls. Installing a TestFlight build over the existing App Store version (the update path), and deleting and reinstalling the app and then restoring a backup, can only be checked on a real device. Whether the backup file was overwritten was checked in the device's Files app under iCloud Drive, by comparing file size and modification time before and after the install.

System dialogs on first launch. The order in which system dialogs such as App Tracking Transparency (ATT) appear relative to the app's own screens has to be seen on a real install. A reinstall also resets that consent history, so first work out whether the state you want to test can actually be produced on the device.

Real ads. The Google Mobile Ads SDK automatically treats an emulator as a test device. logcat prints This request is sent from a test device., and even a release build with production ad unit IDs shows only test ads. That means a no-fill (the ad server finds no ad to serve) cannot be reproduced on an emulator either. On a real device that is not registered as a test device, logcat prints a setTestDeviceIds hint and real ads are requested. In that case, never tap the ad (invalid clicks), watch a production ad to the end at most once per session, and judge the rest from the ad-load results in logcat.

Preparing a real iPhone

On a real iPhone, agent-device installs an XCTest runner on the device. The XCTest runner is a test app that reads the screen and taps on your behalf, and it has to be signed with a development certificate to install. The first run takes a while because the runner gets built, and the cable has to stay connected throughout.

Automatic signing. According to the agent-device docs, if an Apple developer account is signed in to Xcode, the team ID is all you need. To change the runner's bundle ID, also set AGENT_DEVICE_IOS_BUNDLE_ID.

export AGENT_DEVICE_IOS_TEAM_ID=<team id>

Manual signing. If Xcode has no account, or the existing development profile is Xcode-managed, automatic signing will not work. In that case, create a development profile for the runner yourself with an App Store Connect API key. fastlane's sigh creates a profile for a wildcard app ID (*) containing the registered devices.

# key.json: key_id, issuer_id, key (contents of the .p8 file), in_house: false
fastlane sigh --development --app_identifier "*" --team_id <team id> \
  --api_key_path key.json --provisioning_name "<profile name>" \
  --output_path <folder> --skip_install --skip_certificate_verification

Putting the .p8 file path in key.json (key_filepath) failed; putting the file contents in key worked. Copy the generated profile to ~/Library/MobileDevice/Provisioning Profiles/<UUID>.mobileprovision, and delete the key file once you are done.

Then pass the signing details as environment variables. The signing identity is the SHA-1 value shown by security find-identity -v -p codesigning, and the profile was specified by the name given to sigh.

export AGENT_DEVICE_IOS_TEAM_ID=<team id>
export AGENT_DEVICE_IOS_SIGNING_IDENTITY=<certificate SHA-1>
export AGENT_DEVICE_IOS_PROVISIONING_PROFILE="<profile name>"
export AGENT_DEVICE_STATE_DIR=<separate state folder>

AGENT_DEVICE_STATE_DIR is the agent-device daemon's state folder. A separate one was set to keep this work apart from the default daemon used by other work. Bundling these variables and the working directory (cd) into one wrapper script runs every command in the same environment.

Mac and device settings. Besides signing, three things have to be right for the runner to work.

Developer tools security on the Mac must be enabled. If it is off, the runner stops at Developer mode is disabled for Apple development tools. This is a Mac-side setting, separate from Developer Mode on the device; run sudo DevToolsSecurity -enable with admin rights.
iPhone Settings → Developer → "Enable UI Automation" must be on. If it is off, the runner installs and the test starts, but nothing responds.
Keep the screen on and unlocked. When the runner timed out at "Starting XCTest runner", the device was locked. A retry after unlocking never happened, so the cause is only circumstantial, but checking the lock state first with xcrun devicectl device info lockState --device <UDID> is the safer habit.

Handling apps on an iPhone

Listing and deleting installed apps on a real device is done with Xcode's devicectl.

xcrun devicectl list devices
xcrun devicectl device info apps --device <UDID>
xcrun devicectl device uninstall app --device <UDID> <bundle-id>

The runner is removed the same way. Its bundle ID is com.callstack.agentdevice.runner.uitests.xctrunner, and running open after deleting it builds and installs the runner again.

When installing a test build from TestFlight while the App Store version of the same app is already installed, a warning says "You may lose the app's data." The Install button in that warning overwrites the app. It is safer to have the agent stop at this warning and ask a human.

Deleting an app wipes its on-device data. If the app has a backup, take a screenshot showing that backup is on in the app's settings and that the last backup time is just now before deleting; otherwise, stop instead of deleting.

One more screen-handling note: tapping a toggle in a settings screen by ref sometimes hits the middle of the row and does not flip it. In that case, use the element's rect from snapshot --json and tap the right edge of the toggle by coordinates.

Connecting a real Android device

A real Android device connected over USB debugging behaves exactly like an emulator. With wireless debugging, the device may not show up in adb devices right after connecting. It appears only after the _adb-tls-connect service shows up in adb mdns services, so do not conclude it is missing; check again a few seconds later.

A wireless serial is a long mDNS name and easy to mix up with an emulator. Add --serial to every adb and agent-device command. agent-device sessions are created per working directory, so if an emulator was already in use from the same directory, you get an "already bound to android device" error. In that case, open a separate session with --session <name>.

adb mdns services
agent-device open <bundle-id> --platform android --serial <serial> --session <name> --foreground
agent-device close --session <name>
adb disconnect <serial>

Three things commonly go wrong on Android:

Quote adb shell commands as one string. Unless the whole thing is wrapped, as in adb shell "cmd notification post -t 'title' <tag> 'body'", adb splits the arguments on spaces.
Buttons hidden by the keyboard are not found. Close the keyboard with adb shell input keyevent KEYCODE_BACK and snapshot again.
Quote selectors that contain spaces. Use the label="..." form; passing flags as one variable blob gives "Unknown flag".

When something other than the app, such as an interstitial ad, is on top, a snapshot alone makes it hard to tell what state the app is in. Check the system state with adb as well.

adb -s <serial> shell dumpsys activity activities | grep topResumedActivity   # what is in front right now
adb -s <serial> shell pidof <package>                                          # is the app process alive
adb -s <serial> logcat -d | grep -E "FATAL|AndroidRuntime"                     # traces of a crash

Match the snapshot against whether the front screen is the ad or the app, whether the app is still alive behind it, and whether a crash left a record. Ad screens put many webview and link nodes into the accessibility tree, so filter those out when reading a snapshot.

References

agent-device repository: https://github.com/callstack/agent-device
agent-device agent setup docs: https://oss.callstack.com/agent-device/docs/agent-setup
fastlane sigh docs: https://docs.fastlane.tools/actions/sigh/
Android wireless debugging: https://developer.android.com/tools/adb#wireless

Test Automation with agent-device (Part 1)

Krenin — Sat, 24 Oct 2026 18:08:13 +0000

agent-device is an open-source tool from Callstack that lets an AI agent drive iOS and Android apps and check what is on screen. This post covers why it was picked for testing a Flutter app, how to install it for Claude Code, the basic command loop, and what to watch out for with Flutter. It is based on version 0.21.31 (2026-10-24).

Why agent-device

Flutter ships Android and iOS apps from one codebase, so the device-testing tool has to cover iOS too. Google Artemis, the first candidate, is a natural-language testing agent, but it supports Android only; iOS is still just a roadmap item.

The other candidates looked like this:

mobile-mcp exposes taps, screenshots, and the accessibility tree on iOS and Android over MCP (the protocol an AI uses to call external tools). It has no sessions, replay, or evidence storage, so a verified flow is hard to hand off to CI.
Maestro is an E2E framework that pins flows in YAML, and it has an MCP server. It suits regression runs of fixed flows better than an agent exploring the screen.
Arbigent is an AI agent built on top of Maestro. Its author rates it 1/5 for speed and 3/5 for reliability, and its YAML format was still changing.
Commercial services (Drizz, Minitap, and others) bundle a real-device farm with a natural-language agent, but the only public material was vendor material, so cost and quality could not be checked.

agent-device handles iOS and Android simulators, emulators, and real devices with one tool, and its docs state Flutter support explicitly. It ships both a CLI and an MCP server, can save a verified flow as an .ad script to replay later, and can export to Maestro YAML. It is MIT-licensed.

Installation

Install the CLI globally and check the environment.

npm i -g agent-device@latest
agent-device doctor

doctor checks connected devices, toolchains, and dev-server status. In this setup it passed with only warnings for unused platforms (HarmonyOS, Vega).

claude mcp add -s user agent-device -- agent-device mcp

The agent-device repository includes three agent skills: agent-device, ios-simulator, and android-emulator. The official instruction is npx skills add callstack/agent-device, but Claude Code's auto mode safety check blocked it. In that case, clone the repository and copy the three folders under skills/ into ~/.claude/skills/.

To undo everything, run claude mcp remove agent-device -s user and npm uninstall -g agent-device, then delete the three skill folders.

The basic loop

The basic loop is: prepare a device, open the app, act, verify, close. Pick the device with --platform and --device.

agent-device boot --platform android --device "<emulator name>" --headless
agent-device install /absolute/path/app-debug.apk --platform android --device "<emulator name>"
agent-device open <bundle-id> --platform android --device "<emulator name>" --foreground
agent-device press @e5 --settle
agent-device fill @e11 "text" --settle
agent-device wait text "<expected text>"
agent-device screenshot /absolute/path/home.png
agent-device close

open reads the screen and returns a ref such as @e5 for each element, like buttons and text fields. With --settle, press and fill wait until the screen goes quiet and return only what changed, so the agent can pick its next target without rereading the whole screen.

Three things commonly go wrong here:

Refs go stale when the screen changes. Pick a new ref from the diff returned by each action.
A screenshot is not verification. Verify with wait text or wait <selector>, and keep screenshots as evidence only.
Give install an absolute path. A relative path resolves against the agent-device daemon's working directory and fails with "App binary not found".

For a Flutter app, build the iOS simulator app with flutter build ios --simulator --debug and the Android app with flutter build apk --debug.

Flutter gotchas

agent-device reads the screen through the accessibility tree. In Flutter, that means only widgets with text show up by default; icon buttons and containers do not. Add Semantics(identifier:) or semanticLabel to those widgets so a selector can find them. Flutter's Key does not appear in the accessibility tree.

Labels in the accessibility tree follow the language the device is set to, so the same app reads with different label text depending on that setting. A text-based selector changes with the device language, which makes identifier the more stable choice.

On iOS, AdMob test banners show up as platform_view, webview, and link nodes. Selectors in replay scripts should avoid those nodes.

References

agent-device repository: https://github.com/callstack/agent-device
agent-device agent setup docs: https://oss.callstack.com/agent-device/docs/agent-setup
Google Artemis: https://github.com/google/artemis
Maestro Flutter support docs: https://docs.maestro.dev/platform-support/flutter

Building a game demo with Claude Code in one day

Krenin — Sun, 11 Oct 2026 16:57:06 +0000

On a Sunday afternoon I pulled in a game template. That was 1:46 PM. The next time I looked up it was 3 AM, and the last commit message read "trial (demo)". There were 86 commits in between.

The result is Reptile Catcher. You go out to catch lizards by stunning them, sell them, and breed them for prettier patterns. It runs on Godot, and Claude Code wrote the code. My part was setting the direction, playtesting, and picking the assets.

A working loop in five minutes

Five minutes after the template went in, the catch-sell-breed loop was running. Honestly, I expected that much. AI can do that these days.

What mattered was the next minute. The very next commit was a balance simulator where a bot plays the game for you.

Incremental games are a numbers game. Change one upgrade price and a 30-minute stretch can turn into three hours. I can't replay the whole thing from the start every time. So every time Claude changed a number, it ran the bot, and I made calls based on numbers like "4h 19m to the ending (bot)". Later we split the bot into five personalities: one that buys attack first, one that buys the cheapest thing, one that only breeds, and so on.

The bot also caught a bug. While measuring the demo length, one bot never finished. It turned out it was stuck trying to buy an upgrade that was locked in the demo. The bug was on the bot side, but it showed me the lock rules were scattered across the code.

Changing direction at 2:20 PM

The first 30 minutes produced a screen of tabs and buttons. At 2:20 PM I changed direction completely: pixel art, catching as a "run" where you move a character yourself, and upgrades as a tree that branches out like a mind map.

Normally, throwing things out at that point would scare me. But rebuilding was so cheap that I barely hesitated. That's the biggest difference I've felt working with AI.

AI generates the sprites, I pick

The pixel sprites came from PixelLab, connected to Claude through MCP. Claude generated a few candidates, and I picked one with notes like "number 3, but make the neck shorter." Claude recorded the seed of each picked sprite in a doc.

The funny part is why some species got replaced.

The house gecko looked like every other species once it was shrunk to a 32-pixel side view, so I changed it to a skink.
The flying lizard's wing membrane didn't show up at 32 pixels, so I changed it to a frilled lizard.
The bearded dragon's walk cycle looked like a puppy no matter what, so I replaced it with a horned lizard.

Instead of regenerating until I got the sprite I wanted, I changed the game to fit what the tool draws well. For a small game, that was much faster.

My job was removing what the AI built

Around 7 PM I removed a whole set of features Claude had built that afternoon: auto-catching, auto-selling, and offline progress. Ten minutes later, prestige and the fame tree went too.

I had decided this game would be "no failing, no prestige, one straight run that ends in 4 to 6 hours." Idle features and prestige had no place in that. With the AI building this fast, features pile up quickly, but deciding what to cut was still up to me. Looking back, that was most of my work on day one.

Still, a week to ship

By 3 AM the demo scope was cut, but the actual release was October 11. In between, Claude added four bosses, drew the area backgrounds, fixed the demo lock bugs, and prepared the screenshots, cover, and web build for the itch.io page.

So "made in a day" is only true up to the demo. Building took a day, but polishing and shipping took a week. I suspect the next game will go the same way.

If you're building a numbers-heavy game with AI and don't have a simulator yet, I'd suggest having the AI build a bot right after the core loop. For me, that was the best decision of the whole project.

The demo is on itch.io. Come catch some lizards 🦎
👉 https://krenin.itch.io/reptile-catcher