Making Appium Inspector Work with Flutter Apps: Record, Replay, and Run on Real Devices

堀正明 — Thu, 05 Nov 2026 13:26:23 +0000

Appium can drive Flutter apps through the Appium Flutter Driver, which speaks the same protocol as Flutter's own flutter_driver. Appium Inspector, the GUI most people use to explore an app and record a first test, does not work with it. In the Flutter context there is no page source to inspect, you can't tap by widget, and the recorder can only produce coordinate-based steps. Those steps break as soon as the test runs on a phone with a different screen size.

Since early 2025 I have been fixing that with three pieces of open-source code:

appium-handler: a Dart package you add to the app under test. It builds an XML page source from the live widget tree, and it performs actions inside the app.
appium-flutter-driver (a fork): routes the standard W3C commands that Inspector sends to that handler.
appium-inspector (a fork): records taps as Flutter widget finders instead of coordinates, adds a Flutter right-click menu, and generates Dart and JavaScript test code.

Today these tools run a production app's QA suite of about 15 test-spec sheets on an Android phone and an iPhone. This post explains how they work, what it took to make recorded tests replay reliably, and what I learned along the way, including three fixes that were merged into the upstream driver.

TL;DR

With the handler embedded in your app, Appium Inspector shows a Flutter screen's widget tree, and taps are recorded as find.byTooltip(...), find.text(...) and similar finders. Inspector generates Dart (integration_test, Patrol) or JavaScript (WebdriverIO).
Most of the work was not about tapping. It was about locators that looked fine when recorded but pointed at the wrong widget, or at nothing, on the next run.
Text entry had three separate failure modes, and each one reported success. All three are fixed.
Real iPhones work, not just simulators. Three driver fixes were merged upstream (#933, #936, #940).

How the pieces fit together

Appium Inspector (GUI) / generated test scripts
  │  W3C: getPageSource, getWindowRect, performActions, findElement, click
  ▼
appium-flutter-driver (FLUTTER context)
  │  flutter:requestData "<command>:<args>"  (over the Dart VM service)
  ▼
appium-handler (inside the app under test)
     returns the page source as XML, runs tap / enterText / ...,
     and answers with {foundBy, value, ...} for the recorder

The upstream driver already had a channel into the app: flutter_driver's requestData lets a test send a string to a handler registered by the app. The fork uses that channel for the commands Inspector needs:

W3C command	Sent to the handler as
`getWindowRect`	`getScreenSize`
`getPageSource`	`getPageSource` (XML built from the widget tree)
`performActions`	`performActions:<json>`
`findElement`	`findElement:<args>`
`click`	a `tap` action on that element

When you tap a point on the screenshot in Inspector, the handler finds the widget under that point inside the app. It performs the action there and returns a locator for that widget, such as {foundBy: "byTooltip", value: "Increment"}. Inspector records that locator, not the coordinates, so the generated code reads find.byTooltip("Increment").

What you can do in Inspector

Inspect the widget tree. The App Source panel shows the XML built by the handler. In a Debug build, it lists only the widgets your app's code creates, not the hundreds of framework-internal ones. Each node has a class (the widget type), text, key, tooltip, semanticLabel, bounds, and a few Flutter-specific attributes described below. Inspector refreshes it every few seconds while you work.

Record taps as widget finders. Start the recorder and tap on the screenshot. Each step is stored with the locator the handler returned, and the generated code can be switched between Dart and JavaScript from the language menu.

Use the Flutter right-click menu. In a Flutter session, right-clicking the screenshot opens a menu with actions that flutter_driver alone doesn't give you:

Tap: tap the widget and record it with a widget finder.
Enter text: type into a TextField or TextFormField.
Check text: assert what a widget displays.
Check existence: assert that a widget is on screen.
tapDirect: tap a widget that something else is covering (more on this below).

Choose between overlapping widgets. One point on the screen usually belongs to several widgets: a Text inside an InkWell inside a Card. The menu lists every candidate at that point, most specific first. Picking one sends that widget's page-source id along with the action, so the handler acts on exactly that widget. Getting this menu to behave took more than expected. The first version used a stock dropdown component, which closed itself whenever the periodic refresh re-rendered the screenshot. It is now a small position: fixed menu with its own state. Long candidate lists scroll, and the submenus are positioned from the parent item's bounding box so the scrolling container doesn't clip them.

Use "Tap By Element". Upstream Inspector has a mode that taps an element by sending the standard WebDriver click command. The Flutter driver didn't implement click in the Flutter context, so nothing happened. The fork now routes click to the handler's tap, and the handler can resolve an element from its id alone, using the center of its bounds. Element ids that come from appium-flutter-finder are base64-encoded finders. The fork detects those and uses flutter_driver's own tap for them, so apps without the handler keep working.

Small fixes so Inspector treats the session as Flutter. Inspector decides a lot from the session type. Flutter sessions were shown as "Native App Mode" because they have no browserName, and the "Web/Hybrid App Mode" button failed because Inspector tried to tag WebViews. Flutter sessions are now recognized by automationName: flutter, start in the FLUTTER context, and skip WebView lookups.

Inside: a page source you can trust

Inspector, the recorder, and your test scripts all read the same XML. When it is wrong, everything downstream is wrong in confusing ways.

Screens you can't see are not in the tree. Flutter's Navigator keeps every previous route in the element tree, offstage. They used to appear in the XML, often at a negative x where the iOS-style page transition parks them. Two things broke. byText matched text on screens the user couldn't see. A recorded Type#N index also counted widgets that the Flutter finder (which skips offstage widgets) never sees, so it pointed at a different widget on replay. The handler now uses the same debugVisitOnstageChildren walk the finders use, but only for screen-level hiders: Offstage, an unselected IndexedStack child, SliverOffstage, and the Overlay's offstage entries. List items scrolled out of view are kept, because tests need them as scroll targets.

Debug and Profile builds produce very different trees, so locators don't depend on the tree's shape. The filter that hides framework-internal widgets relies on widget-creation tracking, which only exists in Debug (JIT) builds. One screen had about 80 nodes in Debug and about 1,600 for the same route in Profile. They cannot be made identical. Instead, everything a locator depends on is read from the live widget tree:

typeIndex: the widget's index in Flutter finder order, the same order find.byType(T).at(N) uses. It is not the position among same-typed nodes in the XML, which disagreed with the finder for framework-created types like InkWell and Icon.
label and hint on TextField/TextFormField, read from InputDecoration. The rendered label is built inside the framework and never shows up in a Debug page source, and the hint disappears once you type.
In Profile and Release, identity-less pass-through wrappers are collapsed so the tree stays readable.

RichText text is included. The text attribute used to come only from a widget's data property, so anything drawn with Text.rich or an app-built RichText was empty. You couldn't find it with byText, and scripts couldn't read it. The handler now emits the span's plain text. It skips the RichText that Text and Icon create internally, so strings aren't duplicated and icon glyphs don't leak in.

Keys are reported only when they exist. Widgets without a key used to show key="null", the string, which then turned up as a "locator". Empty and "null" keys are now left out.

The root element tells you which handler version you're talking to. <tree> carries offstageFilter, hiddenElements, skippedNodes, unresolvedNodes and collapsedNodes. This sounds minor, but it saved hours. More than once a fix "didn't work" only because the phone still had an old build: with appium:noReset: true, Appium does not reinstall an app that is already present. If <tree> has none of these attributes, you are looking at an old handler.

Inside: locators that survive replay

When the handler resolves a widget, it keeps two jobs apart:

Performing the action uses the exact widget, looked up by its page-source id. That is unambiguous. But the ids are re-minted on every getPageSource call, so they can never be recorded, and because Inspector refreshes often, an id from the last refresh may already belong to another widget. The handler checks the widget it found against the clicked point, so a stale id fails instead of acting on the wrong widget.
Reporting a locator for generated code picks the most stable one available, in this order: tooltip → semantics label → value key → field label → text → tap-target label → Type#N.

Most replay bugs were in step 2.

Only ValueKey<String> is recorded as a key. The key used to be recorded exactly as the widget tree prints it. Apps often attach a GlobalKey to a button, for example to position a coach-mark tooltip. That button was recorded as byValueKey: "[GlobalKey#8cb82]", and the hash changes on every app launch. The recording replayed within the same session and failed on the next one. Even a plain ValueKey('submit') came out as Key("[<'submit'>]") in generated Dart. Now only a ValueKey<String> is recorded, as its plain string. Other keys fall through to the next locator.

Text fields are found by their label (byFieldLabel). A text field without a key used to be recorded either as TextFormField#N, which shifts whenever a field is added above it, or by the text typed into it, which is useless. If the field's label is unique on the screen, it is now recorded as byFieldLabel: "TextFormField|Email". Dart code generation emits find.widgetWithText(TextFormField, "Email").

Taps on icons are labeled by the nearby text. A stock BottomNavigationBar item's label is built by the framework, so it isn't in a Debug page source. Tapping the icon was recorded as Container#N, which at replay pointed at an unrelated 1-pixel divider. The handler now walks up to the nearest InkResponse or GestureDetector and uses its single, screen-unique text as the locator.

The same idea covers a common layout: a child-less InkWell stretched over its content with Stack and Positioned.fill. The InkWell has no text of its own, so the handler looks for text drawn inside the tap target's area. Replaying such a tap uses the text's center. If it went through flutter_driver's tap instead, it would wait forever for the covered text to become hit-testable.

When the exact widget can't be found, fall back step by step. If the id is stale, the handler tries the same locator chain on the device. Plain ByType usually fails with "found N widgets", so taps and scrolls continue to ByTypeIndex and, as a last resort, a coordinate tap. A coordinate tap reports no locator, so the recorder never stores it as if it were stable. Some widgets really have nothing to identify them: a full-screen coach-mark overlay has no tooltip, label, key or text. Those still end up as coordinate taps, and the fix is to choose a more meaningful widget from the candidate menu.

Inside: actions that don't silently do nothing

tapDirect bypasses hit testing. Coach marks, dialog barriers, and other overlays sit on top of the widget you want. A normal tap at that point hits the overlay. tapDirect resolves the target widget and invokes its tap callback directly. Use it sparingly, because it can press a button a real user couldn't reach at that moment. Still, it is the difference between "this screen can be automated" and "it can't".

Text entry had three separate traps, and all three reported success.

flutter_driver ignores the finder for text entry. Its EnterText command types into whatever currently has keyboard focus. Without a prior tap, that is nothing. The handler taps the resolved field first, for every kind of locator.
Text-entry emulation must be registered before that tap. TestTextInput.register() installs a fake text-input channel that learns which field is active by intercepting the TextInput.setClient call made when the field gains focus. Tap first, and that call goes to the real platform channel, no client is captured, and the text is dropped without an error.
Drop the existing focus first. If the field already had focus (you clicked it in Inspector, or an earlier step tapped it), tapping it again does not re-attach its input connection. No setClient call reaches the fake channel, and the field stays empty. The handler unfocuses the current focus and waits a frame before registering and tapping.

After typing, the handler sends the keyboard's Done action, so onSubmitted and onFieldSubmitted run as they would for a real user. The response includes submitted: true, and the Dart code generator emits a matching tester.testTextInput.receiveAction(TextInputAction.done).

Animated screens don't hang. flutter_driver's waitFor can hang forever on a screen with a looping animation (a spinner, a shimmer), in two independent ways. With frame sync on, it waits for transient callbacks to reach zero, which never happens while something keeps ticking. Separately, it waits for the finder itself, which never succeeds if the target never becomes hit-testable. The handler turns frame sync off for that call when an animation is running and gives up after 2 seconds, so the fallback chain can move on. Timing logs showed that building the page source was not the problem (20–134 ms at 800+ nodes). The real signal was a multi-minute gap between two log lines.

Generated code you can keep

The recorder generates:

Dart: integration_test and Patrol variants, so recorded steps can run as Flutter integration tests. They share one finder mapping: byType "InkWell#3" becomes find.byType(InkWell).at(3), and byFieldLabel becomes find.widgetWithText(...).
JavaScript (WebdriverIO): each step becomes a retryFlutterAction(...) call. It re-reads the page source, sends the action, and retries until the device reports that it found the widget:

await retryFlutterAction(
  driver,
  {type: 'tap', foundBy: 'byText', value: 'Sign in'},
  (foundBy) => Boolean(foundBy),
  'Could not resolve the widget to tap (foundBy=byText, value=Sign in)',
);

The JavaScript boilerplate deliberately contains only app-independent helpers: logging, a page-source dump on failure, the retry, and a prompt for manual input. Helpers that only make sense for one app (dismissing its coach marks, finding a button by its fixed position) belong in that app's own test project. A unit test in the Inspector fork keeps them out.

One thing is intentionally not supported. Steps recorded in "Tap By Element" mode show "Code generation for action 'findAndAssign' is not currently supported", because their XPath locators embed the volatile page-source ids. Use the Flutter right-click menu for steps you want to keep.

Real devices, and fixes sent upstream

An emulator and a simulator were enough to build all of this, but a real QA suite runs on real phones, and the iPhone surfaced a series of driver problems. Three were general enough to send upstream, and the maintainers merged them:

#933 appium:dartVmServicePort never reached the app on iOS. The launch flags were added to a copy of the capabilities that XCUITest never saw, so the engine picked a random port and the driver connected to a port nobody was listening on.
#936 mobile: launchApp (the XCUITest way to relaunch an app with arguments) was not usable. In the FLUTTER context it was unsupported. From NATIVE_APP the app relaunched, but the Dart VM socket kept pointing at the dead process.
#940 While mobile: activateApp or mobile: launchApp waited for the app's VM service, the session's newCommandTimeout kept running, so a slow relaunch could end the session in the middle of the command.

Fixes that stayed in the fork:

An iPhone paired over Wi-Fi made every connection fail with "socket hang up". usbmuxd lists such a device twice under the same UDID, once for USB and once for the network. The library call that forwards the VM-service port took the first entry, which could be the network one. The forwarded connection then hit a service bound to the phone's loopback interface and failed with no hint why. The fork prefers the USB entry and logs why a forwarded connection could not be opened.
A clear error when the app wasn't built for testing. If the VM service is reachable but no isolate registers ext.flutter.driver, session creation used to fail with a generic "cannot connect" message. The real cause, usually a build without the flag that enables the extension or an old build still installed, was buried in a debug line. The error now says so directly.
A reconnect race after relaunching the app on iOS. WDA's launch command waits for the app to become idle before it returns. By then the app had already logged its new VM-service URL. The driver cleared the log after the launch, so it waited for a line that had already arrived and failed with "No observatory URL". The fork fixed this by forgetting the old URL before the launch. Upstream later fixed the same race independently (#945, #947), and the fork now uses upstream's version.

That last one taught me something about forks. The connection code in this driver changes upstream every few weeks. Every fork-only change in that area is a merge conflict waiting to happen, and every general fix merged upstream is one fewer. I now merge upstream locally, run the same checks CI runs (npm test && npm run lint && npm run format:check) before pushing, and plan to send the remaining general fixes upstream.

Lessons from running it every day

The suite covers sign-up, payments, account recovery, and cancellation on a Pixel phone and an iPhone SE. Most steps were recorded in Inspector and then edited by hand. A few lessons apply to any Appium-plus-Flutter setup:

Check that your change actually reached the device before debugging it. For the handler, that means rebuilding the app and uninstalling it before the run, because noReset keeps the old build. For the Inspector plugin, it means a hard reload in the browser. For the driver, it means restarting the Appium server, which keeps the driver it loaded at startup. "I fixed it and nothing changed" was, most of the time, one of these.
The protocol between the three repos is just strings. foundBy values, action types, and page-source attribute names are never validated at either end, so a mismatch fails silently. A byToolTip vs byTooltip capitalization bug was found exactly that way. The handler repo's CLAUDE.md keeps one "protocol contract" list, and a rename means searching for the same literal in all three repos.
Don't trust one build type. Debug (Android) and Profile (iOS) page sources differ by an order of magnitude. Check a failure against both before deciding what the tree "looks like".
On iOS, match Xcode to macOS, and keep the iPhone's iOS no newer than Xcode. When the phone updated to an iOS newer than Xcode supported, the WebDriverAgent build failed. An old cached build hid the problem until a dependency upgrade cleaned it out. For iOS 17 and later, keep the RemoteXPC tunnel running, and keep the phone unlocked while it starts.
Watch the browser cache. At least one "the bug is still there" in the Inspector turned out to be the old bundle still loaded in the browser.

Working with Claude Code

Almost all of this was written together with Claude Code, across Dart, TypeScript and React in three repos plus the test suite. What made that work:

Each repo has a CLAUDE.md with the rules: the protocol contract, the fork's customizations, how to verify a change on a device. A separate design-notes file explains why the rules exist. The rules file stays short, and the history isn't lost.
Evidence before fixes. The most expensive mistakes were confident diagnoses made from logs that didn't actually show the cause. Asking for a page-source dump, a device log or a timing log first, and reading it before changing code, was always cheaper.
Small, verified steps. Plan first, change one thing, run it on the device, record the result. A fix isn't done until a real run shows it.

Limitations

The app under test must include appium-handler and be built in Debug or Profile mode. Release builds can't be driven through flutter_driver.
OS-level dialogs on iOS (such as the notification permission prompt) are not always detectable from the Flutter context. The suite falls back to asking the tester to tap them.
"Tap By Element" can't generate reusable code (see above). Merging it with the Flutter right-click menu is on the list.
The driver is a fork. Next, I plan to send the remaining general fixes upstream (the USB-entry preference, the connection retry, the clearer errors) so the fork stays small.

Getting started

You need Node.js, Appium 3, and for iOS, Xcode on the same version as macOS.

npm i -g appium@3.6.0

# Driver (fork)
git clone https://github.com/masaaki-hori/appium-flutter-driver
(cd appium-flutter-driver/driver && npm install && npm run build)
appium driver install --source=local ./appium-flutter-driver/driver

# Inspector (fork), used as an Appium plugin
git clone https://github.com/masaaki-hori/appium-inspector
(cd appium-inspector && npm ci && npm run build:plugin)
appium plugin install --source=local ./appium-inspector/plugins

appium server --use-drivers flutter --use-plugins inspector
# then open http://127.0.0.1:4723/inspector

In the app under test, add flutter_driver (from the SDK) and appium_handler (path: to your clone of appium-handler) to pubspec.yaml, and register the handler. Guard it behind a compile-time flag so it never ships:

import 'package:appium_handler/appium_handler.dart';
import 'package:flutter_driver/driver_extension.dart';

void main() {
  if (const bool.fromEnvironment('ENABLE_APPIUM')) {
    final handler = AppiumHandler();
    enableFlutterDriverExtension(handler: handler.appiumHandler);
    handler.buildDriverExtension();
  }
  runApp(const MyApp());
}

Build with --dart-define=ENABLE_APPIUM=true. Use a Debug APK on Android and a Profile build on iOS. Start a session with these capabilities (Android shown):

{
  "platformName": "Android",
  "appium:automationName": "flutter",
  "appium:deviceName": "<your device>",
  "appium:app": "/path/to/app-debug.apk"
}

After rebuilding the app, uninstall the old one before the next run.

If you try it, I'd like to hear what breaks. Issues and PRs are welcome on any of the three repositories.

Earlier write-ups of this work are on Zenn, in Japanese: making Inspector work with the Flutter driver and the follow-up fixes. The code they link to under `baleen-studio/ is out of date; use the masaaki-hori/` repositories above.

DEV Community: 堀正明