DEV Community: Mohammad Haidar

Our Test Runner Couldn't See Inside a WebView. Then We Gave It AI.

Mohammad Haidar — Thu, 10 Sep 2026 12:52:21 +0000

Test IDs are a nightmare in a hybrid app, and in some containers there is nothing to attach one to. We added AI selectors to our test runner so it could find elements from pixels instead.

Every end-to-end test begins with a negotiation. You want to tap a button, so the button needs a stable identity, so someone opens the app code and adds a test ID. Multiply that by every element in every flow and the real cost becomes clear. It was never writing the tests. It was instrumenting the app so the tests could find anything at all.

In a hybrid app, that bill comes three times over.

The nightmare

Our app ships as one binary assembled from several kinds of container. Some screens are native. Some are cross-platform. Some are embedded web. A single user journey walks through all of them without noticing.

The test runner notices. A test ID has to be added in whichever layer actually draws the element, in that layer's language, with that layer's build and release cycle. One flow crossing three container types meant three separate instrumentation changes before anyone could write its first line.

Some elements have nothing to attach an ID to. A system dialog ships without test IDs and always will. And then the webviews, where it stopped being expensive and became impossible. Inside an embedded web container the native view tree often collapses into a single opaque node. No button, no label, no hierarchy to walk. The runner would face a screen full of content a human reads perfectly well and report that there was nothing to select.

What we did before

Nothing good. We tapped fixed percentages of the screen and hoped no layout would shift. We wrote flows that stopped politely at the boundary of the thing they were meant to test. We left comments marking journeys as blocked, waiting on instrumentation that never arrived.

Whole regions of the app went untested, not because nobody cared but because the vocabulary for describing them did not exist. Every workaround was a promise to come back later, and none were kept.

Until we stopped needing selectors

There is exactly one thing every container in our app has in common. Native, cross-platform, embedded web, and the system's own dialogs all end up as pixels. A screenshot does not care which layer drew it.

So we stopped asking the view tree where things were and started asking an AI to look at the screen.

We maintain a fork of our test runner, which already shipped a couple of AI-backed assertions, so the extension point was there. We added three commands of our own, all built on a vision model that receives the screenshot and answers questions about it.

The first takes a natural-language query and hands back a point you can tap. You describe the element the way you would describe it to a colleague looking over your shoulder, and colour, order, and position are all fair game, because the AI is looking at the same picture you are. No identity on the element, no change to the app, and no dependency on a hierarchy existing.

The second command takes a reference image instead of words, matching a component snapshot against a live screen. The third evaluates a condition in plain language, so a flow can branch on what the AI sees rather than on the presence of a selector.

Making an AI selector survive CI

This is where most AI testing demos quietly end. Asking a model where the button is and tapping its answer looks wonderful once and falls apart across a suite that runs nightly. Screens are dense, elements are small, and a coordinate off by a few percent lands on the neighbouring row.

So the AI gets up to three passes at the same screen, each one narrowing the answer from the last.

Take a screen with three radio buttons stacked vertically. None of them carries a test ID, and the only thing separating the one you want from the other two is that it is red. The test says exactly that:

- extractPointWithAI:
    query: the third radio button, the red one
    outputVariable: RED_OPTION
- tapOn:
    point: ${RED_OPTION}

Two lines, and this is what runs underneath them:

          THE SCREEN                      WHAT THE VIEW TREE SAYS
   ┌──────────────────────────┐
   │                          │             RadioButton
   │    ◯    Option A         │             RadioButton
   │                          │             RadioButton
   │    ◯    Option B         │
   │                          │             three identical nodes.
   │    ◯    Option C         │             no ids. no colour.
   │         ^^^^^^^^ red     │             nothing to select on.
   │                          │
   └──────────────────────────┘
                 │
                 │   screenshot  +  serialized view hierarchy
                 ▼
   ┌────────────────────────────────────────────────────────┐
   │  PASS 1    "the third radio button, the red one"       │
   │                                                        │
   │            reasons: three stacked controls, the        │
   │            lowest one is tinted red                    │
   │            answer:  22%,68%                            │
   └────────────────────────────────────────────────────────┘
                 │
                 │   crop 30% of the screen around 22%,68%
                 ▼
   ┌────────────────────────────────────────────────────────┐
   │  PASS 2    ┌────────────────────┐                      │
   │            │                    │   same question,     │
   │            │   ◯   Option C     │   ten times the      │
   │            │       ^^^^^^^ red  │   pixels per element │
   │            └────────────────────┘                      │
   │                                                        │
   │            answer:  44%,52%  of the crop               │
   │            mapped:  21%,66%  of the full screen        │
   └────────────────────────────────────────────────────────┘
                 │
                 │   the point  +  the full screenshot
                 ▼
   ┌────────────────────────────────────────────────────────┐
   │  PASS 3    "is 21%,66% the element I asked for?"       │
   │                                                        │
   │            answer:  yes  (or a corrected point)        │
   └────────────────────────────────────────────────────────┘
                 │
                 ▼
          tapOn:  point: 21%,66%

   any pass fails  →  keep the previous pass's answer

Pass one gets the view hierarchy alongside the screenshot, which matters more than it sounds. Even when the tree cannot produce a unique selector, as with three identical radio nodes, it usually still carries the real strings and the real bounds, so the model reads labels instead of transcribing them off an image. Colour and fill it cannot tell you at all. That part only exists in the pixels. When the tree is opaque, the screenshot alone still works, just with less help.

Pass two exists because a full screen gives the model very few pixels per element. Cropping to a region around the first answer asks the identical question with an order of magnitude more detail, then maps the refined answer back to full-screen coordinates.

Pass three shows the model the point it picked and asks whether it got it right. An AI grading its own answer sounds circular. It catches real mistakes.

Every pass fails soft, falling back to the previous answer rather than failing the test, and each appends to a reasoning log attached to the command. When something breaks at three in the morning, the report says what the AI thought it was looking at, not merely that a tap missed. Passes are configurable and the default is one, so precision is there when a flow needs it and nobody pays for extra AI calls otherwise.

What it unlocked

The radio buttons are the easy case to picture, and there are harder ones where a test ID would not have helped even if someone had added it.

The handle of a price range slider is a position on a track. No identity expresses "drag from here to about a third of the way along", because what the test needs is a coordinate, not a name. A seat map renders differently on every run, so no fixed identity will ever point at an available seat, and the flow has to branch on what is actually on screen.

Those were never naming problems. There was nothing to name. Plenty of elements sit in a perfectly readable hierarchy and still have no useful identity, because the difficulty is not naming. It is meaning.

The discipline that keeps AI honest

An AI selector is the last resort, not the default. Our guidance ranks AI below text and below test IDs, and that ordering does real work.

The clearest evidence is that the migration runs both ways. When product copy churns and an exact-string assertion breaks for the third time, a team rewrites it as a description of what the screen should show, and it stops breaking on wording. But when a component finally gets a stable identity, teams go back and replace the AI call with the plain selector.

That is the right instinct. A plain selector is fast, free, and deterministic. An AI call is none of those things. It is for the cases where no selector can exist, or where creating one costs more than the test is worth. Reaching for AI first trades a maintenance problem for a slower, more expensive one.

The same discipline shows up as something we built that nobody uses. The command that matches a reference image works fine and has never appeared in a single test. When a sentence does the job, nobody goes hunting for a screenshot.

The AI command we are building now

It asserts that everything on screen is written in the expected language, with an ignore list for what is meant to stay untranslated, like brand and station names. The AI reports back every string that looks wrong, the language it thinks that string is in, and why.

We ship in a lot of languages, and localization gaps are the classic bug no assertion catches, because there is nothing to select. A missing translation is not a missing element. It is a present element with the wrong words inside it, and the only way to find one used to be a human who reads that language looking at the screen.

That is the real shape of the win. Not that AI made the tests smarter, but that it let them ask questions our old vocabulary could not express.

Takeaway

The hidden cost of end-to-end testing is instrumentation, and hybrid apps pay it several times over. In the containers where no selector exists, that cost is not high. It is infinite, and the tests simply do not get written.

Put an AI behind the selector and the container type stops mattering, because pixels are the one interface every layer shares. Keep it as the fallback, since the best test ID is still a real test ID. The point is that you no longer need one to write the test.

My Most Useful Repo Is Empty, and My AI Agent Lives in It

Mohammad Haidar — Mon, 07 Sep 2026 15:36:34 +0000

How an empty host repo binds two codebases for an AI coding agent: instructions, skills, memory, and verification that ends on a device.

A native module moved from the Android shell into the React Native repo, and one method got a new name on the way. The JS kept calling the old one. The build passed, the app started, and a whole screen quietly stopped working. Nothing in either repo could have flagged it, because the name lived in both and neither side knew about the other.

That is the failure I built this setup to catch. It starts from a simpler observation.

AI-assisted coding is great inside one repo. The agent reads the repo's instructions, sees the whole tree, and its mistakes stay local.

But what if your app ships from two repos, and every real change has to land in both, in the right order? Open the agent one level up and it sees every project on your machine, with nothing telling it how these two relate.

My answer is a third repo. Empty of application code. It holds the binding between the other two, the instructions that only make sense across them, the skills, and the memory. The agent runs from there. I use Claude Code. The pattern ports to any agent that reads instruction files.

The setup, and the gap

One Android APK, built from a React Native monorepo and a native Kotlin shell. The React Native side publishes its bundle and its bridge modules' native code as an npm package. The Android side pins that package by exact version and points Gradle into node_modules. No over-the-air updates. The JS a user runs is whatever version the pin says.

Between them sits a hand-maintained bridge. Module names, method names, event names, and storage keys exist on both sides and must match exactly. No codegen checks them. When one side drifts, nothing fails at build time. The call resolves to undefined at runtime. That is the story at the top.

Each repo's docs cover its own half. Nobody owns the gap.

The host repo

The whole thing fits in one screen:

host/
├── CLAUDE.md            the contract between the two repos
├── notes/               cross-session state, written for the agent
└── .claude/
    ├── settings.json    points at the two repos
    ├── skills/          bridge wiring, local builds
    ├── agents/          the review agent
    └── agent-memory/    what that agent has already learned

The settings file at .claude/settings.json is the whole trick:

{ "additionalDirectories": ["../rn-app", "../android-shell"] }

A session started here sees exactly those two trees and nothing else. The host's CLAUDE.md imports each repo's own CLAUDE.md, so repo-internal rules stack underneath. On top, the host adds what only makes sense across the boundary: how the repos bind, the four identifier kinds that must match, where native modules live during an ongoing migration, areas owned by other teams, and a list titled "things this file does not yet cover" so the agent says "not specified" instead of inventing.

Plus the one rule that exists only because there are two repos: plan before editing, because order matters.

Skills that compose

Procedures live in skills, markdown files the agent pulls in when a task matches their description. The interesting part is how they chain.

The bridge-wiring skill knows the two paths for a native module: the short one entirely inside the React Native repo, and the legacy one that touches the shell. It defaults to the short path and flags the legacy one. It also carries the migration checklist, with one rule: never rename a module while moving it.

The local-build skill knows how to run a debug build against Metro, and how to build a release-style APK against an unpublished bundle by pointing the shell's package.json at a local path instead of a published version.

A review agent, with its own memory of past pitfalls, reads a pull request through one team's ownership boundaries and comments only on changed lines that team owns.

On their own, each is a checklist. Together they close a loop the repos cannot close by themselves.

Verification that reaches the device

Ask for a new native module. The wiring skill writes the TS wrapper and the Kotlin class and registers it. Then, instead of stopping at "done", the local-build skill takes over: build the bundle locally, link it into the shell, assemble the APK, install it, and exercise the feature.

That last step is where the silent undefined would otherwise surface in QA days later. Now it surfaces in the same session, on the actual APK a user would get. The agent reports what it saw, not what it expects.

Memory

Because the agent runs from the host, its memory is scoped to cross-repo work rather than to either repo. Notes on long-running efforts that span both sides sit next to the instructions, each with a status and a last-updated date. The review agent keeps its own store of pitfalls it has already found. The next session starts from what was verified, not from zero.

What it does not fix

The bridge is still hand-maintained. The loop catches mismatches. It does not prevent them. Codegen would, and that is a bigger change than a markdown file.

The rules only work if they are read. When the agent slips back into one-repo thinking, the fix is usually a sharper sentence, not a longer file.

Takeaway

If one artifact is built from more than one repo, the contract between them lives nowhere. Give it a home your tools read, scope the agent to exactly the repos involved, and let the skills compose into a loop that ends on a device.

The empty repo is just that, with a working directory attached.