Test IDs are a nightmare in a hybrid app, and in some containers there is nothing to attach one to. We added AI selectors to our test runner so it could find elements from pixels instead.
Every end-to-end test begins with a negotiation. You want to tap a button, so the button needs a stable identity, so someone opens the app code and adds a test ID. Multiply that by every element in every flow and the real cost becomes clear. It was never writing the tests. It was instrumenting the app so the tests could find anything at all.
In a hybrid app, that bill comes three times over.
The nightmare
Our app ships as one binary assembled from several kinds of container. Some screens are native. Some are cross-platform. Some are embedded web. A single user journey walks through all of them without noticing.
The test runner notices. A test ID has to be added in whichever layer actually draws the element, in that layer's language, with that layer's build and release cycle. One flow crossing three container types meant three separate instrumentation changes before anyone could write its first line.
Some elements have nothing to attach an ID to. A system dialog ships without test IDs and always will. And then the webviews, where it stopped being expensive and became impossible. Inside an embedded web container the native view tree often collapses into a single opaque node. No button, no label, no hierarchy to walk. The runner would face a screen full of content a human reads perfectly well and report that there was nothing to select.
What we did before
Nothing good. We tapped fixed percentages of the screen and hoped no layout would shift. We wrote flows that stopped politely at the boundary of the thing they were meant to test. We left comments marking journeys as blocked, waiting on instrumentation that never arrived.
Whole regions of the app went untested, not because nobody cared but because the vocabulary for describing them did not exist. Every workaround was a promise to come back later, and none were kept.
Until we stopped needing selectors
There is exactly one thing every container in our app has in common. Native, cross-platform, embedded web, and the system's own dialogs all end up as pixels. A screenshot does not care which layer drew it.
So we stopped asking the view tree where things were and started asking an AI to look at the screen.
We maintain a fork of our test runner, which already shipped a couple of AI-backed assertions, so the extension point was there. We added three commands of our own, all built on a vision model that receives the screenshot and answers questions about it.
The first takes a natural-language query and hands back a point you can tap. You describe the element the way you would describe it to a colleague looking over your shoulder, and colour, order, and position are all fair game, because the AI is looking at the same picture you are. No identity on the element, no change to the app, and no dependency on a hierarchy existing.
The second command takes a reference image instead of words, matching a component snapshot against a live screen. The third evaluates a condition in plain language, so a flow can branch on what the AI sees rather than on the presence of a selector.
Making an AI selector survive CI
This is where most AI testing demos quietly end. Asking a model where the button is and tapping its answer looks wonderful once and falls apart across a suite that runs nightly. Screens are dense, elements are small, and a coordinate off by a few percent lands on the neighbouring row.
So the AI gets up to three passes at the same screen, each one narrowing the answer from the last.
Take a screen with three radio buttons stacked vertically. None of them carries a test ID, and the only thing separating the one you want from the other two is that it is red. The test says exactly that:
- extractPointWithAI:
query: the third radio button, the red one
outputVariable: RED_OPTION
- tapOn:
point: ${RED_OPTION}
Two lines, and this is what runs underneath them:
THE SCREEN WHAT THE VIEW TREE SAYS
┌──────────────────────────┐
│ │ RadioButton
│ ◯ Option A │ RadioButton
│ │ RadioButton
│ ◯ Option B │
│ │ three identical nodes.
│ ◯ Option C │ no ids. no colour.
│ ^^^^^^^^ red │ nothing to select on.
│ │
└──────────────────────────┘
│
│ screenshot + serialized view hierarchy
▼
┌────────────────────────────────────────────────────────┐
│ PASS 1 "the third radio button, the red one" │
│ │
│ reasons: three stacked controls, the │
│ lowest one is tinted red │
│ answer: 22%,68% │
└────────────────────────────────────────────────────────┘
│
│ crop 30% of the screen around 22%,68%
▼
┌────────────────────────────────────────────────────────┐
│ PASS 2 ┌────────────────────┐ │
│ │ │ same question, │
│ │ ◯ Option C │ ten times the │
│ │ ^^^^^^^ red │ pixels per element │
│ └────────────────────┘ │
│ │
│ answer: 44%,52% of the crop │
│ mapped: 21%,66% of the full screen │
└────────────────────────────────────────────────────────┘
│
│ the point + the full screenshot
▼
┌────────────────────────────────────────────────────────┐
│ PASS 3 "is 21%,66% the element I asked for?" │
│ │
│ answer: yes (or a corrected point) │
└────────────────────────────────────────────────────────┘
│
▼
tapOn: point: 21%,66%
any pass fails → keep the previous pass's answer
Pass one gets the view hierarchy alongside the screenshot, which matters more than it sounds. Even when the tree cannot produce a unique selector, as with three identical radio nodes, it usually still carries the real strings and the real bounds, so the model reads labels instead of transcribing them off an image. Colour and fill it cannot tell you at all. That part only exists in the pixels. When the tree is opaque, the screenshot alone still works, just with less help.
Pass two exists because a full screen gives the model very few pixels per element. Cropping to a region around the first answer asks the identical question with an order of magnitude more detail, then maps the refined answer back to full-screen coordinates.
Pass three shows the model the point it picked and asks whether it got it right. An AI grading its own answer sounds circular. It catches real mistakes.
Every pass fails soft, falling back to the previous answer rather than failing the test, and each appends to a reasoning log attached to the command. When something breaks at three in the morning, the report says what the AI thought it was looking at, not merely that a tap missed. Passes are configurable and the default is one, so precision is there when a flow needs it and nobody pays for extra AI calls otherwise.
What it unlocked
The radio buttons are the easy case to picture, and there are harder ones where a test ID would not have helped even if someone had added it.
The handle of a price range slider is a position on a track. No identity expresses "drag from here to about a third of the way along", because what the test needs is a coordinate, not a name. A seat map renders differently on every run, so no fixed identity will ever point at an available seat, and the flow has to branch on what is actually on screen.
Those were never naming problems. There was nothing to name. Plenty of elements sit in a perfectly readable hierarchy and still have no useful identity, because the difficulty is not naming. It is meaning.
The discipline that keeps AI honest
An AI selector is the last resort, not the default. Our guidance ranks AI below text and below test IDs, and that ordering does real work.
The clearest evidence is that the migration runs both ways. When product copy churns and an exact-string assertion breaks for the third time, a team rewrites it as a description of what the screen should show, and it stops breaking on wording. But when a component finally gets a stable identity, teams go back and replace the AI call with the plain selector.
That is the right instinct. A plain selector is fast, free, and deterministic. An AI call is none of those things. It is for the cases where no selector can exist, or where creating one costs more than the test is worth. Reaching for AI first trades a maintenance problem for a slower, more expensive one.
The same discipline shows up as something we built that nobody uses. The command that matches a reference image works fine and has never appeared in a single test. When a sentence does the job, nobody goes hunting for a screenshot.
The AI command we are building now
It asserts that everything on screen is written in the expected language, with an ignore list for what is meant to stay untranslated, like brand and station names. The AI reports back every string that looks wrong, the language it thinks that string is in, and why.
We ship in a lot of languages, and localization gaps are the classic bug no assertion catches, because there is nothing to select. A missing translation is not a missing element. It is a present element with the wrong words inside it, and the only way to find one used to be a human who reads that language looking at the screen.
That is the real shape of the win. Not that AI made the tests smarter, but that it let them ask questions our old vocabulary could not express.
Takeaway
The hidden cost of end-to-end testing is instrumentation, and hybrid apps pay it several times over. In the containers where no selector exists, that cost is not high. It is infinite, and the tests simply do not get written.
Put an AI behind the selector and the container type stops mattering, because pixels are the one interface every layer shares. Keep it as the fallback, since the best test ID is still a real test ID. The point is that you no longer need one to write the test.
Top comments (0)