If you have ever automated more than one Android phone, you have met this bug: the flow taps (540, 1620), which was the Sign in button on your test phone — and on the next phone it is the space between two buttons, because that phone has a different screen, a different font size, or a banner the first one did not show.
Coordinates describe where something happened to be. What you actually want to say is what to tap. On Android that is answerable, because every screen is a tree of elements.
The screen is a tree
Android can describe what is on screen as a hierarchy of views. Each node has a class (android.widget.Button) and attributes:
-
resource-id— the developer's id, e.g.com.example.shop:id/sign_in -
text— what the element says -
content-desc— its accessibility label (icon buttons usually have one) -
class,package,bounds,index, and flags likeclickableandenabled
That is the same model Appium Inspector shows you, and it means a selector language you probably already know works on it: XPath.
//*[@resource-id='com.example.shop:id/sign_in'] (the sturdiest, when the app has ids)
//android.widget.Button[@text='Sign in'] (a button by its exact text)
//*[@content-desc='Search'] (an icon button by its label)
//*[contains(@text,'Order #')] (any element containing text)
(//android.widget.EditText)[2] (the second text field)
The element is found on the screen as it is now, wherever it sits, and the tap lands inside its bounds. A banner pushing the button down, or a phone with another resolution, no longer matters.
Not all selectors age the same way
A selector that finds the element today can stop finding it after the next app update. A rough order, from sturdiest to most fragile:
- resource-id — survives most redesigns.
- content-desc — stable for icon buttons, and translated in other languages.
- text — fine until the copy changes or the phone runs another language.
- a child's or neighbour's text — "the row that contains Order #1042".
-
position (
[2]) — breaks as soon as anything is added above it.
In WarpX Phone the Inspector does this ranking for you: you click an element on the phone's screen, and it lists the selectors that name only that element, scored by how likely each is to keep working, and stores them together. At run time it tries the best one first and falls back to the next if the app has changed. If no selector names the element alone, it tells you so rather than handing you one that will match the wrong row.
When the element is not on screen yet
Two cases cover most "element not found" failures:
- It is further down a list. Scroll the screen's main list and look again, a bounded number of times, before giving up.
- It has not loaded yet. Wait for it to show, or for a spinner to go, woken by the screen changing rather than by a fixed sleep.
Both are better than a sleep(3000) followed by a tap, which is either too slow or too short — usually both, on different phones.
When there are no elements at all
Games, maps, canvas-based apps and some web views draw everything as a single picture. There is no tree to query. Two fallbacks still beat coordinates:
- Find it by its picture. Crop the button once; at run time, look for that picture on the screen and tap where it is found.
- Read the text. OCR an area (a coin balance, a code in an image), or find a word anywhere on the screen and tap it.
Coordinates stay useful for one thing — gestures that are about a place, like a swipe across a feed — and even then it helps to store them as shares of the screen (0.5, 0.8) rather than pixels, so they land in the same spot on another resolution.
A note on mirroring taps
Many phone-farm tools offer "do it on one phone, the others follow". It is great for a quick manual job. But it copies what you did, not what the apps show: if one phone has a pop-up the others do not, the same touch lands on different things. For anything longer than a few steps, a flow that finds elements is the sturdier choice.
Try it
WarpX Phone drives real Android phones over USB or Wi-Fi from a Windows PC, with no root: live screens, the Inspector and Verify for selectors, and drag-and-drop workflows built from blocks like Element action, Wait for element, Image search and Read text (OCR).
- How selectors, Inspect and Verify work: Finding elements on a phone
- Every phone block, with a picture of what it does: the block reference
- The app: warpx.app/phone — free for one phone
What does your flow do when an element is missing — retry, scroll, or fall back to a picture? I'd like to hear how others handle it.
Top comments (0)