The first version of my GUI agent clicked by pixel coordinates. It worked in demos and failed constantly in practice, and the reason is boring: pixels do not move, but content does.
Here is what I changed, in order of impact.
1. Read the tree before you click
Both desktop and mobile expose an accessibility tree: uiautomator dump on Android, UI Automation on Windows, the accessibility APIs on macOS. Nodes come with labels, roles and bounds. Clicking the node whose text is "Send" survives a re-layout; clicking (984, 183) does not.
The tree is not a luxury. It is the difference between an agent that works on your machine and an agent that only works on the screenshot you tuned it against.
2. When the tree lies, use the keyboard
Some toolkits (I am looking at you, Compose and Flutter) expose one giant node for the entire screen. Every tap resolves to the same element. Coordinates do not save you here either.
What does work: move focus with TAB / arrow keys, confirm the focused field by reading the tree again, then insert text. Typing into "whatever has focus" is fragile; typing into "the field the tree says is focused" is not.
3. Verify after every action, not after every task
Read → act → read. It feels wasteful. It is faster overall, because you stop discovering five steps later that step three landed on the wrong window and everything after it was theatre.
The check is cheap: did the screen change at all, did the expected text appear, did the node count change. One boolean per step.
4. Treat "nothing happened" as an error
An empty diff is ambiguous: either the action was a no-op, or it silently failed, or the UI has not repainted yet. Make the agent wait for a specific predicate ("the button is gone", "the row exists") with a timeout, and fail loudly when it never becomes true. Silent no-ops are how agents burn ten minutes of your day and produce a confident lie.
5. Never trust a cached screen
Screenshots go stale in under a second. Cache the plan, not the pixels.
None of this is model work. It is plumbing, and it is where reliability actually comes from. A mediocre model with tree-based selection, focus verification and predicate waits will beat a great model clicking blind - every time I have tested it.
The desktop and phone halves of this setup are described in the earlier posts in this series. Project page if you want the whole picture: https://agent-home.ai
Which failure mode has cost you the most debugging time: stale screens, ambiguous no-ops, or a tree that reports the whole screen as one node?
Top comments (0)