Your computer-use agent clicks 20 pixels off: screenshots vs the accessibility tree

Every computer-use agent has to answer one question before it can click anything: what is on the screen right now?

There are two ways to answer it. Send the model a screenshot and let it find the button in the pixels. Or ask the operating system, which already knows: every serious UI toolkit publishes an accessibility tree for screen readers, with the role, name, state and position of each control.

Most demos use screenshots because they work everywhere. In practice you want both, in a specific order. Here is how each one behaves, how to look at the tree yourself, and where each breaks.

What the accessibility tree gives you

A screenshot is a grid of pixels. The tree is a list of facts:

button  "Export"        enabled   @ (812, 64)
textbox "File name"     focused   value "report-q3"
menuitem "CSV"          @ (830, 140)

That changes three things for an agent:

• It clicks the element, not a guessed coordinate. The agent says "press the Export button" and the OS hands back exactly where it is. No off-by-20-pixels misses on a crowded toolbar.
• It knows state. Disabled, checked, focused, expanded, the current value of a field. From a screenshot the model has to infer a greyed-out button from a slightly lighter shade.
• It is cheap to send. A screen with 40 controls is a few dozen short lines of text. A screenshot costs roughly width x height / 750 tokens on Claude (per Anthropic's vision docs), so about 1,600 tokens for a 1092x1092 image, every single step. Text is also something the model can search and quote precisely.

Where the tree is empty or lies

The tree is only as good as the app's accessibility work. The usual gaps:

• Canvas apps. Design tools' canvases, games, maps, charting widgets. The window is one big element with nothing inside.
• Remote screens. A VNC, RDP or Citrix window is just a video. The tree stops at the window frame.
• Electron and Chromium, until asked. Chromium builds its full web accessibility tree only when it detects an assistive technology. On macOS, Electron apps expose it once a client sets AXManualAccessibility; Chrome itself has --force-renderer-accessibility. Before that, you see a shell with almost no children.
• Custom-drawn controls with no names. You get button "" twelve times in a row.
• Stale positions during animations and scrolling. Read the tree, then act fast, or read it again.

That is why screenshots stay in the toolbox: they work on anything that draws pixels.

The screenshot trap: coordinates and scaling

If you do click from screenshots, the most common silent failure is scaling, not the model's eyesight.

On a Retina Mac the screenshot is in physical pixels and clicks are in logical points (half). On Windows, display scaling of 125% or 150% does the same thing. And many agents downscale the screenshot before sending it to the model, so the model's coordinates are in a third coordinate space.

Symptoms: clicks land up-left or down-right of the target, consistently, by a fixed ratio. Fix: keep one conversion function from "image the model saw" to "screen points" and run every click through it. Test it by clicking the four corners of a known window.

Look at the tree yourself

Before you decide which approach your agent needs for an app, spend two minutes with the platform inspector:

• macOS: Accessibility Inspector (Xcode, Open Developer Tool). Hover anything to see role, title and frame.
• Windows: Accessibility Insights for Windows, or Inspect.exe from the Windows SDK. Both read UI Automation.
• Linux: Accerciser, which reads AT-SPI. Most of this assumes X11; Wayland deliberately limits what other programs can see and inject.
• Android: adb shell uiautomator dump /sdcard/view.xml && adb pull /sdcard/view.xml. You get the view hierarchy with text, resource ids and bounds.

If the inspector shows named buttons and fields, a tree-first agent will be fast and accurate there. If it shows one big pane, plan for screenshots and OCR.

The order that works

1. Tree first. List the interactive elements with names, roles and positions. Let the model pick one by id.
2. Text next. When the tree has no labels, OCR the window and match on visible text.
3. Screenshot last. For canvases, icons without names, or "is this the right page?" checks, send the image.
4. Verify after acting. Re-read the tree or the text after each important step. Do not assume the click worked.

Keeping the screenshot as the fallback instead of the default makes most steps cheaper and more precise, and you still cover the apps that draw everything themselves.

Why we care

We build Auten, an MCP server that gives Claude Code, Codex, Cursor or any MCP client hands on a real screen: your computer (Mac, Windows, Linux) and your Android phone over USB (beta). It works in exactly this order: the agent gets a numbered list of real buttons and fields from the accessibility tree, and OCR and screenshots are there when an app has no accessibility info. If you want to try it: auten.ai.

Which apps have the worst accessibility trees you have had to automate? Tell us in the comments.