I run an AI agent in a browser all day. Every "look at this page" is a context-window decision, and most agent stacks make the expensive choice by default: they screenshot the page and let a vision model figure it out.
There is a cheaper way that also clicks better. The browser already maintains a description of the page for screen readers: the accessibility tree. Hand that to the agent instead. This post is the measured math, the failure modes, and when you genuinely still need pixels.
What the agent actually sees
An accessibility-tree snapshot of a page reads like a text outline of the interface. Hacker News looks like this:
table "Hacker News new | past | comments | ask | show | jobs | submit" @e1
link "Hacker News" @e5
link "new" @e6
link "submit" @e12
link "login" @e13
Every line is an element the agent can act on. The @e1, @e5 refs are the important part: the agent does not compute "the submit button is at x=830, y=210" from pixels. It reads link "submit" @e12 and clicks @e12. Coordinates die when the page re-renders or the window resizes. A ref to the named element survives.
The measured math
Same page, two formats, measured during a live session:
- accessibility tree of the full page: 2.4 KB of text
- 1000px-wide JPEG of the same page: 16 KB
Call it 6 to 10x, before you count the second cost: a screenshot is not just bigger, it needs a vision-capable model tier to interpret. Text goes into any model. And the tree has no guessing: an agent that clicks pixel coordinates sometimes clicks confidently wrong. Refs are the difference between "click at (831, 205)" and "click the element the page itself calls submit".
A browsing task is not one look. It is snap, decide, act, re-snap, twenty times. Twenty steps at a 6x difference is not a rounding error, it is the difference between a task that fits your context window and one that does not.
What the tree does not see
Honesty time, because this is where the pitch usually gets oversold.
Div soup, gone. The accessibility tree contains semantic elements. I measured a LinkedIn feed page: 2911 DOM elements, 784 of them divs. The tree had zero divs and 121 buttons, every single one with an accessible name. The tree is not "the DOM but smaller", it is the page minus everything that was never information in the first place.
Canvas is invisible. Excalidraw, Figma, anything that draws: the tree sees a rectangle. If your target app is a canvas, you need pixels, full stop.
Some state only lives in pixels. In my Excalidraw test the agent double-clicked (the page requires a trusted event, more on that below) and created a text element. The new element appeared in the tree. The text it contained only appeared after commit, and reading it back required a screenshot. Tree for structure, pixels for verification: that is the actual division of labor.
Synthetic events get ignored. Some apps check isTrusted and quietly drop synthetic clicks. The honest answer is a mode that drives the same events DevTools sends, isTrusted=true, and the equally honest caveat: those events and the DevTools protocol are detectable by the page. There is no stealth mode in what I built, on purpose.
Verdicts beat silence
The underrated part of tree-first agents is failure reporting. Because the tree is cheap, you can snapshot before and after every action and answer the only question that matters: did that click actually do anything?
The scheme I use returns a verdict per action: succeeded, needs_human (a login wall appeared, call the human), blocked (rate limit, back off), or uncertain (the event fired, nothing observably changed, check before retrying). In the Excalidraw case the single click came back uncertain, a screenshot confirmed nothing had happened, and a trusted double-click got through. Compare that to a screenshot agent that clicks, screenshots again, and hopes.
Try it
Two paths, pick either:
Playwright MCP has an accessibility-snapshot mode; if you are already in that ecosystem, turn it on and stop screenshotting by default.
chrome-bridge, my tool, is built tree-first. A tiny Chrome extension plus a zero-dependency Node CLI, and the agent drives the Chrome you are already logged into (the sessions and 2FA you already have, instead of a fresh profile that hits a login wall on step one). Setup is one paste: install the extension, click its toolbar button, copy the setup prompt, hand it to your agent, done. It works with any agent that can run a shell command: Claude Code, Codex, Cursor, GLM, Kimi, local models. No MCP server, no account, everything on 127.0.0.1.
node cli.mjs snap example.com # the tree, ~2.4KB
node cli.mjs click example.com @e14 # click by ref
node cli.mjs shot example.com out.png # when you truly need pixels
GitHub: https://github.com/siropkin/chrome-bridge
Chrome Web Store: https://chromewebstore.google.com/detail/chrome-bridge/kmhjlnokjigmnimgjjmiahlinjbcebkg
The rule of thumb
Snap first, shot last. The tree answers "what is on this page and what can I click" for a tenth of the price, and it cannot misclick a coordinate it never guessed. Pixels are for canvas, for visual verification, and for the moments the tree says uncertain. Treat them that way and the token bill takes care of itself.
Top comments (1)
@siropkin, “tree for structure, pixels for verification” is an excellent operating rule. I especially like the per-action verdicts (
succeeded,needs_human,blocked,uncertain) because they keep an agent from treating lack of visible change as permission to retry blindly. I’m exploring a similar before/after evidence boundary in agent-inspect; do you retain the accessibility snapshots as a compact trajectory so failures can be replayed or regression-tested later?