Accessibility tree vs screenshots: the token math behind my browser agent

Ivan Seredkin — Sat, 12 Sep 2026 21:02:37 +0000

I run an AI agent in a browser all day. Every "look at this page" is a context-window decision, and most agent stacks make the expensive choice by default: they screenshot the page and let a vision model figure it out.

There is a cheaper way that also clicks better. The browser already maintains a description of the page for screen readers: the accessibility tree. Hand that to the agent instead. This post is the measured math, the failure modes, and when you genuinely still need pixels.

What the agent actually sees

An accessibility-tree snapshot of a page reads like a text outline of the interface. Hacker News looks like this:

table "Hacker News new | past | comments | ask | show | jobs | submit" @e1
  link "Hacker News" @e5
  link "new" @e6
  link "submit" @e12
  link "login" @e13

Every line is an element the agent can act on. The @e1, @e5 refs are the important part: the agent does not compute "the submit button is at x=830, y=210" from pixels. It reads link "submit" @e12 and clicks @e12. Coordinates die when the page re-renders or the window resizes. A ref to the named element survives.

The measured math

Same page, two formats, measured during a live session:

accessibility tree of the full page: 2.4 KB of text
1000px-wide JPEG of the same page: 16 KB

A correction, forced by a reader who did the math properly: 6-10x was a bytes-to-bytes comparison, and bytes are not the unit on the image side. Image token cost scales with pixels, not file size: this 1000x713 frame is roughly 950 tokens whether the JPEG weighs 16 KB or 185 KB. Tokenize the 2.4 KB tree and it lands near 840 tokens. On a light page, token count is close to parity. The savings are real, but they live somewhere else. Text runs on any model tier, a screenshot needs a vision tier: that is the price-per-token win. Refs survive re-renders, pixel coordinates do not, and an agent that clicks coordinates sometimes clicks confidently wrong: that is the correctness win. And the tree is queryable: a scoped snap of one story row on this page measured 147 bytes, on the order of 50 tokens at the same bytes-per-token ratio. Cost per decision is a slice of the tree; a screenshot is all or nothing.

A browsing task is not one look. It is snap, decide, act, re-snap, twenty times. Twenty full-page screenshots bill a vision tier at ~950 tokens a look. Twenty scoped tree reads bill a text tier at a slice each. The gap that matters shows up on the bill, not in the token count.

What the tree does not see

Honesty time, because this is where the pitch usually gets oversold.

Div soup, gone. The accessibility tree contains semantic elements. I measured a LinkedIn feed page: 2911 DOM elements, 784 of them divs. The tree had zero divs and 121 buttons, every single one with an accessible name. The tree is not "the DOM but smaller", it is the page minus everything that was never information in the first place.

Canvas is invisible. Excalidraw, Figma, anything that draws: the tree sees a rectangle. If your target app is a canvas, you need pixels, full stop.

Some state only lives in pixels. In my Excalidraw test the agent double-clicked (the page requires a trusted event, more on that below) and created a text element. The new element appeared in the tree. The text it contained only appeared after commit, and reading it back required a screenshot. Tree for structure, pixels for verification: that is the actual division of labor.

Synthetic events get ignored. Some apps check isTrusted and quietly drop synthetic clicks. The honest answer is a mode that drives the same events DevTools sends, isTrusted=true, and the equally honest caveat: those events and the DevTools protocol are detectable by the page. There is no stealth mode in what I built, on purpose.

Verdicts beat silence

The underrated part of tree-first agents is failure reporting. Because the tree is cheap, you can snapshot before and after every action and answer the only question that matters: did that click actually do anything?

The scheme I use returns a verdict per action: succeeded, needs_human (a login wall appeared, call the human), blocked (rate limit, back off), or uncertain (the event fired, nothing observably changed, check before retrying). In the Excalidraw case the single click came back uncertain, a screenshot confirmed nothing had happened, and a trusted double-click got through. Compare that to a screenshot agent that clicks, screenshots again, and hopes.

Try it

Two paths, pick either:

Playwright MCP has an accessibility-snapshot mode; if you are already in that ecosystem, turn it on and stop screenshotting by default.

chrome-bridge, my tool, is built tree-first. A tiny Chrome extension plus a zero-dependency Node CLI, and the agent drives the Chrome you are already logged into (the sessions and 2FA you already have, instead of a fresh profile that hits a login wall on step one). Setup is one paste: install the extension, click its toolbar button, copy the setup prompt, hand it to your agent, done. It works with any agent that can run a shell command: Claude Code, Codex, Cursor, GLM, Kimi, local models. No MCP server, no account, everything on 127.0.0.1.

node cli.mjs snap example.com           # the tree, ~2.4KB
node cli.mjs click example.com @e14     # click by ref
node cli.mjs shot example.com out.png   # when you truly need pixels

GitHub: https://github.com/siropkin/chrome-bridge
Chrome Web Store: https://chromewebstore.google.com/detail/chrome-bridge/kmhjlnokjigmnimgjjmiahlinjbcebkg

The rule of thumb

Snap first, shot last. The tree answers "what is on this page and what can I click" for a tenth of the price, and it cannot misclick a coordinate it never guessed. Pixels are for canvas, for visual verification, and for the moments the tree says uncertain. Treat them that way and the token bill takes care of itself.

chrome-bridge: let any AI agent drive your real logged-in Chrome

Ivan Seredkin — Sun, 06 Sep 2026 23:16:22 +0000

Ask an agent to check something on a website and most tools spin up a clean browser with zero cookies. Playwright and friends drive a browser they launched, with a profile of their own. MCP browser bridges need an MCP-capable client and a configured server. Either way you start every session logged out.

chrome-bridge takes the other road: it drives the Chrome you are already logged into. A tiny unpacked extension talks over WebSocket to a local zero-dependency Node server, and any agent that can run a shell command can drive the browser:

node cli.mjs snap localhost:8082         # compact a11y snapshot with element refs
node cli.mjs click localhost:8082 @e4    # click by ref
node cli.mjs fill localhost:8082 @e2 "hello@example.com"

A snap is the whole page as a compact text tree with refs you act on, roughly 10x cheaper on tokens than a screenshot:

table "Hacker News new | past | comments | ask | show | jobs | submit" @e1
  link "Hacker News" @e5
  link "new" @e6
  link "submit" @e12
  link "login" @e13

A few things that make it pleasant to live with:

refs survive re-snaps, so the agent re-checks with --diff instead of re-reading the whole page
driven tabs get a purple pill that narrates what the agent is doing right now, plus a live command feed in your terminal
multiple Chrome profiles are supported, and it refuses to guess which one you meant
works with Claude Code, Cursor, Qwen, GLM, Kimi, or a plain script hitting the local HTTP endpoint, no MCP setup

Setup is one paste into your agent plus one click to load the extension (Chrome requires that click). MIT licensed, zero dependencies, Node >= 18.

Repo and full agent manual: https://github.com/siropkin/chrome-bridge

Feedback welcome, especially on token cost of the snapshots against real workloads.

DEV Community: Ivan Seredkin