I gave Claude Code one sentence in a terminal: make a collage of my beach photos and send it to my Telegram channel, so I have it on my phone. Four minutes later it was on my phone. Claude Code had opened Photos, selected the album, exported eight pictures, built the collage, checked it in Preview, pasted it into Telegram, written the caption and sent it.
None of those apps has an API for that. Photos doesn't, and I wasn't going to wire up a Telegram bot to post as myself. The agent drove the apps through agent-desktop, an open-source CLI from Vercel Labs that does one thing: it reads each app's accessibility tree and acts on it.
That tree is the same structure macOS hands to VoiceOver. So the question that kept coming back while I watched the run was not "can an agent use my Mac". It was: how much of my Mac can it actually see?
One tree, two readers
Every native window on macOS publishes a tree of elements: a role (button, textfield, menuitem), a name, sometimes a value, and the actions it accepts. A screen reader walks that tree and speaks it. agent-desktop walks the same tree and returns it as JSON with a reference for each element, so the agent can say "press @e12" instead of guessing coordinates on a screenshot.
One structure, two readers. Whatever an app leaves out of its tree is invisible to both.
That framing changes what "computer use" means. It isn't vision. The agent doesn't look at pixels; it reads what the app chose to describe about itself. If the app describes itself well, the agent is fast and precise. If it doesn't, there is nothing to read.
What 23 apps expose
So I measured it. I opened the apps I have installed, one at a time, and read the main window's tree with agent-desktop three times each: how many elements it exposes, what share of them have a name, and how many accept an action. Only counts, no content.
A few things stood out:
- Telegram exposes 1,493 elements and every single one has a name. Each chat, each message, each counter. For an agent it is the most legible app on the machine.
- Spotify exposes almost as many, but half of them unnamed. A button with no name is a button the agent has to guess. A screen reader user has to guess too.
- The Android emulator exposes one element: the window. The whole phone inside it is a picture. The agent can see that there is a window called "Android Emulator" and nothing else.
- Docker Desktop didn't answer at all. Three reads, 45 seconds each, no tree.
- ChatGPT went from 12 elements to 406 between the first read and the second. Just opened, it exposed a shell; a few seconds later, the whole interface. Measuring cold would have called it opaque.
- The Terminal exposes 17 elements, because the entire transcript is one text field. The agent can read it, but as one block of text, not as structure.
Two caveats. These are counts from one machine on one day, of whatever window each app opened with, and a bigger tree is not automatically a more usable one. The share of named elements says more than the size. But the extremes are not subtle: between "one element" and "1,493 named elements" there is no measurement error that changes the conclusion.
I later repeated the census on a Windows 11 machine, where agent-desktop reads the UI Automation tree instead, and the picture held. The same app reads about the same on both systems: Telegram exposes 1,509 elements there against 1,493 here, Slack 483 against 616. The cold first read happens there too: Spotify gave 183 elements, then 1,651. Of the rules further down, the one about the current desktop held; the other two didn't, because on Windows a background app's menu reads as enabled and the save dialog belongs to the app's own process.
Exposed is not the same as operable
The collage run taught me the second half, and it's the part a census doesn't show. Telegram, the most legible app in the chart, was also where the agent struggled most.
To send an image it pastes it into the message box and Telegram opens a send panel with a caption field and two buttons. Everything in that panel is in the tree, with good names. And yet:
Named, but not operable. The caption is on screen and empty in the tree; the buttons are listed and can't be pressed.
- The caption field reports an empty value while the screen clearly shows the text. The agent couldn't confirm what it had typed by reading; it had to take a screenshot of that area and look.
- Send and Cancel accept no press action. The tool refused to click them semantically, and a physical click through the tool failed too. What worked was the keyboard: Enter sends, Escape cancels.
- Even typing failed there. The field never reported having focus, so the tool wouldn't type into it. The workaround was to put the text on the clipboard and paste.
Telegram isn't alone. Other things I hit along the way, each turned into a rule the agent now follows:
- Menus only work in the frontmost app. Ask Photos for File › Export while it's in the background and the item reads as disabled. Bring the app forward and it's enabled.
- The save dialog is a different process. The panel where you choose a folder doesn't belong to Photos, so keyboard shortcuts sent to Photos never reach it. In one rehearsal, a semantic click on a folder inside it saved the export straight into that folder.
- Actions can land late. A paste that seemed not to have worked showed up a moment later, and the panel ended up with two copies of the image. Every step needs a fresh look before the next one.
- The agent only sees the current desktop. Switch to another macOS Space and every window there disappears from the tree.
- Some trees are too big to read whole. A long Telegram chat produced a tree over the tool's 1 MB limit; the agent had to read the window shallowly and then drill into the message list.
None of these is exotic. They're the same seams a keyboard or screen-reader user hits: a value the app doesn't publish, a control with no action, a dialog in another process.
What a screen reader user would get
I didn't run these tasks with VoiceOver, so I'll be careful here. But VoiceOver asks the same API the same questions. A caption field that reports an empty value is, to it, an empty field. A button with no press action is a button that VoiceOver's own activate command has nothing to call; it would have to fall back to simulating a mouse click, as the agent did. An emulator window with one element is, to a blind user, one unlabelled window.
The difference is who pays. A screen reader user meets these gaps one at a time and works around them with patience. An agent meets them too, but then they become rules: "press Enter, the button doesn't take clicks", "paste, don't type". The workarounds the agent needed are a map of where an app's accessibility is broken.
It reminded me of what I found measuring agents on degraded code: in large repositories, how the code is written halves what an agent solves. Here it's the same lesson one layer up. The agent is only as good as what it can read, and what it can read is whatever someone bothered to describe.
If you build apps, or agents
If you build apps: accessibility used to be about people who use assistive technology. It still is, and that alone is reason enough. But the same tree is now the interface for every agent that operates a computer the way a person would. Name your controls, publish your values, give your buttons a press action. That's the whole checklist, and it serves both readers at once.
If you build agents on top of this: treat each app as having its own rules, and verify after every action, because the tree will sometimes tell you something worked when it didn't, or that it failed when it hadn't. The run that finally went clean, in a bit under four minutes, wasn't a smarter model. It was a short list of what each app lets you do.
If an app is accessible, it's a tool. If it isn't, for an agent it barely exists.
Measured on macOS 15.7 with agent-desktop 1.0.2: main window of each app, three reads, counts only. Figma and Cursor didn't launch from the command line on this machine and aren't in the chart. The Windows comparison was measured on Windows 11 with the same method; the chart shows the macOS run.
Originally published on javieraguilar.ai
Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.
Top comments (0)