- Book: AI Agents Pocket Guide
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Your coding agent can now open Finder, read every button in the window as JSON, and click one by its ID, while your own pointer stays exactly where you left it.
That is the pitch of agent-desktop, which Vercel Labs released as version 1.0.2 on October 30. The README calls it "Computer use for AI agents: read and operate desktop apps through the OS accessibility tree." It is a Rust CLI plus a C library, Apache-2.0, and it runs on macOS, Windows and Linux. It contains no model. It is pitched as computer use for any coding agent, and the agent drives it by running shell commands.
What shipped, and when
The builder's launch post went up on X at 23:48 UTC on October 30, according to CellCog. Vercel's Chris Tate then described it as "agent-browser, for your whole computer," a nod to Vercel's agent-browser, which does the same thing for web pages. Vercel CEO Guillermo Rauch shared it with "An agent on every desktop."
The project is older than the launch: it started in February and later moved under the Vercel Labs organization, per ShiftDelete. Windows 10 1809+ support arrived in v0.10.0 on October 18, and v1.0.2 on October 30 added a docs site and made AGENTS.md the single instructions file for agents. MarkTechPost and AlexTech covered it the next day. The repo had about 6,900 stars on November 2, per its GitHub page.
The docs list the floors: macOS 13 or later (Intel or Apple Silicon) through the AX API, Windows 10 1809 or later through UI Automation, and Ubuntu 24.04 or later through AT-SPI, on x64 and ARM64.
How the loop works
The tool does one step per call. Each command prints one JSON envelope and exits. The observe, act, verify loop lives in your agent. Adapted from the README:
# setup
npm install -g agent-desktop
agent-desktop status
agent-desktop skills get desktop
# look: interactive elements, each with a ref
agent-desktop snapshot --app Finder -i
# act on a ref from that snapshot
agent-desktop click @s8f3k2p9:e8
# look again to check it worked
agent-desktop snapshot --app Finder -i
The ref values above are the README's examples. Yours come from your own snapshot. Refs are tied to the snapshot that produced them: v0.10.0 removed bare refs like @e8, and the README says they now fail with INVALID_ARGS. If the UI changed, the call returns STALE_REF and the agent should snapshot again. If several windows match, AMBIGUOUS_TARGET lists candidate window IDs to pass with --window-id. Every error carries a disposition, and the README says to retry only when error.disposition.retry is safe. Put that rule in your agent's AGENTS.md or system prompt.
Calls are headless by default. Per the README, ref actions go through the accessibility API "without moving the pointer or taking focus." A global --headed flag allows physical mouse and keyboard input when an app needs it. With several agents at once, each subagent gets its own colored overlay cursor, ShiftDelete reports, and actions that need the one physical pointer take turns.
The agent reads the app's accessibility tree and acts on one node by ref. Your pointer isn't part of it.
The token numbers are the builder's own
The project's headline argument against screenshot-based computer use is cost. Every figure for that so far comes from the project itself.
The builder's posts report 273 tokens per look at Apple Maps when you snapshot one region, against 3,811 for the whole window, and a simulated 52-step task dropping from 198k tokens to 31k, as summarised by CellCog. AlexTech puts the skeleton mode's saving at 78% to 96% against full-tree dumps in Slack, VS Code and Notion, and calls these "self-reported figures not verified by third parties." CellCog also points out that there is no shared yardstick for these numbers yet.
The technique behind the saving is in the README. You map a dense app shallowly first, then drill into the part you care about:
agent-desktop snapshot --skeleton --app Slack -i --compact
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact
I haven't run agent-desktop myself, so I can only relay these numbers. If you try it, count tokens on your own app. Slack and Notion are not your internal Electron tool.
What it can't see
The docs are blunt about the limits. It reads accessibility trees and does not match pixels, so a UI that publishes no accessibility tree is invisible to it. How well it works depends on how well each app's interface is built.
It also does not automate web pages. For those, use agent-browser. For web content inside a Chromium or Electron app, agent-desktop opens a DevTools port that agent-browser can connect to.
The CLI has 60 command names, the same on every platform. Four of them (key-down, key-up, mouse-down, mouse-up) fail closed everywhere. And a successful action does not prove the UI changed, which is why the docs recommend a follow-up read such as wait --element ... --predicate value.
No accessibility tree, no view. Pixel-only UIs stay out of reach.
Before you hand it to an agent
A few things in the docs are worth settling before the first run.
Who gets the permission. On macOS the Accessibility permission goes to the app that runs agent-desktop. If your agent runs in a terminal, that terminal is the app asking, so decide on purpose which one gets it. agent-desktop permissions --guide walks you through it, and the README also requires TMPDIR to be owned by you with mode 0700. On Windows no grant is needed for apps at your terminal's integrity level.
What ends up in transcripts. The README warns that "Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content." The FAQ goes further: "redaction does not make a screenshot safe to publish." Your agent's transcript now holds whatever was on screen, so treat it like a log with secrets in it.
What leaves the machine. The FAQ says the native CLI "does not require a remote model service." The optional JEV helper is different: it sends goals and screen descriptions to the model provider you configure.
What git won't show you. A click in Finder leaves no diff. Keep desktop actions on a short leash: name the app, name the task, and use --headed only when headless fails. I build Hermes IDE, so weigh this with that in mind. Hermes runs each agent in a real terminal, so an agent there can call agent-desktop like any other CLI. But Hermes, by its README, "observes what agents report on purpose and what git shows." Its Review Desk can show you every file an agent changed, turn by turn. It cannot show you what the agent clicked in another app.
Would you give it your desktop?
A cheap first test: point your agent at one app, let it run snapshot only, and read what comes back before it gets to click anything.
Which native app would you trust an agent to drive first, and which one would you never hand over? Tell me in the comments.
If this was useful
agent-desktop is a tool, and the interesting design work sits in the loop around it: when to look again, when a retry is safe, when to stop and ask. That loop is what the AI Agents Pocket Guide is about, from tool calling to guardrails for agents that act on their own.
If you run more than one coding agent, Hermes IDE is free, source-available and on macOS, Windows and Linux, and the code is on GitHub.
Top comments (0)