Vercel Labs just open-sourced agent-desktop (v1.0.2, Apache-2.0), a Rust CLI and C library that gives any shell-capable coding agent computer-use on macOS, Windows, and Linux by reading the OS accessibility tree instead of screenshots or pixel matching. A snapshot returns JSON with a stable ref for every actionable element. The agent acts on the ref, then snapshots again. No vision model required for the core loop.
For people who ship Windows tools—overlays, dual-track audio pipelines, live STT, BYOK clients—this is more interesting than another “AI clicks the screen” demo. Pixel-based computer use is slow, brittle under DPI scaling, theme changes, and multi-monitor setups, and it burns tokens on every frame. Accessibility trees are the same structured data screen readers already use. They are deterministic, cheap to traverse, and already exposed by Win32, UIA, and Electron apps that bother to implement them.
Architecture in one paragraph
The binary is pure Rust with platform adapters. On Windows it talks to UI Automation (UIA). A snapshot command walks the tree (or a skeleton + drill-down), assigns compact refs like @s8f3k2p9:e8, and dumps one JSON envelope. Actions (click, type, focus, expand, scroll, etc.) take a ref and are headless-by-default—they use the accessibility APIs rather than injecting mouse events that can fight with the user’s real cursor. There is also a C-ABI library so you can load it once from Python, Go, or Node instead of forking a process every call. Progressive skeleton traversal is the token-saving trick: ask for a shallow overview, then drill into the region you care about. Self-reported numbers put dense Electron apps at 78–96 % fewer tokens versus dumping the full tree.
# install
npm install -g agent-desktop
# permissions / status
agent-desktop status
agent-desktop skills get platform # Windows-specific notes
# snapshot an app
agent-desktop snapshot --app "Notepad"
# act on a ref (PowerShell needs quotes because @ is the splatting operator)
agent-desktop click --ref '@s8f3k2p9:e12'
agent-desktop type --ref '@s8f3k2p9:e12' --text "hello"
Windows-side practicalities
UIA is not free. Some older Win32 controls expose almost nothing; Electron and WinUI apps vary wildly in how complete their trees are. Overlay windows and apps that set WS_EX_LAYERED or deliberately hide from accessibility (to stay out of screen-share or screen-reader enumeration) will be invisible or incomplete. That is the same tradeoff you already hit when you want a floating overlay that does not appear in Zoom or Teams captures.
Latency matters if you are also running a real-time audio graph. A full tree walk on a busy Electron app can be tens of milliseconds; skeleton + root drill is cheaper. Pin the agent-desktop process or use a job object so it does not fight your WASAPI capture threads or a local STT model for CPU. Because the actions are accessibility-driven rather than synthetic input, they usually do not steal focus in the same way SendInput does—useful when you still want the user’s keyboard to work.
# PowerShell example – quote the ref
agent-desktop snapshot --app "YourApp" | ConvertFrom-Json
# then
agent-desktop click --ref '@s8f3k2p9:e3'
Where it actually helps developer workflows
The interesting use is not “agent fills out a form for me.” It is:
- Driving your own internal tools or test harnesses without writing fragile UI automation in WinAppDriver or FlaUI every time the layout changes.
- Letting a coding agent open a settings dialog, toggle a feature flag, or inspect a log window while it is also editing code—without you pasting screenshots into the chat.
- Building a small supervisor that watches an overlay or a dual-track recorder and restarts or reconfigures it if the accessibility tree shows it is stuck.
Compared with screenshot + vision-model loops (Claude computer use, GPT-6.1 Terra computer-use, etc.), you trade generality for determinism and cost. If the app exposes a decent tree, refs stay stable across theme changes and DPI; if it does not, you fall back to screenshots (the CLI still has a screenshot command) or write a proper API. That is the same practical decision you make when choosing between a local STT model and a cloud endpoint: own the path when you can, accept the network when you cannot.
Tradeoffs and limits
- No GPU, plain CPU. Good.
- Does not automate web pages inside browsers (use something like agent-browser for that).
- Apps that deliberately suppress accessibility (some screen-share-safe overlays, DRM-ish UIs, certain games) are invisible.
- Still early; error recovery hints are in the JSON, but you will write your own retry and verification loop. The skill docs (
agent-desktop skills get desktop) spell out the recommended observe-act-verify pattern. - Token savings are real only if you use skeleton traversal. Dumping the entire tree of Visual Studio or a fat Electron app still hurts.
Not a perfect match, but the same idea: structured intermediate representation beats raw pixels or raw tokens.
If you are already building Windows desktop agents, overlays, or live audio tools, drop the binary next to your process and start calling it. The loop stays in your agent; the tree walk stays in the Rust binary. That separation is the part worth copying.
I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.
Top comments (0)