Vercel Labs dropped agent-desktop 1.0.2 on October 30, 2026. It is a Rust CLI and C library that lets any shell-capable agent read and operate native desktop apps on Windows, macOS, and Linux by walking the OS accessibility tree instead of guessing from screenshots. On Windows the backend is UI Automation (UIA). The loop stays in the agent: snapshot returns JSON with stable refs, the agent acts on a ref, then snapshots again to verify. No pixel matching, no vision model required for the core path, headless by default.
This is the practical opposite of the screenshot-and-coordinate approach that most computer-use agents still use. Screenshots are fragile: DPI scaling, themes, window position, and partial occlusion all break coordinate maps. Accessibility trees give structured roles, names, bounds, and available actions. When the tree is complete and the app actually exposes it, the agent can target a button by ref instead of calculating (x, y) that will be wrong the moment the user moves the window.
What the tool actually does
Install is one line:
npm install -g agent-desktop
agent-desktop version
A minimal Windows loop looks like this:
agent-desktop snapshot --app Notepad -i
# returns JSON with snapshot_id and refs like @s8f3k2p9:e3
agent-desktop click @s8f3k2p9:e3
agent-desktop snapshot --app Notepad -i
The snapshot command walks the UIA tree (or AX on macOS, AT-SPI on Linux), assigns deterministic refs in depth-first order, and embeds the snapshot id so the ref is self-describing. Actions that go through the accessibility API do not move the pointer or steal focus unless you pass --headed. That matters for real workflows: you can keep typing while an agent operates another window.
On Windows the requirement is modest. Apps at the same integrity level as the terminal need no extra permission grant. Elevated processes still need the usual care. The tool reports completeness on the observation and on any truncated node, so you know when the walk hit a time or depth budget.
Token economics and drilling
Dense Electron apps are the classic problem. A full snapshot of Slack or VS Code can be thousands of tokens. agent-desktop lets you start with a skeleton:
agent-desktop snapshot --skeleton --app Slack -i --compact
Then drill into one region with a root ref. The project claims 78–96 % fewer tokens on dense apps via this skeleton + drill-down pattern. Whether those numbers hold for your app depends on how well the accessibility tree is implemented, but the architecture is sound: you only pull the subtree you need into the context window.
Partial observations are first-class. If the walk runs out of time it returns what it saw and marks the boundary. That is more useful than a hard failure when you are operating live UI that is still loading.
Windows internals and tradeoffs
UIA is not a perfect map of the visual tree. Many Win32 apps still expose thin or missing trees. Custom-drawn controls, games, and some Electron builds that ship with accessibility disabled will return almost nothing useful. In those cases you fall back to screenshots or keyboard simulation. agent-desktop keeps the screenshot command as a separate path so you are not forced to choose one forever.
Integrity levels matter. A medium-integrity agent-desktop process cannot freely inspect a high-integrity window without the usual UAC and UIAccess gymnastics. For most developer tooling this is fine; for system-level automation you still need the old elevated helper patterns.
The C library (libagent_desktop_ffi) exists so you can load it once from Python, Go, or Node instead of forking a process on every call. Latency for a snapshot is dominated by the UIA walk, not by process startup, once you are in-process. That is the difference between a tool that can sit in a tight observe-act loop and one that feels like a batch script.
Error codes are machine-readable and include recovery hints. A STALE_REF tells you the UI changed under the previous snapshot; the safe retry is another snapshot, not a blind click. That is the kind of detail that keeps an agent from thrashing.
Where this fits developer workflows
If you already run local agents that drive tests or fill forms, the accessibility path reduces the amount of vision-model tokens and the flakiness of coordinate math. You still need a model that can plan, but the observation side becomes structured data instead of pixels. For tools that overlay or inject into existing Windows processes (real-time transcription, translation during calls, local STT pipelines), having a reliable way to read and write UI state without screenshots lowers both latency and the chance of visual glitches.
The same pattern shows up in other recent work: Factory’s Droid 1.0 collapses hundreds of tool calls into one model request, Gemini Computer Use went generally available, and several groups are pushing agents that prefer structured OS surfaces over raw pixels. agent-desktop is the open, cross-platform piece that makes the structured path concrete on the desktop.
Practical limits
- Apps that do not implement accessibility (or implement it poorly) remain hard. You will still see empty or shallow trees.
- Linux needs a running session and AT-SPI; headless boxes without the bus are out of luck for the tree path.
- Refs are snapshot-scoped. Crossing snapshots without a fresh snapshot is an error by design.
- The tool is not an agent. It does not plan, it does not hold state between calls, and it does not embed a model. You still own the loop.
That last point is the useful one. The observe-act-verify cycle stays in your code or your agent harness. You can add retries, timeouts, or parallel sessions without fighting a black-box computer-use model that only returns mouse coordinates.
agent-desktop is Apache-2.0, ships native binaries for x64 and ARM64, and the current release is 1.0.2. The repository is at vercel-labs/agent-desktop. If you are building Windows-side automation that needs to survive theme changes, DPI scaling, and live UI updates, the accessibility tree is the lower-friction path once the app cooperates.
I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.
Top comments (0)