agent-desktop 1.2.0 Turns the Accessibility Tree into 67 MCP Tools — Overlay Latency Still Wins

#mcp #windows #automation #agents

On 5 November 2026 Vercel Labs shipped agent-desktop 1.2.0. The release notes are short. The practical change is that agent-desktop mcp now serves a Model Context Protocol server over stdio and exposes every leaf CLI command as a tool. Tool names and argument schemas are generated from the clap definitions at runtime. A call returns the same JSON envelope the CLI already produced. Agents that previously had to shell out or parse stdout now get structured, typed access to the desktop.

agent-desktop itself is the Apache-2.0 Rust CLI and C library that lets agents see and operate desktop applications on macOS, Windows, and Linux through the operating-system accessibility tree rather than screenshots. On Windows that means UI Automation (UIA). You snapshot the tree, receive stable element references, then click, type, set values, or read text without ever looking at pixels. Version 1.2.0 adds the MCP surface on top of that existing capability.

The MCP surface in practice

The server is started with a single command:

agent-desktop mcp

Clients that speak stdio MCP (Claude Code, Cursor, Codex, Crush, etc.) configure it like this:

{
  "mcpServers": {
    "agent-desktop": {
      "command": "agent-desktop",
      "args": ["mcp"]
    }
  }
}

Tool names are desktop_ plus the command with hyphens turned into underscores. snapshot becomes desktop_snapshot. list-windows becomes desktop_list_windows. session start becomes desktop_session_start. There are roughly 67 tools. Global flags passed to the mcp subcommand apply to every call. The server handles both the current stateless MCP protocol and the legacy initialize handshake.

--stdin support landed in the same release for type, set-value, and clipboard-set. You can pipe UTF-8 text instead of stuffing long strings into argv. That removes one class of quoting and length problems when an agent needs to inject a paragraph into a form or a chat window.

On Windows the binary already carries the usual application manifest. That removes a set of “why won’t the accessibility client attach under a scheduled task or overlay process” failures. The remaining work is the usual: pin the DLL search path, keep the service account’s PATH correct, and make sure the target application has not disabled UIA or marked itself as unresponsive.

What UIA buys and what it does not

UIA is faster and more reliable than pixel-based computer use for any application that exposes a proper accessibility tree. Element references survive window moves and DPI changes. You do not burn GPU cycles on screenshot encoding or vision-model latency. For form filling, menu navigation, or reading status text this is the correct layer.

The same path does not help when the workload is real-time audio. A local agent that can click a button in 20 ms is still useless for a live interview or call if the microphone path has already burned 80–120 ms of buffering and the loopback path has added another 40–60 ms before the first sample reaches your process.

Typical measurements on a recent Windows 11 machine with an onboard Realtek and a USB headset still show:

Exclusive-mode WASAPI capture: 10–20 ms buffer + driver
Shared-mode / loopback: 30–80 ms depending on the audio engine mix period
Bluetooth or USB class driver: another 20–50 ms
Process isolation around an overlay: extra context switches and the kernel’s audio engine queue

The accessibility tree is no longer the long pole once you leave pure UI automation and enter a live speech pipeline. Speculative decoding and INT4 help the GPU side of a local model. They do nothing for the WASAPI shared-mode period, the Bluetooth SCO/A2DP latency, or the fact that a global hotkey overlay must still pull samples through the same engine.

Architecture that actually ships

A practical combination on Windows therefore looks like this:

Exclusive or low-latency WASAPI capture (or a careful shared-mode loopback with a short mix period) feeding a local STT process.
A separate low-latency overlay process that owns the hotkey and the on-screen translation or suggestion surface.
An agent (local or remote) that uses agent-desktop’s MCP tools only for the non-real-time parts: opening the interview app, pasting a prepared answer into a form, switching windows, or reading a status message.
Process boundaries that keep the audio path out of the agent’s address space so a tool-calling timeout cannot stall the capture graph.

The MCP server itself is a thin stdio wrapper. Each tool call spawns or reuses the native binary, executes the command, and returns the JSON. That design keeps the attack surface small and the implementation simple. It also means every tool call pays a process-start or IPC cost. For interactive UI automation that cost is negligible. For a 50 ms audio loop it is not. Keep the two planes separate.

Trade-offs worth measuring

UIA vs. screenshot computer-use: UIA wins on latency and reliability for structured apps; screenshot wins when the target has no accessibility tree or when you need visual verification.
stdio MCP vs. HTTP MCP: stdio is simpler for local agents and needs no auth. HTTP is better when the desktop process must live on another machine or behind a sandbox.
Overlay process vs. in-process injection: an overlay can be killed and restarted without taking down the audio engine; injection is faster but couples failure domains.
Local STT vs. cloud STT: local removes network RTT but still sits behind the WASAPI buffer. Cloud can be faster on the model side if your uplink is clean and the provider’s time-to-final is under 50 ms.

None of these choices remove the need to measure the actual capture-to-overlay latency on the machine you will use. The numbers above are typical, not universal.

Practical next step

If you already run agents that need to drive desktop applications, add the MCP server and point one of your local clients at it. Snapshot a window, inspect the element refs, and issue a type or click. Then measure how long the same agent takes to do the same work when the target is a real-time audio overlay instead of a static form. The difference is the budget you still have to spend on the capture path.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.

DEV Community