DEV Community

Cover image for agent-desktop computer use: an AI agent driving your desktop
Span Observer
Span Observer

Posted on Originally published at span.observer

agent-desktop computer use: an AI agent driving your desktop

#a11y #aiagents #computeruse #mcp

agent-desktop computer use: the vercel-labs/agent-desktop repository offers a command-line tool that lets an AI agent see and operate desktop applications on macOS, Windows and Linux. It does not rely on screenshots: it reads the operating system’s accessibility tree and returns it as JSON, with a reference (a « ref ») for every element the agent can act on. The repository is published by Vercel Labs under the Apache 2.0 licence.

For an SRE, this kind of tool changes the risk perimeter. An agent that clicks and types acts on a real machine, with the user’s rights. What does the repository say, how do you install it, and what can you trace and control when an agent drives a desktop?

A note on honesty: span.observer has not installed or run agent-desktop. The commands come from the README, the docs/ files, CONCEPTS.md and the project documentation, read on 5 November 2026. The images below reproduce those texts. The star count could not be read at the source that day, so it is not quoted.

agent-desktop computer use: what the repository says

  • Principle: the tool reads an application’s accessibility tree and prints it as JSON. The agent acts on a ref, rereads the interface to check the result, then repeats. According to the README, the loop lives in your agent: agent-desktop has no model inside.
  • Form: a CLI and a C library. The repository changelog dates version 1.2.0 to 5 November 2026.
  • Licence: Apache 2.0, according to the README and the LICENSE file.
  • Commands: the README counts 60 command names, 56 of them operational. Four (key-down, key-up, mouse-down, mouse-up) refuse to run until a daemon can release a held key or button.
  • MCP: version 1.2.0 adds agent-desktop mcp, an MCP server over stdio that exposes each command as a tool.

A word on maturity: the project comes from a lab of a well-known vendor, but it is young and moves fast (the changelog lists breaking changes in 1.1.0). Read the changelog before each upgrade.

A tour of the commands and outputs

The first image reproduces the README quick start and the snapshot excerpt it shows. It displays the JSON envelope: a snapshot_id, a ref counter, the complete flag and, for each actionable element, a ref_id that embeds the snapshot identifier.

Excerpt from the vercel-labs/agent-desktop README: three commands (snapshot, click, snapshot) on Finder, and the JSON envelope of a snapshot with snapshot\_id, ref\_count, complete and a treeitem element carrying a ref\_id

Source : github.com

This design has a useful security consequence. According to the README, a ref belongs to its snapshot: a bare ref such as @e8 is rejected with INVALID_ARGS. When the interface changes, the action returns STALE_REF or AMBIGUOUS_TARGET. The agent therefore does not click blindly on a stale position; it has to read the screen again.

The second image shows the error contract described in docs/json-output.md, without its optional suggestion field. Every error carries a disposition field: delivery says whether the action went out, and retry says whether it can be replayed. The documentation says to retry automatically only when retry is safe, and to treat unknown as unsafe.

Excerpt from docs/json-output.md: JSON envelope of a STALE\_REF error on the click command, with recovery (refresh\_snapshot\_then\_retry\_original) and disposition not\_delivered, retry safe

Source : github.com

Installing step by step

The commands below come from the README. Requirements, according to its platform table: macOS 13 or later, Windows 10 1809 or later, or Ubuntu 24.04 or later (X11 or Wayland).

  1. Install the npm package, which includes native x64 and ARM64 binaries:

    npm install -g agent-desktop

  2. On macOS and Linux, Homebrew also works: brew install agent-desktop. The README states that there is no official Scoop package.

  3. Check the platform, permissions and supported surfaces:

    agent-desktop status
agent-desktop permissions --guide   # macOS: walks you through the Accessibility grant

  4. Run the observe, act, verify loop:

    agent-desktop snapshot --app Finder -i
agent-desktop click @s8f3k2p9:e8
agent-desktop snapshot --app Finder -i

  5. For an MCP client, declare the server (README excerpt):

    { "mcpServers": { "agent-desktop": { "command": "agent-desktop", "args": ["mcp"] } } }

In PowerShell, the README asks you to quote every ref with single quotes, because a bare @token is the splatting operator.

Security: what is documented, what remains your job

  • Private data: the README warns that snapshots, clipboard reads, screenshots and command output can contain private desktop content. It asks you to keep them out of public logs.
  • Headless by default: ref actions use the accessibility API without moving the pointer or taking focus. The global --headed flag allows physical input.
  • Secrets: type, set-value and clipboard-set accept --stdin to keep text out of process arguments. The README advises running secret steps as separate commands, because batch JSON text appears in process arguments.
  • Permissions: on macOS, the program that runs agent-desktop needs the Accessibility permission. Granting that right to an agent amounts to handing it the workstation.
  • Functional limit: the tool only sees what an application publishes in its accessibility tree. The repository FAQ says so: an app with no tree is not visible.

These safeguards do not replace those of the agent itself. See guardrails for AI agents in production for the action limits to set on the production side, and detecting MCP tool poisoning if you connect the agent-desktop mcp server to a client that loads other tools.

Observing an agent that drives a desktop

The repository documents a session and trace mechanism, not OpenTelemetry telemetry: our searches through the repository’s Markdown files find no mention of OpenTelemetry, OTLP or Prometheus. Do not assume any.

The third image reproduces the « Sessions and traces » section of the command reference. A session is a folder under ~/.agent-desktop/sessions/<id>/ with a manifest and, by default, a trace/ directory. Trace segments are JSONL files, one per process.

Excerpt from the repository's commands-system.md: session start command with a name, session end, and four lines on trace show, trace export, redaction of sensitive fields and the unredacted screenshots option

Source : github.com

  • Timeline: trace show returns the merged event timeline; trace export writes a self-contained HTML viewer. Neither needs an OS permission.
  • Redaction: trace lines replace sensitive fields (text, value, name, title, url…) with { "redacted": true }. With --screenshots, the before and after PNGs are not redacted: treat them as sensitive screenshots.
  • Isolation: a session owns its snapshot namespace. Several agents may share a session id, provided each uses the qualified refs from its own snapshot.
  • Status: status reports session_id, tracing and artifacts.

To link these traces to your monitoring, the work remains yours: emit one span per action from the agent’s orchestrator, with the command, the error code and the duration, and never the field contents. The principle matches an agent driving Chrome through DevTools or a headless browser for agents: the tool acts, and the instrumentation is built around it. The project documents no Dynatrace integration; an OTLP export from your orchestrator to Dynatrace SaaS remains possible, and its configuration belongs to the Dynatrace documentation.

Best practices

  • Run the agent on a dedicated account or machine, with no open session on sensitive data.
  • Retry a command only when error.disposition.retry is safe.
  • Keep the headless default; use --headed only for a specific step.
  • Pass secrets through --stdin, as separate commands.
  • Use named sessions, remove old ones with session gc, and keep trace folders private.
  • Log the command name, the error code and the duration in your own telemetry, never the content read.

Frequently asked questions

Does agent-desktop need screenshots?

No. According to the repository FAQ, commands that take a ref find the element in the accessibility tree, not by matching pixels. The screenshot command exists separately.

Does agent-desktop also drive web pages?

No. The documentation says it does not automate web pages and points to agent-browser for that.

Does a remote model receive my data?

The FAQ states that the native CLI does not require a remote model service. Your agent, however, sends what it reads to its own model: you must control that flow. An optional helper in the repository, JEV, sends your goal and screen descriptions to the provider you configure.

Conclusion

agent-desktop brings computer use built on the accessibility tree, with strict JSON output and a clear error contract. Its strength for operations is traceability: sessions, traces and redaction by default. Its limits are those of the genre: an agent acting on a real workstation, private data everywhere, and no standard telemetry to plug in. Read the documentation again before each deployment.

span.observer is an independent community blog, not affiliated with Vercel or agent-desktop, nor with Dynatrace LLC. Brands mentioned belong to their owners. Tools change quickly: check commands and options in their official documentation.

Originally published on span.observer. French version: https://span.observer/fr/agent-desktop-computer-use-agents-ia/.

Top comments (0)