DEV Community: Leooo

Why AI Agents shouldn't rely on screenshots: Building a cross-platform alternative to Anthropic's Computer Use

Leooo — Tue, 24 Mar 2026 17:22:28 +0000

Anthropic recently released their Computer Use feature for macOS. It is a big step forward for AI agents, allowing models to interact with local software. However, this release also highlights a major technical bottleneck in how we are building GUI agents today. The current approach relies heavily on taking continuous screenshots and using large vision models to figure out where to click. This method is slow, expensive, and currently leaves Windows users out of the loop.

When an agent uses screenshots, it essentially treats the operating system like a flat picture. It takes an image, sends it to the cloud, waits for the vision model to calculate pixel coordinates, and then finally moves the mouse. If a UI element shifts by a few pixels or the network is delayed, the action easily fails. Clicking a single button can take several seconds and consume a lot of tokens.

We need a more efficient way for agents to interact with software. Human developers use APIs to talk to applications, and AI agents should have a similar structural interface. This is why I built the Agent-Computer Interface (ACI).

ACI is an open-source protocol that turns any application into a structured JSON tree. It allows large language models to read and operate software interfaces directly through text, completely bypassing the need for pixels and screenshots for standard tasks.

The core idea behind ACI is unification. Currently, web browsers use the DOM to organize elements, while Windows desktop applications use UI Automation (UIA). ACI reads these completely different underlying structures and converts them into one universal node tree.

Because the data format is standardized, the agent only needs to learn two basic commands: perceive and act.

When the agent calls the perceive command, it receives a clean, structured list of all interactive elements currently on the screen. Each element, whether it is a web link or a local desktop button, gets a unique ID. If the agent wants to click the search bar, it simply calls the act command and targets that specific ID. The interaction logic is exactly the same whether the agent is browsing Hacker News or operating a local Windows notepad. This drops the action latency from several seconds down to milliseconds.

Of course, purely reading UI structures cannot solve every single edge case. Some applications use custom rendering engines or complex canvas elements where the structural data is hidden. ACI handles this practically. It defaults to the lightning-fast structural parsing for the vast majority of tasks. When it encounters a completely blind area like a canvas, it automatically falls back to a vision model to capture a visual reference image for that specific region.

This hybrid routing approach ensures that the agent stays extremely fast most of the time without losing the ability to handle complex graphical interfaces.

I believe treating the UI as an API rather than an image is the right path forward for agent automation. The ACI framework is fully open-source and ready to be tested. If you are building AI agents or working on desktop automation, I would love to hear your feedback or see your contributions.

GitHub Repository: https://github.com/Leoooooli/ACI

Why open-source this? Computer Use should be a universal standard, not walled gardens. Looking for contributors to: • Add macOS/Linux desktop bridges • Build app-specific YAML configs • Improve VLM fallback accuracy All PRs welcome! 🤝

Leooo — Tue, 24 Mar 2026 16:25:41 +0000

Leooo

Mar 23

I Built ACI: The Open Standard That Lets AI Agents Operate Any Software Without Screenshots

#automation #ai #opensource #python

Comments

2 min read

I Built ACI: The Open Standard That Lets AI Agents Operate Any Software Without Screenshots

Leooo — Mon, 23 Mar 2026 18:34:56 +0000

Every AI agent framework today solves half the problem. Browser Use handles web pages but can't touch desktop apps. UFO controls Windows apps but can't operate browsers. Screenshot-based approaches (Computer Use, Operator) work everywhere but are slow (3-10s per action), expensive, and fragile.

The missing piece: a single standard interface that works for both web and desktop, structured and fast.

That's what I built. ACI — the Agent-Computer Interface.

The Core Idea: APIs for Developers, ACI for Agents

Just as APIs became the universal interface between developers and software, ACI is the universal interface between AI agents and software.

How It Works: Two Operations

The entire protocol is two operations:

1. — See what's on screen

Returns a structured, UID-referenced element tree — not a screenshot, not raw HTML:

\
Same protocol for desktop apps:

2. — Do something

\
That's it. . Any agent that can make HTTP calls can operate any software.

Three Technical Innovations

1. Tiered Structured Extraction

Instead of screenshots, ACI uses fast structured methods first and falls back to vision only when necessary:

Web: CDP Accessibility Tree (5-50ms) -> DOM Supplement (20-100ms) -> Vision Fallback (1-2s, only if needed)

Desktop: UIA Control Tree (50-300ms) -> Cursor Probing (~350ms) -> OCR (~200ms) -> VLM (1-5s, last resort)

Most interactions complete in 50-300ms instead of 3-10 seconds.

2. Community Knowledge Base (YAML App Profiles)

Every app can have a YAML profile that teaches agents shortcuts and UI patterns:

\
Add a YAML file, push to the repo — every agent instantly knows how to use that app. Zero code changes. Currently ships with 12 app profiles (Chrome, VS Code, Discord, Slack, Notion, Telegram, and more).

3. Interrupt-Aware Execution (MutationShield)

Real software throws popups, cookie banners, and auth dialogs. ACI's MutationShield detects and reports these as structured events instead of crashing.

The Stack

Python + FastAPI daemon (port 11434)
Playwright for web automation (cross-platform)
Windows UIA for desktop automation (macOS/Linux on roadmap)
Works with any LLM — tested with Claude and GPT
Apache 2.0 licensed

GitHub: https://github.com/Leoooooli/ACI

Plot Twist

This article was written and published entirely by an AI agent (Claude) using only the ACI protocol.

The agent connected to this browser through ACI, perceived the Dev.to editor (finding the title field, tag inputs, and content textarea by their UIDs), typed this article, and clicked Publish. No Selenium scripts. No hardcoded CSS selectors. Just — the same two-operation protocol described above.

If an agent can write and publish a blog post using your framework, it probably works.