Stop trying to give agents "eyes" to look at human interfaces. Give them terminals, deterministic APIs, and native protocols instead.
We have officially entered the era of Agentic AI. The tech world is buzzing with demos of agents navigating the web, taking control of your mouse, and "seeing" your screen to fill out Excel spreadsheets or code in VS Code.
There is an undeniable allure to the idea of an AI using a computer exactly like a human does. It feels intuitive. It fulfills the sci-fi promise of a humanoid companion sitting at a keyboard. It suggests a universal compatibility where we don't need to rewrite our software because the AI can just "look" at it.
But as a researcher in Computer Vision and Robotics, I see a fundamental design flaw in this anthropomorphic approach. We are falling into a trap of mimetic design—building digital tools that mimic human biological constraints rather than leveraging silicon strengths.
The Graphical User Interface (GUI) is an expensive, lossy abstraction layer created to limit human cognitive load. We rely on buttons, icons, colors, and spatial layouts because our primate brains cannot memorize thousands of CLI flags or parse raw JSON streams in real-time. But forcing an AI—which excels at processing structured text, intricate logic, and massive data streams—to "look" at and "click" on an interface designed for human eyes is like forcing a supercomputer to count on its fingers.
The future is not in visual agents that click. It is in blind agents that execute.
The Distinction: Perception vs. Interaction
To be clear: I am not arguing against Computer Vision. AI should be able to see.
If I sketch a website layout on a napkin, I want the AI to look at it and generate the HTML. That is Perception—using vision to bridge the gap between human intent and digital structure.
But once that website exists, the AI should not test it by visually scanning a browser window and trying to click buttons. It should use a testing framework like Playwright or Selenium. That is Interaction.
We must not confuse the two. Using vision for interaction is like driving a car by pointing a camera at the speedometer instead of reading the sensor data directly. It adds latency, noise, and fragility where none should exist.
The Paradox of "High Frequencies" and The Latent Space
Those of us working with Large Multimodal Models (VLMs) know a dirty secret: AI vision is fundamentally different from human vision.
When a human looks at a screen, we perceive crisp edges and state changes instantly. An AI, however, "sees" by compressing an image into tokens and mapping them into a latent space. While a modern model can perfectly describe a sunset or a cat (low-frequency visual data), it often hallucinates on fine details (high-frequency data).
This leads to what I call the High-Frequency Paradox:
State Ambiguity: It struggles to distinguish between a "disabled" gray button and an "active" gray button. The semantic difference is massive (one works, one doesn't), but the visual difference in the latent space is negligible.
Text Degradation: It misreads small text in complex IDE menus or confuses similar-looking icons (like "Debug" vs. "Run").
Hallucinated Interactivity: It often hallucinates the state of a checkbox or assumes a static label is a clickable button because it appears in a position where buttons usually reside in its training data.
The more precise the UI interaction needs to be, the less reliable the "visual" agent becomes. We are asking a probabilistic engine to interact with a deterministic interface via a lossy visual channel. It is a recipe for fragility.
The Android Experiment: CLI vs. GUI
To test this hypothesis, I conducted a series of rigorous experiments developing Android applications entirely through AI agents. The goal was to see which modality allowed the agent to actually ship working code.
Attempt 1: The Visual Agent
I asked the agent to use Android Studio visually. I fed it screenshots of the IDE and asked it to perform standard tasks like "click the build button," "open the AndroidManifest.xml," or "fix the red error line."
Result: A frustratingly high failure rate. The agent would frequently hallucinate menu positions that had moved in recent updates. When an error popup appeared, the agent would often misinterpret the screenshot, missing the subtle "details" button that contained the actual stack trace. It tried to click UI elements that were merely decorative, wasting cycles in a loop of visual trial and error.
Attempt 2: The Blind Agent
I forced the AI to "close its eyes." I forbade it from using the GUI entirely. Instead, I gave it access to the Terminal and standard, deterministic tools:
-
adb(Android Debug Bridge) for installation, log retrieval, and testing. -
gradlew(The Gradle Wrapper) for building and dependency management. -
logcatfor real-time debugging. - Standard file system access for code editing.
Result: The success rate skyrocketed to nearly 100%.
Why? Because text is deterministic and self-correcting.
A CLI command like ./gradlew assembleDebug is absolute. It removes the ambiguity of "where is the button?". But more importantly, it solves the Versioning Problem.
If a GUI button moves or changes its icon in a new version of Android Studio, the visual agent fails. But if a CLI command is deprecated, the terminal returns a specific, text-based error:
Error: flag --old-flag is deprecated, use --new-flag instead.
The "blind" AI reads this error—which is in its native language (text)—understands the logic, updates its internal context, patches the command, and runs it again. It creates a perfect, closed feedback loop that visual agents simply cannot replicate. The error message becomes the instruction manual.
Addressing the Critics
There are two common counter-arguments to this approach.
1. "But Visual Agents are Universal!"
It is true that visual agents can interact with any software, even legacy apps without APIs. But this is a universality of mediocrity.
We have seen this movie before. Robotic Process Automation (RPA) tools spent the last 20 years trying to automate workflows by simulating clicks on screens. The lesson from two decades of RPA is clear: visual automation is fragile, expensive to maintain, and breaks constantly. AI visual agents inherit all these problems—plus the added risk of probabilistic hallucination. They are effectively Technical Debt upon arrival.
2. "But Vision Models are Improving!"
"Wait for GPT-5," they say. "Vision accuracy will be 99.9%."
This argument mistakes capability for architecture.
Even if an AI could read a screen with 100% pixel-perfect accuracy, using a massive visual transformer to read the text "Submit" on a button is an astonishing waste of compute compared to sending a POST request. It is like arguing that self-driving cars make trains obsolete. Trains are efficient because of the rails (constraints), not because of the driver. Similarly, text protocols provide the "rails" that make agents reliable and efficient.
The Interface Hierarchy of Reliability
When designing agentic workflows, we need to stop treating all interfaces as equal. As engineers, we should evaluate interfaces based on a hierarchy of reliability:
Native APIs/SDKs: Maximum reliability, minimum overhead. The agent speaks directly to the machine logic.
CLI Tools: Deterministic text I/O, excellent error reporting, self-correcting capabilities.
Structured Protocols: (JSON-RPC, GraphQL, REST). Explicit intent, no visual parsing required.
Accessibility APIs: (Apple's Accessibility Tree, Windows UI Automation). Uses the OS structure without needing pixel analysis.
DOM/HTML Parsing: For web apps (better than pixels, but prone to breakage).
Visual Interaction: The Last Resort. High compute cost (10x processing for vision vs text), high fragility, high latency.
Each step down this ladder represents a degradation in stability and an increase in "hallucination surface area."
The Inversion of Control
The next leap in software engineering will be Inversion of Control:
Legacy: We built GUIs to hide complex text tools and APIs from humans because they were too difficult to memorize.
Future: We will build APIs and Headless modes to expose those tools back to AI because GUIs are too ambiguous to interpret.
Imagine an IDE that has no window unless a human explicitly asks to "see" the code. The AI writes, compiles, tests, and deploys using only the compiler and the shell. It doesn't need syntax highlighting—it needs syntax correctness.
The economics will eventually dictate this shift. Visual agents consume significantly more compute for significantly less reliability. Enterprises will not pay that premium indefinitely.
Conclusion
The impulse to build AI that uses computers "like humans do" is understandable. It makes for great demos. But it is a trap. We are creating agents with superhuman text processing capabilities, then handicapping them with a visual channel designed for completely different cognitive architectures.
The best "interface" for an AI isn't a 4K monitor. It's a shell prompt, a robust API doc, and a deterministic environment.
The first platforms to ship native, headless agent protocols will own the next decade of developer tooling.
Let's stop trying to give them eyes. Let's give them direct system access.
What's your experience with visual vs programmatic agents? Share your experiments in the comments.
Top comments (0)