Traditional Selenium and RPA scripts shatter the moment a UI layout changes. Browser and computer-use agents promise something sturdier: give them a goal, and they work out the clicks themselves. But before you build one, you need to make a key design decision.
The loop every agent runs
Every browser agent follows the same perception-action loop: perceive the screen, reason about the goal, predict an action (click, type, scroll), execute it, then evaluate the new state. The biggest choice is the first step. How does the agent perceive the page?
Option A: DOM-based perception
The agent receives a simplified version of the page's HTML. Interactive elements (links, buttons, inputs) are stripped down and mapped to numerical IDs. The model then says "click element 14" instead of guessing coordinates.
Pros
- Precise: no pixel guessing
- Cheaper in tokens than sending images
- Works well with automation frameworks like Playwright
Cons
- Limited to the browser
- Messy or heavily scripted pages can be hard to simplify
The open-source browser-use library follows this approach for Python. It wraps an LLM around a Playwright instance so you can hand it a high-level goal like "log into this portal and download the latest invoice" in plain language.
Option B: Vision-based computer use
A multimodal model looks at a screenshot, calculates pixel coordinates, and fires synthetic mouse and keyboard events. Anthropic's computer use API is a well-known example. Because it works at the screen level, it is not tied to the browser.
Pros
- Controls the whole operating system, including native apps
- Works on legacy software and remote desktop-style environments
Cons
- Coordinate errors on dense layouts, such as tightly packed data tables
- Higher cost per step
How to choose
| Your target | Better fit |
|---|---|
| Dynamic web data, SaaS workflows | DOM-based |
| Legacy desktop software, Citrix | Vision-based |
| Browser-native, integrated experience | Watch tools like Google's Project Mariner |
Where both approaches fail
Neither approach is immune to the same production problems:
- Async loading: the model acts before a component finishes hydrating, and the click goes nowhere
- UI drift: layouts change in ways the model did not expect
- Hallucination loops: after a failed action, the model keeps inventing fixes instead of raising an error
Blind retries do not solve this. Add verification after every step and strict limits on steps and runtime.
Security is not optional
Whichever perception mode you pick, the agent ingests whatever is on the page. A page with hidden instructions can attempt prompt injection. Never run the agent on your host machine. Use ephemeral, isolated Docker containers with no internal network access.
Takeaway
Perception is only part of the story. Choose DOM-based when your world is the web and you want precision and lower cost. Choose vision-based when you need to reach beyond the browser. In both cases, reliability comes from the guardrails around the model: verification, limits and sandboxing.
Full article: Browser Agents: Build Reliable Web Automation
Top comments (0)