"Just automate it through the UI" sounds simple until you're the one maintaining that integration. When an API is missing, incomplete, or unsuitable for a task, an agent can often operate through the visible interface instead, the same DOM, accessibility tree, or screen pixels a person would use. That's a real capability. It's also a different engineering problem than an API integration, not an easier one.
The interface as integration layer
An API is built for software-to-software communication: defined actions, structured data, predictable error responses. Strip that away and an agent has to work with whatever's actually on screen:
- A web page's DOM and browser state
- An accessibility tree exposing control names, roles, and values
- A desktop app's window and UI controls
- A mobile UI hierarchy
- Screen pixels interpreted via OCR or computer vision
- Standard keyboard, pointer, touch, and gesture input
The core loop changes shape: observe state, pick one bounded action, perform it, verify the result, then pause or recover if the evidence is insufficient. "Can click" isn't the bar. Whether the system can recognize uncertainty and stop safely is the actual bar.
Six methods, ranked by how much structure you get
| Method | What it uses | Best fit | Main limitation |
|---|---|---|---|
| Browser automation | DOM, browser protocol, page state | Stable browser tools and web forms | Selectors and page states can change |
| Accessibility interaction | Roles, labels, values, control hierarchy | Accessible web/desktop/mobile UIs | Metadata may be missing or inaccurate |
| Screen and OCR interaction | Pixels, screenshots, visible text | Legacy apps, remote desktops | Visual interpretation is less deterministic |
| Keyboard/pointer/touch input | Standard user input | Cross-app and real-device tasks | Needs pairing with reliable observation |
| RPA | Rules, selectors, OCR, files | Narrow, repeatable legacy workflows | Exception handling grows over time |
| Hybrid API + UI | Approved APIs for some steps, UI for gaps | Partially integrated workflows | Requires careful state reconciliation |
The pattern worth internalizing: pick the most structured permitted method for the specific task, not the one with the broadest reach. Broad reach (screen + OCR) is also the least deterministic option, reach for it because you need to, not because it's the default.
Browser automation (W3C WebDriver, Playwright) is usually the strongest option for stable web apps, locate by semantic role and label, wait for load state, verify the confirmation state actually appeared. Still breaks on dynamic rendering, nested frames, A/B tests, and expiring sessions, explicit post-transition checks matter more than assuming a click worked.
Accessibility-tree interaction (WAI-ARIA, Accessible Name and Description Computation, plus platform frameworks like Microsoft UI Automation, Android UI Automator, Apple XCTest) gives you a semantic layer instead of raw pixels, when the app actually implements accessibility correctly. When it doesn't, you're back to guessing.
Screen and OCR interaction is the fallback for legacy apps and remote desktops with no exposed structure at all, and it's exactly as fragile as it sounds. Least deterministic, most universally applicable, use it when nothing else works, not as a default.
Authorization comes before method selection, not after
This is the part that's easy to skip past: no-API access doesn't override terms of service, platform rules, or the account owner's actual permissions. Before picking a method, confirm the task is something the agent (and the person deploying it) is actually authorized to do. A capability existing isn't the same as it being sanctioned.
Two hard lines, regardless of which method you're using:
- CAPTCHAs and MFA prompts are boundaries, not obstacles. Pause and hand control back to a person, or use a vendor-approved path. Never attempt to defeat them.
- Consequential actions need a human checkpoint. Sending a message, submitting a form, changing account settings, moving money, anything hard to reverse, should require confirmation before it executes, no matter how confident the agent's reasoning looks in the logs.
What actually needs verification
Real production systems working this way need instrumentation that API-only systems mostly don't:
- Screen-state capture at each step
- Action traces (what was observed, what was decided, what was sent)
- Replayable sessions for debugging failures after the fact
- Explicit post-action verification, not an assumption that the action succeeded
- Failure classification (expired session vs. layout change vs. genuinely blocked)
If you're building or evaluating a system like this, the useful question isn't "can it click the button." It's "when the button isn't where it expected, what happens next, and can a human see exactly what the agent tried."
We build Aiden around this exact problem for physical devices, HDMI-based screen capture plus USB HID input, no API dependency, verification and human confirmation built into the action loop rather than bolted on after. Repo: github.com/AidenAI-IO/aiden-firmware.
Curious what verification patterns others here have found actually catch failures early, screenshot diffing, structured re-query of the accessibility tree, something else?
Top comments (0)