Most AI agents use software the way a human would: by looking at the screen.
They load the whole page, parse the UI, hunt for the right button, and guess what happens when they click it.
That works in a demo. It breaks in production.
One redesign, one renamed button, one A/B test — and the agent is lost. It can't tell a checkout button from a cancel button when both are just rectangles on a screen.
Screen-scraping makes agents:
→ Slow — every action waits on rendering and parsing the page
→ Expensive — each step burns tokens on UI noise instead of the task
→ Brittle — any UI change silently breaks the whole workflow
The agent isn't the problem. The interface is.
Software was built to be looked at. Agents need software built to be used.
What's the most fragile agent workflow you've seen break on a UI change?
Top comments (0)