If you write UI tests, you know that a single class name change ruins your whole pipeline. Things are shifting fast. Computer-Using Agents are leaving the demo stage and actually parsing desktop environments well enough to replace old RPA scripts. OpenAI and Anthropic are taking two totally different approaches to letting an LLM loose inside an OS.
Anthropic goes for direct visual and mouse control through API abstractions. It treats the screen as a canvas of elements read by vision models. OpenAI targets tighter integration loops. It optimizes for latency and multimodal grounding so the agent reacts to DOM updates and native UI states at the same time. On benchmarks like OSWorld, the gap between raw inference and specialized tool-use stands out. The model has to build a multi-step execution plan, handle popups, and recover when a button fails.
Old RPA vendors sold rigid coordinate clicking and xpath matching. Update the padding or font rendering on a legacy app, and the bot breaks. CUAs fix this by abstracting intent. Tell the agent to match invoices against a bank ledger, and it doesn't care if the ERP interface came out in 1998 or 2024. It reads the labels, parses the tables, and clicks dynamically.
Edge cases still hurt. We tested an agent on a broken CSV export from an old banking portal. Instead of fixing the payload, it got stuck in a retry loop on the download button. Economics are changing, but deterministic code still handles errors. You can't just point an agent at a messy workflow and walk away.
For internal tooling, you skip custom API wrappers for legacy systems without endpoints. Instead of spending weeks building bespoke connectors, you point a CUA at a staging environment and let it handle the UI. We looked at the trade-offs between OpenAI and Anthropic, plus how to avoid the fallback loops that drain compute budgets.
Check out the full computer-using agent breakdown on Kluvex to see how these models compare in production.
Top comments (0)