Key Takeaways
- Sai agent recorded a 73% success rate on the OSWorld 2.0 benchmark, which tests multi-step task completion inside a live operating system environment, not isolated language model evaluations.
- That 73% rate means roughly three-quarters of defined operational tasks, including file management, software configuration and data validation, could be handled without human intervention, shifting the cost of the remaining 27% to exception management rather than routine execution. Sai, a computer-use agent built by Simular, hit 73% on OSWorld 2.0 this week, beating GPT-5.6 Sol (62.57%, per OpenAI) and Opus 5 (70.57%, per Anthropic) at roughly two-thirds the cost of either. OSWorld 2.0 is one of the harder agent evaluations around: it drops agents inside a live operating system and asks them to complete multi-step tasks, not answer questions about them. That score puts autonomous OS-level task completion within reach for enterprise IT, back-office and data operations teams.
What OSWorld 2.0 Actually Tests
OSWorld 2.0 measures whether an agent can operate a real simulated desktop: configuring user accounts, installing software, generating reports across multiple applications, validating data across disparate systems. These are not reasoning puzzles. They are the kind of digital chores that consume significant staff time across IT operations, HR administration and back-office functions, work that rarely requires advanced cognitive judgment, just reliable sequential execution. Most benchmarks test language model performance in isolation; OSWorld 2.0 tests whether an agent can actually get things done in a dynamic, interactive setting.
The distinction matters for builders. An agent that scores well on language benchmarks can still fall apart when it has to click through a GUI, recover from an unexpected dialog box or chain 10 sequential actions without losing state. A 73% completion rate means Sai handled that complexity across a defined task set at a level that starts to look deployable for specific workloads, not just impressive in a lab. Simular’s open-source Agent S framework was the first system to beat the human baseline on the original OSWorld benchmark, in December 2025, scoring 72.6% against a 72.36% human baseline, so this builds on an existing track record rather than a first attempt. For more on how true AI agents differ from simpler automation workflows and why that gap matters when reading benchmark results, the OSWorld 2.0 tasks demand adaptive decision-making, not scripted execution.
Where the 27% Gets Expensive
The 27% failure rate is where deployment economics get complicated. Failed tasks tend to cluster around ambiguous instructions, situations outside the agent’s training distribution and edge cases requiring contextual judgment. Those are also frequently the tasks where a failure causes downstream damage: a misconfigured user account, a corrupted report, a skipped validation step.
The Business Case, Honestly Stated
The 73% score is a technical proof point, not a deployment guarantee. Simular is framing the result around cost as much as capability, citing roughly two-thirds the inference cost of GPT-5.6 Sol or Opus 5 for the same task set. For enterprise deployment, that cost gap matters as much as the accuracy gap: it determines whether running the automatable tasks at volume stays economical while exception handling for the 27% failure rate stays lean enough not to eat the savings.
Originally published at https://autonainews.com/sai-agent-hits-73-on-osworld-2-0-in-key-automation-benchmark/
Top comments (0)