DEV Community

Daniel Kim
Daniel Kim

Posted on

An 87% Benchmark Score Just Embarrassed Every Big Lab's Browser Agent. The Winner Never Looks at the Screen.

Browser automation for AI agents

Every agent framework shipped in the last eighteen months eventually hits the same wall: the model can reason, plan, and call tools all day, but the moment the task requires "go to this website and do something," you have to decide how the agent actually touches a browser. That decision is no longer obvious. Four distinct approaches have hardened into production options, they solve the problem in genuinely different ways, and picking the wrong one for your use case shows up later as cost blowouts, flaky test suites, or agents that silently fail on the 5% of sites that don't play by the rules.

The four are Playwright driven through Microsoft's Playwright MCP server, the open-source Browser-Use library, Browserbase's Stagehand SDK, and Anthropic's Claude computer use tool. They get lumped together constantly under "AI browser agents," but they're not competing on the same axis. One skips vision entirely and hands the model a structured tree of the page. One rebuilt itself around a lower-level browser protocol to go faster. One is a commercial SDK designed to convert AI actions into deterministic scripts over time. And one is the fallback that works when none of the other three can — at a cost most teams underestimate until the bill arrives.

Why this comparison matters right now

Two things changed in 2026 that make this decision urgent instead of academic. First, coding agents like Claude Code, Cursor, and Codex started shipping with browser tools as a default capability, not an add-on — which means the choice of browser backend is now baked into how millions of agent sessions behave, not just something a scraping team debates in a design doc. Second, the tooling itself split architecturally. For most of 2025, "AI browser agent" meant one thing: wrap Playwright, take a screenshot, ask a vision model where to click. That's no longer true. Stagehand's v3 release removed its Playwright dependency and moved to a CDP-native architecture that talks to the browser directly over the Chrome DevTools Protocol, citing a 44% improvement on complex DOM interactions. Browser-Use made the identical architectural bet independently, also dropping Playwright for direct CDP control and citing the same 44% figure on shadow DOM and iframe handling. Two competing open-source projects converged on the same conclusion at the same time: the abstraction layer that made browser automation approachable for the last decade was becoming the bottleneck for the thing everyone actually wants to build now.

Meanwhile Playwright MCP took the opposite bet — stay on top of Playwright, but ditch pixels for structure — and Anthropic's computer use tool held its ground as the only approach that needs neither a DOM nor a browser protocol at all, because it's just looking at a screen the way a person would. Four camps, four different tradeoffs, and no vendor is going to tell you when its own approach is the wrong one.

It's worth being clear that this isn't the entire market — OpenAI ships its own computer-using agent behind its API, and Google has been building an equivalent capability into its own agent stack. Both follow the same vision-first pattern as Anthropic's tool rather than the accessibility-tree or CDP approach, which is part of why Browser-Use's benchmark result is notable: an open-source library beat agents built by all three labs, including the ones whose own vision-based tools it wasn't even using. This piece focuses on the four options a working developer is actually choosing between today — the two lab-run, general-purpose "operate a computer" tools converge on the same tradeoffs as Anthropic's, so the interesting decision space is the one laid out below.

Picture the concrete decision: you're building an agent that needs to check order status across a dozen vendor portals, none of which expose an API. Do you write Playwright scripts by hand and accept that every portal redesign breaks a selector? Do you hand the task to an agent that reasons its way through each portal fresh, accepting the token cost and occasional wrong turn? Do you invest in a caching layer that gets cheaper the more it runs? Or, if two of those twelve portals are legacy Java applets with no DOM at all, do you fall back to an agent that just looks at the screen? Most real systems end up mixing two or three of these, which is exactly why understanding what each one is actually good at — not just what the landing page claims — is worth the hour it takes to read this.

What each one actually does

Playwright MCP is a Model Context Protocol server, maintained by Microsoft under an Apache-2.0 license, that exposes browser control as a set of over 50 discrete tools — navigate, click, type, snapshot, manage tabs, mock network requests, inspect storage, generate PDFs, and more. The critical design choice is what it hands back to the model after each action: not a screenshot, but Playwright's accessibility tree, a structured, semantic, text-based representation of what's on the page. There's no vision model in the loop at all. Every interaction is deterministic in the sense that the tool call operates on a real DOM element, not a guessed pixel coordinate.

Browser-Use is an open-source Python library — MIT licensed, 109.7k GitHub stars, built by Magnus Müller and Gregor Žunič — that turns natural-language task descriptions into autonomous multi-step browser sessions. You describe the goal ("apply to this job posting," "extract pricing from these 40 competitor pages"), and the agent plans and executes the sequence of actions itself, including recovery when something on the page doesn't match expectations. It supports every major LLM provider, runs locally or against a hosted cloud version, and on the Odysseys leaderboard — 200 long-horizon web tasks — it posted an 87.4% average success rate, ranking first ahead of agents built by OpenAI, Google, and Anthropic themselves.

Stagehand, maintained by Browserbase, bills itself as "the SDK for browser agents" rather than a testing framework. It exposes Playwright-compatible methods (goto, click, locator) alongside three AI-native primitives — act, observe, and extract — that accept natural-language instructions. Its distinguishing idea is caching: AI-identified actions get progressively converted into deterministic replay scripts, so a workflow that costs real inference tokens the first time it runs gets cheaper and faster every time after, without needing an LLM call at all for the steps that already worked. It ships in TypeScript, Python, and Go, and it's designed to pair with Browserbase's managed, serverless browser infrastructure, though it doesn't strictly require it.

Claude's computer use tool is Anthropic's vision-based approach: the model receives screenshots, reasons about what it sees, and outputs coordinates and keystrokes the way a human would operate a mouse and keyboard. It requires no DOM access, no accessibility tree, no browser automation library at all — which is precisely the point. It works on canvas-rendered applications, legacy software with no accessible markup, and anything else where there's no structured hook to grab onto, because it's controlling the screen, not the page.

How they're actually built — and what changed

The through-line across all four tools in 2026 is a retreat from screenshots as the default interface between model and browser, everywhere except the one tool built explicitly for the case where screenshots are unavoidable. Playwright MCP never used them in the first place. Browser-Use and Stagehand both used to sit on top of Playwright and both ripped that dependency out in favor of talking to Chrome directly over CDP, independently arriving at the same headline number for the speedup on the hardest cases (shadow DOM, iframes, nested and dynamic content) — a strong signal that Playwright's abstraction, built primarily for deterministic testing, was adding real latency for agents that need to observe and re-plan on every step.

That convergence matters more than it looks. It means the "wrap Playwright, ask GPT where to click" pattern that defined the first wave of browser agents is now considered legacy by the two projects that built it. What replaced it isn't one thing — it's a fork between accessibility-tree-as-source-of-truth (Playwright MCP) and CDP-as-source-of-truth-with-AI-on-top (Browser-Use, Stagehand). Anthropic's computer use tool sits outside this entire conversation because it was never trying to solve the same problem; it's the tool for when there is no tree and no protocol to hook into, and its architecture hasn't needed to chase this trend because vision-based control was always going to be slower and less deterministic by design, not by implementation quality.

The practical difference between "accessibility tree" and "raw CDP" is worth spelling out, because it explains why two teams solving similar problems landed on different layers of the stack. The accessibility tree is a browser-native structure built for screen readers — every element gets a role, a label, and a position in a semantic hierarchy, and Playwright already knew how to read it because that's what its own getByRole and getByLabel locators are built on. Handing that tree to an LLM means the model reasons over roughly the same representation a blind user's screen reader would use, which is compact, well-understood, and already filtered down to what's meaningfully interactive. CDP, by contrast, is the low-level protocol Chrome itself exposes for anything that wants to instrument or control the browser — DOM mutations, network events, rendering state, input dispatch — with no built-in notion of "what's a button." Building directly on CDP means Browser-Use and Stagehand get to skip Playwright's overhead and decide for themselves what to extract and how to trim it for a model's context window, which is where Stagehand's "hybrid accessibility tree trimming" and Browser-Use's own DOM-state extraction come from — both ended up reimplementing a leaner version of the same tree Playwright MCP gets for free, just with tighter control over token budget and latency.

Why developers should actually care

Cost is the sharpest differentiator, and it isn't close. Claude's computer use tool can burn 50,000 to 200,000+ tokens on a single multi-step workflow because every action requires a fresh screenshot and a vision-model reasoning pass — at Anthropic's current published rates that's real money per task, not a rounding error, and it compounds fast at any kind of scale. Playwright MCP and Browser-Use, by only sending structured text (the accessibility tree, or targeted CDP state) instead of images, use a fraction of that per step. Stagehand goes further: its caching model means the first run of a workflow costs inference tokens, but every subsequent run of the same flow can skip the LLM call entirely and replay the deterministic script, which is a genuinely different cost curve than the other three — it degrades toward near-zero marginal cost for repeated tasks instead of staying flat.

Latency follows the same pattern for the same reason: screenshot-plus-vision-reasoning is inherently slower than reading a tree or issuing a CDP command, and removing Playwright's translation layer (Stagehand v3, Browser-Use) shaved meaningful time off the hardest interaction cases.

DX cuts differently. Playwright MCP is the most familiar to anyone who's written Playwright tests — the mental model transfers directly, and it slots into any MCP-compatible agent (Claude Code, Cursor, and others) with no code, just config. Browser-Use optimizes for "describe the goal, let the agent figure out the steps," which is the fastest path from zero to a working automation but the hardest to debug when it goes sideways in a way you didn't anticipate. Stagehand explicitly targets teams that want a middle ground — write deterministic code where you can, drop into natural language only where the page is unpredictable — which is a genuinely different engineering posture than "hand the whole task to an agent" or "write imperative test code."

Lock-in is where the vendor incentives show. Playwright MCP and Browser-Use are both fully open source with permissive licenses and no required hosted component — you can run either entirely on your own infrastructure forever. Stagehand is open source too, but its natural gravitational pull is toward Browserbase's paid, managed browser infrastructure, and the caching/optimization story gets meaningfully better the more of the stack you run on their platform. Claude's computer use tool has no self-hosted alternative at all; it's an Anthropic API capability, full stop.

Security deserves more attention than any vendor's landing page gives it. Playwright MCP's own documentation states outright that it is "not a security boundary" and points teams to MCP security best practices before deploying it — a direct admission that giving an LLM 50+ browser tools, including network mocking and storage access, is a real attack surface if the agent processes untrusted content. Computer use carries a starker version of the same risk: an agent that can see and click anything on a screen and has no concept of scope is explicitly recommended by Anthropic to run only in a sandboxed environment with limited permissions, never on a primary machine, because a model that misreads a malicious page can act on it exactly as if a human had.

Maintainability is the long-game question nobody markets well. Stagehand's replay-script caching is explicitly a maintainability feature — it converts "AI figured this out once" into "this is now a boring deterministic script," which is a real answer to the classic complaint that AI-driven automation is unpredictable in production. Browser-Use and Playwright MCP don't have an equivalent built-in mechanism; every run re-derives the plan, which is more resilient to site changes but also means you're paying the reasoning cost and accepting the non-determinism every single time, forever.

Practical use cases per option

Playwright MCP is the right default when you're wiring browser access into an existing coding agent (Claude Code, Cursor, any MCP client) for one-off or exploratory tasks — filling out a form during development, checking that a UI change rendered correctly, scraping a page while you're already in a terminal session. It's free, it's fast to set up, and because it's deterministic and text-based, it's also a legitimate fit for QA and test automation, not just agentic exploration — several teams are using it as a Playwright-compatible test runner that an LLM can drive interactively rather than only through hand-written scripts.

Browser-Use fits autonomous, long-horizon tasks where you want to describe an outcome and let the agent figure out the path: bulk job applications, competitive price monitoring across dozens of sites with different layouts, lead generation, or any workflow where writing per-site selectors doesn't scale. Its benchmark strength is specifically on multi-step tasks that require recovering from unexpected page states, which is the exact scenario hand-written scripts break on.

Stagehand plus Browserbase is built for production pipelines that run the same handful of workflows repeatedly at volume — checkout testing, data extraction jobs, monitoring flows — where the cost curve of "pay AI tokens once, replay a script forever after" actually pays off, and where you want managed, scaled browser infrastructure instead of running headless Chrome instances yourself.

Claude's computer use tool earns its cost premium specifically where the other three can't function at all: canvas-rendered apps (design tools, some dashboards, games), legacy desktop software, or any interface with no accessible DOM or CDP hook to grab. It's also the right tool for genuinely general-purpose "operate this computer" tasks that go beyond a browser tab — the same capability extends to any application on screen, which none of the browser-specific tools can claim.

Getting started with each

The integration effort differs enough to be a real factor in which one you reach for first. Playwright MCP is a single npx @playwright/mcp@latest away from running as a local MCP server, and any MCP-compatible client — Claude Code, Cursor, Claude Desktop — picks it up through a few lines of JSON config; there's no account, no API key beyond whatever LLM you're already using, and no code to write unless you want to customize its capabilities. Browser-Use is a pip install browser-use plus an API key for whichever model provider you're using, and you're writing a few lines of Python to hand it a task string — the library handles planning and execution internally, which is also why debugging it means reading agent traces rather than stepping through code you wrote. Stagehand is an npm install @browserbasehq/stagehand (or the Python/Go equivalent), and while it can run against a local Chromium instance for development, production use assumes a Browserbase API key and account — the free/local path exists but isn't where the SDK's caching and infrastructure story is optimized for. Computer use requires no browser library at all, just a beta flag on an Anthropic API call plus your own sandboxed execution environment (a VM or container with a virtual display) to actually give the model a screen to see and a mouse to move — which is also the most infrastructure any of the four demands before you write a single line of task logic.

What the marketing leaves out

Browser-Use's benchmark win is real but survivorship-biased: CAPTCHA solving is explicitly gated behind Browser Use Cloud's "stealth browsers," meaning the fully open-source, self-hosted path hits a wall on any site with real bot defenses — the 87.4% score was not run against a CAPTCHA gauntlet.

Stagehand's caching pitch undersells how much it depends on target pages staying stable. A replay script is only as good as the assumption that the site didn't change its layout since the script was recorded; the SDK's own value proposition is "self-healing" recovery when that assumption breaks, which is another way of saying the caching benefit evaporates on any site that changes frequently, and you're back to paying the AI-driven cost on every run.

Playwright MCP's "not a security boundary" disclosure is unusually honest for a README, but it's easy to skim past, and Microsoft doesn't put a security review checklist in front of you before you npm install fifty tool calls into an agent's hands.

Anthropic's computer use accuracy numbers are the most understated risk in this whole category: the tool started at roughly 15% on the OSWorld benchmark and has climbed to somewhere around 22%, against a human baseline of 72%. That's a meaningfully better trajectory than a year ago, but it is still a tool that gets a majority of complex real-world desktop tasks wrong, and the token cost for a single workflow can already run into real dollars before you factor in retries from those failures.

Comparison table

Dimension Playwright MCP Browser-Use Stagehand (+ Browserbase) Claude Computer Use
Maintainer Microsoft Independent (Müller/Žunič) Browserbase, Inc. Anthropic
License / cost Apache-2.0, free, self-hosted MIT, free; paid cloud for scale/CAPTCHA MIT SDK; pairs with paid managed infra API-metered, no self-host option
Model input Accessibility tree (text) CDP state + optional vision Hybrid: CDP + trimmed accessibility tree Screenshots (pixels)
Underlying protocol Playwright (on top of CDP) Direct CDP (Playwright removed) Direct CDP as of v3 (Playwright removed) None — OS-level screen/input
Determinism High (tool calls hit real DOM elements) Medium (agent re-plans each run) High after first run (script caching) Low (coordinate guessing from pixels)
Reported benchmark N/A (deterministic tool, not scored) 87.4% on Odysseys (#1 among labs) Not independently benchmarked in sources reviewed ~22% on OSWorld vs. 72% human
Token/step cost Low (text only) Low–medium Low first run, near-zero on cached replays High (50k–200k+ tokens/task)
Handles non-DOM UIs (canvas, desktop apps) No No No Yes
Explicit security caveat Yes ("not a security boundary") CAPTCHA/stealth gated to paid cloud Depends on target-page stability Yes (sandboxing strongly urged)
Best fit Dev-tool integration, QA, exploratory agent tasks Autonomous long-horizon multi-site tasks High-volume repeated production workflows Non-DOM / legacy / canvas apps

An independent read

None of these four tools is wrong, and the framing of "which one wins" is the wrong question — they occupy different points on a tradeoff between determinism, cost, and generality, and the market is still sorting out whether that will collapse into one dominant pattern or stay permanently fragmented by use case. What is worth being skeptical about is the marketing pattern common to all of them: every vendor benchmarks against the scenario it wins. Browser-Use's leaderboard win says nothing about CAPTCHA-heavy sites. Stagehand's cost story assumes stable target pages. Playwright MCP's speed and determinism say nothing about tasks that have no DOM. And computer use's generality is bought at a cost and accuracy level that would be disqualifying for any of the other three's use cases. The architectural convergence toward CDP-native, non-vision control across two independent open-source projects in the same year is the most objectively interesting signal here — it suggests the industry consensus is quietly settling on "avoid vision when you can" even as the highest-profile lab-backed tool (computer use) is vision-only by necessity, not by choice, because it's solving a problem the others explicitly can't touch.

There's also a structural point worth naming: three of these four tools are, directly or indirectly, funded by the same idea — that browser infrastructure and browser intelligence are separable businesses. Browserbase sells the infrastructure and gives away the SDK; the SDK's design choices (caching, hybrid trimming) happen to also be the choices that make its managed infrastructure more attractive at scale, which isn't a conspiracy, it's just how open-source-with-a-commercial-core projects tend to evolve. Playwright MCP has no such pull, because Microsoft isn't selling browser hosting — it's a pure devtool play, which is part of why it's the least commercially aggressive option and the easiest to trust for teams wary of lock-in. Browser-Use sits in between: fully open source at the core, with a hosted cloud tier that unlocks the CAPTCHA-handling most real-world scraping actually needs, meaning the free tier's benchmark numbers and the paid tier's real-world reliability are not quite the same product. None of this makes any of them dishonest — it just means "free and open source" and "free to actually use at production reliability" aren't always the same claim, and it's worth reading the pricing page for whichever one you pick before you architect around it.

Which reader profile should pick which option

If you're wiring browser access into an existing coding agent for development and testing work, start with Playwright MCP — it's free, familiar if you already know Playwright, and the accessibility-tree approach means you're not paying vision-model costs for routine interactions. If you need an agent to autonomously complete open-ended, multi-step tasks across sites you don't control the structure of, Browser-Use's benchmark performance and permissive license make it the strongest starting point, with the caveat that CAPTCHA-heavy targets will push you toward its paid cloud tier. If you're building a production pipeline that runs the same workflows repeatedly at real volume and you want the cost curve to improve over time rather than stay flat, Stagehand's caching model plus Browserbase's managed infrastructure is worth the commercial relationship, especially if your target pages are stable. And if your actual problem isn't a browser at all — it's a canvas app, a legacy desktop tool, or literally any UI with no DOM to hook into — computer use is the only one of the four that will work, and the cost and accuracy hit is the price of that generality, not a reason to avoid it when it's genuinely the only option on the table.

What's your experience been running these in production — has anyone gotten Stagehand's replay-script caching to hold up on sites that change their layout without warning, or does "self-healing" end up meaning you're paying the AI cost almost every time anyway?

Sources:

Top comments (0)