DEV Community

Cover image for Why agent-browser and Playwright Work So Well Together
Takafumi Endo | ROUTE06
Takafumi Endo | ROUTE06

Posted on

Why agent-browser and Playwright Work So Well Together

#ai #testing #playwright #agentbrowser

We recently added agent-browser to our UI evaluation workflow.

The motivation was simple.

A normal E2E test is great when we already know the path:

await page.getByRole("button", { name: "Settings" }).click();

await expect(
  page.getByRole("dialog", { name: "Settings" })
).toBeVisible();
Enter fullscreen mode Exit fullscreen mode

But sometimes I want to ask a different question:

Can a user find Settings and open it?
Enter fullscreen mode Exit fullscreen mode

I don't necessarily care which valid route the agent takes.

I care whether the product makes the goal achievable.

That is where agent-browser became useful.

The interesting part, though, was not simply letting an AI control a browser.

It was deciding where the agent's responsibility should end.

First: agent-browser and Computer Use are different paths

Before getting into the architecture, one distinction is important.

In our setup, agent-browser is part of the development-side evaluation harness.

Conceptually:

Codex / Claude
      ↓
agent-browser
      ↓
our application
      ↓
independent assertion
Enter fullscreen mode Exit fullscreen mode

agent-browser gives coding agents a browser-oriented interface. Its snapshot workflow exposes an accessibility representation with element references, allowing the agent to inspect the current UI, act, inspect again, and continue toward a goal.

OpenAI Computer Use is a separate execution path.

The current ChatGPT desktop app includes Chat and Work under ChatGPT, alongside Codex. Work can use local files and desktop applications with permission.

When I refer to Computer Use in this article, however, I mean the capability exposed to Codex for interacting with desktop applications.

For native application evaluation, where available, our path is closer to:

ChatGPT desktop app
        ↓
      Codex
        ↓
   Computer Use
        ↓
native application
        ↓
independent verification
Enter fullscreen mode Exit fullscreen mode

We are not handing an agent-browser session over to Computer Use.

We are also not claiming that agent-browser is our native desktop automation engine.

And Computer Use should not be read as "the Office editing API." For example, Excel also has a dedicated ChatGPT for Excel add-in workflow; Computer Use may help Codex reach that environment, but it is not the editing model itself.

These are different interaction mechanisms.

The common principle is only this:

The system performing the action does not have to be the system deciding whether the result is correct.

The rest of this article focuses mainly on the agent-browser side.

Why agent-browser fits exploratory QA

agent-browser works well when the task is expressed as an intent rather than a click script.

Instead of:

Click button A.
Then click menu item B.
Then expect dialog C.
Enter fullscreen mode Exit fullscreen mode

we can give the agent:

Open Settings.
Enter fullscreen mode Exit fullscreen mode

Or:

Find the customer and open the information relevant to their latest issue.
Enter fullscreen mode Exit fullscreen mode

The agent can inspect the current accessible UI through agent-browser and decide how to proceed.

That is useful for flows where we care about whether a goal is discoverable and achievable, rather than whether one exact sequence of selectors still works.

I think of this as exploratory QA.

Not random exploration.

Not a replacement for our normal E2E suite.

A separate lane for asking higher-level questions about the product.

The agent does not grade its own homework

This was the most important design decision.

Suppose the task is:

Open the Settings dialog.
Enter fullscreen mode Exit fullscreen mode

The agent uses agent-browser and eventually reports something equivalent to:

{
  "status": "completed"
}
Enter fullscreen mode Exit fullscreen mode

We do not treat that as a passing test.

After the agent finishes, Playwright checks the resulting application state independently.

Conceptually:

agent:
"completed"

    AND

Playwright:
Settings dialog is visible

    ↓

PASS
Enter fullscreen mode Exit fullscreen mode

The assertion can be very small:

await expect(
  page.getByRole("dialog", { name: "Settings" })
).toBeVisible();
Enter fullscreen mode Exit fullscreen mode

The two signals mean different things.

The agent says:

I believe I completed the goal.

The assertion says:

The application reached the state we care about.

That separation is what makes the combination useful.

Playwright gives the agent more freedom

Initially I thought of Playwright as a safety net around agent-browser.

I now think the opposite framing is more useful:

An independent oracle lets the agent be more exploratory.

The agent using agent-browser can focus on:

Can I accomplish the user's goal?
Enter fullscreen mode Exit fullscreen mode

while Playwright focuses on:

Did the application actually reach the expected state?
Enter fullscreen mode Exit fullscreen mode

The architecture is roughly:

          goal-oriented
           exploration
               │
               ▼
      ┌─────────────────┐
      │ Codex / Claude  │
      │        +        │
      │  agent-browser  │
      └────────┬────────┘
               │
               ▼
          application
               │
               ▼
      ┌─────────────────┐
      │   Playwright    │
      │    assertion    │
      └────────┬────────┘
               │
               ▼
           pass / fail
Enter fullscreen mode Exit fullscreen mode

This is why I don't see agent-browser and Playwright as competitors.

They are useful precisely because they can own different responsibilities.

This pattern is not new

I don't think the individual ideas here are novel.

The broader agent-evaluation ecosystem is already converging on similar principles.

Anthropic describes agent evaluation in terms of tasks, outcomes, and graders, and recommends code-based graders where possible instead of over-constraining the exact trajectory an agent must follow.

WebArena has used a similar shape for years: an agent receives a natural-language web task, while evaluation checks whether the task was functionally completed.

Playwright itself now has Test Agents. A planner explores the application, a generator turns plans into executable tests, and a healer works on failing tests.

There are also projects much closer to this exact browser-QA pattern.

The public qa-skills repository describes an agentic browser-testing workflow where the browser agent receives a natural-language goal, an external oracle determines success, and stable flows can graduate to scripted Playwright tests.

SightCI similarly documents AI exploration followed by promotion of selected runs into Playwright specs.

So the interesting question is not:

Did we invent agentic browser testing?

We didn't.

The useful question for us became:

Where should the boundary sit once this pattern meets a real application with UI, API, authentication, and native document workflows?

That is where our implementation became more interesting.

A real bug changed our original rule

Our first mental model was straightforward:

agent-browser-assisted exploration finds a failure
                    ↓
             we understand it
                    ↓
        add a Playwright regression test
Enter fullscreen mode Exit fullscreen mode

Then we encountered a failure in a real authenticated environment.

A cloud instruction was rejected.

After comparing API behavior under the same authentication context, we traced the issue to an organization-resolution mismatch.

The failure was visible through the product.

But the contract that actually broke was not a UI contract.

Adding another browser regression would therefore have put the check too far away from the failure.

Instead:

real environment exposes failure
            ↓
compare API behavior
            ↓
identify organization-resolution mismatch
            ↓
fix implementation
            ↓
add model-free Hosted E2E coverage
Enter fullscreen mode Exit fullscreen mode

That changed the rule.

Not:

Every problem discovered through exploration should become a Playwright test.

But:

Move an understood failure to the lowest-cost assertion-based layer that still reproduces the contract that failed.

That last part matters.

I don't mean "always turn an integration failure into a tiny unit test."

Some failures only exist across an integration boundary.

The regression still needs to reproduce the contract that actually broke.

Promote failures to the right layer

In practice, that gives us a rule of thumb like this:

Failure Likely regression layer
UI navigation, visibility, dialogs Playwright
Authentication, organization resolution, API contracts API / Hosted E2E
Word, Excel, or PowerPoint output Artifact-level verification

A UI failure may graduate to:

agent exploration
      ↓
UI root cause
      ↓
Playwright regression
Enter fullscreen mode Exit fullscreen mode

An authentication or backend failure may graduate to:

failure discovered
      ↓
API/auth root cause
      ↓
Hosted E2E / API regression
Enter fullscreen mode Exit fullscreen mode

And a native document workflow can use a different oracle:

desktop interaction
      ↓
edit a safe working copy
      ↓
save
      ↓
artifact / structure / reopen verification
Enter fullscreen mode Exit fullscreen mode

The execution mechanism is secondary.

The important question is:

Where can we observe the failed contract most directly and cheaply without losing the behavior that matters?

Known failures should leave the agent loop

This leads to the part of the design I like most.

Imagine an agent using agent-browser discovers a navigation problem today.

We investigate it.

We identify the contract that failed.

We add an assertion-based regression at the correct layer.

What should happen tomorrow?

Ideally, that exact failure should no longer require an AI agent.

AI exploration
      ↓
new failure
      ↓
root cause becomes known
      ↓
assertion-based regression
      ↓
future CI catches it without AI
Enter fullscreen mode Exit fullscreen mode

That means the goal is not to make more and more of the regression suite depend on AI.

It is almost the opposite.

Use the agent where uncertainty still exists.

Once the failure becomes knowledge, compile that knowledge into a cheaper check.

AI should have to discover a bug once. It should not have to rediscover the same bug on every pull request.

Restrict the capability, not only the prompt

There is another practical issue with giving an agent a browser.

A prompt can say:

Do not upload files.
Do not download files.
Do not leave the test environment.
Do not execute arbitrary JavaScript.
Enter fullscreen mode Exit fullscreen mode

But those are instructions to the model.

Where possible, we also want the environment to enforce the boundary.

agent-browser provides controls such as domain restrictions, content boundaries, action policies, confirmations, and output limits.

One detail matters:

Those security controls are opt-in.

A plain agent-browser session should not be assumed to be restricted automatically.

Our evaluation harness therefore treats restrictions as part of the harness configuration rather than relying only on the prompt.

The principle is:

Prompt boundaries are behavioral. Tool boundaries are architectural.

We also avoid treating one action-policy file as a complete security boundary.

At the time of writing, there is an open agent-browser issue describing a mismatch between documented action-policy categories and how individual actions are matched.

So effective permissions should be verified, not merely inferred from policy names.

For us, tool policy is only one layer alongside ephemeral execution, secret minimization, bounded runtime, and disposable browser state.

Pass evidence, not magical state

Another design question is what happens after an exploratory run fails.

One option would be to keep the browser alive and pass the same mutable session through several agents.

Sometimes that is useful.

But I don't want a live session to become the primary interface between evaluation stages.

Our current harness already records structured evidence such as:

scenario
scenario version
agent
target build
prompt hash
latency
coverage
failure domain
violation detail
cleanup state
Enter fullscreen mode Exit fullscreen mode

The direction I want to take this is a richer diagnostic bundle:

final URL
failed step
accessibility snapshot
screenshot
console errors
trace
Enter fullscreen mode Exit fullscreen mode

Then the flow becomes:

agent exploration through agent-browser
          ↓
inspectable evidence
          ↓
developer / Codex diagnosis
          ↓
regression candidate
Enter fullscreen mode Exit fullscreen mode

This is not an argument against traces, storage state, network logs, or screenshots.

Those are exactly the kinds of evidence we may want.

The distinction is between preserving evidence about state and making a hidden, mutable browser session the contract between components.

Evidence is easier to inspect, store, compare, and review.

What we have today

This is not a fully autonomous QA pipeline.

Today, the important pieces are:

✓ agent-browser exploration lane
✓ independent Codex / Claude evaluations
✓ restricted browser capabilities
✓ Playwright assertions outside the agent
✓ structured evaluation evidence
✓ assertion-based tests remain the release source of truth
Enter fullscreen mode Exit fullscreen mode

What we do not yet have is the full promotion loop:

□ complete screenshot / trace diagnostic bundles
□ automatic regression-candidate generation
□ evidence → regression linkage
□ reviewed promotion records
Enter fullscreen mode Exit fullscreen mode

So I would describe the current system as:

A shadow exploration layer around assertion-based QA, not an autonomous replacement for it.

That is intentional.

Final thought

I started this work because agent-browser looked like a good way to let coding agents interact with our application more like a user.

It is.

But the interesting lesson was not "AI can click buttons now."

The useful architecture emerged from deciding what the agent should not own.

An agent using agent-browser is good at asking:

Can I achieve this user goal from the UI in front of me?

Playwright, API checks, and artifact verification are good at asking:

Is this contract actually satisfied?

Those ideas are not new individually.

What surprised me was how well they fit together once we applied them across a real application instead of only a browser benchmark.

Let the agent explore what you don't know yet.

Let assertion-based tests remember what you already learned.

And once a failure becomes understood, move it out of the expensive exploratory loop and into the cheapest test that still protects the contract.

AI should have to discover a bug once.

It should not have to rediscover the same bug on every pull request.

Top comments (2)

Collapse
 
raknaos profile image
Raknaos •

The org-resolution mismatch is the detail that makes this worth more than the usual agent-QA writeup: the failure was only visible through the product, but the broken contract lived in the API, and a browser regression would have pinned the check to the wrong layer. I've made the mirror mistake — turning an API-level bug into a UI test because that's where I first saw it, then wondering why the test kept flaking while the real contract stayed untested.

One question on the promotion step: once a failure moves from agent exploration to an assertion-based layer, does the exploration run retire, or do you keep it as a cheap scheduled probe? I keep going back and forth on whether the agent lane should replay known failures — not to re-verify the fix, but to catch the regression of the regression, where the assertion itself rots after a refactor and nobody re-walks that path manually.

Collapse
 
gyu07 profile image
Takafumi Endo | ROUTE06 •

I wouldn’t retire the exploration run completely.

With AI-driven development, features get added, reshaped, and retired so quickly that even deciding when an E2E test is worth formalizing has become surprisingly difficult. We can delegate a lot of E2E maintenance to AI now, but it’s still far from perfect. I regularly see the same thing you described: even after a failure is promoted into the right assertion-based layer, a different bug appears around it, or the assertion itself gradually stops representing the real product path after a refactor.

So for a known failure, I’d let the deterministic test own the actual contract — in this case, the API-level invariant — rather than having the agent repeatedly re-verify the same fix.

But I’d still keep the agent lane as a cheap scheduled probe, with some intentional variance. The point isn’t to replay the exact regression test; it’s to keep re-walking that area of the product from slightly different angles and catch adjacent failures, or the “regression of the regression” you mentioned when the deterministic test itself has gone stale.

That’s one of the roles I currently find agent-browser useful for: letting Claude Code or Codex periodically exercise those flows. I think computer-use agents can play a similar role as well. I’m still figuring out the right frequency and cost boundary, though.