DEV Community

Cover image for Cutting Playwright MCP Runs from 35k Tokens to $0 Local Replay
Yash Kashyap
Yash Kashyap

Posted on Originally published at yashh2313.hashnode.dev

Cutting Playwright MCP Runs from 35k Tokens to $0 Local Replay

TimeoutError: locator('button.submit-btn') not found
=========================== logs ===========================
waiting for locator('button.submit-btn')
Enter fullscreen mode Exit fullscreen mode

If you've asked Claude Code, Cursor, or Antigravity to run end-to-end Playwright tests on a web app, you've probably watched it fail in two annoying ways. Either the agent reads your JSX files, guesses CSS selectors blind, and trips over CSS modules or client hydration. Or it connects to a live browser MCP that pulls a huge DOM dump or screenshot after every single click.

By step 12 of a normal checkout flow, you've burned 35,000 tokens, waited over a minute, watched /compact eat half your session context, and guaranteed that re-running the exact same regression test tomorrow costs another $0.85 in API tokens.

For open-ended UI exploration that cost makes sense. When you already know the login or checkout flow you want to verify, it's pointless. So I built auto-qa (MIT) to pull the LLM out of the click loop, run Playwright locally for $0, and compile passing runs into plain .spec.ts files.

Auto-QA 3-Stage Pipeline and Benchmarks

Moving the LLM out of the click loop

Instead of keeping an LLM attached on every action, auto-qa works either as a standalone SKILL.md CLI or a local stdio JSON-RPC 2.0 MCP server (scripts/mcp-server.js, exposing 7 auto_qa_* tools) around one simple flow:

First, scripts/discover.js opens the page in headless Chromium (Playwright 1.64+), blocks background Service Workers (serviceWorkers: 'block'), auto-dismisses blocking alert()/confirm()/beforeunload dialogs, and reads page.accessibility.snapshot() plus interactive nodes into roughly 1,200 tokens instead of dumping raw HTML.

Next, your agent reads that ~1.2k-token surface map once and writes a declarative plan.json ("Click Sign In", "Fill Email", "Assert Dashboard visible").

From there, scripts/executor.js executes the whole sequence locally in Playwright (~10.3s for 14 steps locally, $0.00 token cost on repeat runs), and scripts/compiler.js exports passing flows to standard @playwright/test .spec.ts files you can commit straight into CI.

The offsetParent bug that ate an afternoon

Writing a local JSON runner took a couple hours. Making it stop dropping buttons inside production checkout drawers took most of the real debugging.

Auto-QA Fixed Modal Isolation and Native Playwright .spec.ts Mapping

Like plenty of DOM crawlers, my first visibility filter checked el.offsetParent !== null. It looked clean on static forms, then completely skipped position: fixed checkout drawers, cookie banners, and floating sidebars because offsetParent returns null on fixed elements in standard browsers.

discover.js now checks computed style and getBoundingClientRect() instead, and executor.js explicitly looks for active modal overlays (div[role="dialog"]:not(.hidden), .modal:not(.hidden), [aria-modal="true"]:not(.hidden)) so locator resolution searches inside the open modal before falling back to the rest of the page:

const modalLocator = page.locator(
  'div[role="dialog"]:not(.hidden), .modal:not(.hidden), [aria-modal="true"]:not(.hidden)'
);
const hasActiveModal =
  (await modalLocator.count()) > 0 &&
  (await modalLocator.first().isVisible().catch(() => false));
const scope = hasActiveModal ? modalLocator.first() : page;
Enter fullscreen mode Exit fullscreen mode

Inside resolveElement(), actions pick role, label, placeholder, or attribute fallbacks based on whether the step intends to fill or click, so backdrop elements don't hijack clicks behind an active dialog.

What breaks when you point a crawler at third-party sites

Running a Playwright crawler against client sites you didn't write gets weird fast. Untrusted button text with brackets, quotes, or backslashes can break generated regexes, CSS attribute selectors, or exported TypeScript if you interpolate labels directly.

compiler.js and executor.js run sanitizeRegexPattern(), escapeRegex(), and escapeCssString() on every selector, restrict navigation strictly to http: and https: via assertValidHttpUrl() (blocking file:///, javascript:, data:, and Windows UNC paths), and jail MCP file writes inside your workspace with assertInsideDirectory(). npm test runs all 6 security and PII redaction checks (AU-01 to AU-06).

Running it locally

It isn't published on npm (npmjs.org/auto-qa doesn't exist; package.json uses auto-qa-engine), so grab it from git for now:

git clone https://github.com/yashh1323/auto-qa.git
cd auto-qa
npm install
npx playwright install chromium

npm run doctor
npm test

# Test the compiler offline right away
node scripts/cli.js compile examples/client-login-checkout.json -o tests/e2e/checkout.spec.ts
Enter fullscreen mode Exit fullscreen mode

I used this setup to run the 31 end-to-end login and editor tests (auto_qa_tests/) against Clio, my Next.js and Rust writing app.

Where this won't help

Skip this for two cases: full-screen <canvas> or WebGL apps (games, canvas editors, 3D views) where semantic DOM/ARIA locators are blind, and random exploratory fuzzing where you don't know what flow you're testing yet and actually want a turn-by-turn agent poking around.

Top comments (2)

Collapse
 
eliashivemind profile image
Nova Ops •

Moving the LLM out of the click loop for a flow you already know is the right split — paying 35k tokens and ~$0.85 again tomorrow to re-verify the same checkout is the part that never made sense. The detail I'd steal is reading the accessibility snapshot once into ~1,200 tokens and turning that into a plan the local executor can replay, instead of a fresh DOM dump after every click. The offsetParent bug rings true too: a visibility check that looks clean on static forms and then silently skips every position:fixed drawer is exactly the kind of failure that reads as "element not found" for an afternoon. We hit the same cost shape from the schema side — measuring our MCP server before adding a tool, because the standing per-turn bill is the one nobody notices. What happens when the UI drifts under a saved plan.json — does discover re-validate the plan before the executor runs it, or does the flow just fail and get re-discovered from scratch?

Collapse
 
gautamtalksdev profile image
Gautam K •

From 35k tokens to almost nothing by giving the agent stable selectors instead of letting it guess is a great result. Test IDs pay off twice, for people and for agents.