More teams now let an AI agent test their web app: "test the checkout", "check the signup flow". The agent drives a browser through Playwright, clicks through the flow, and reports back: all tests passed.
The problem is how the agent decides that. It looks at the page. If the page shows the success message it expected, the test passes. But a page can show success while the app is broken underneath: the API returned a 500, the server rejected the request, or it stored nothing at all. The test passes, and the bug ships.
I wanted to measure how often that happens, and whether a testing layer could catch these bugs without the agent (or a human tester) writing extra assertions. The result is Vedha, an open-source testing framework and MCP server built on Playwright.
A broken feature that passes its test
Here is an order form with a seeded bug: the order API returns 500 and stores nothing, but the page still shows "Order placed". The same test, with the same check (wait for "Order placed"), is run twice:
Playwright PASS "Order placed" is on the page
Vedha FAILED declared checks passed, but the page reported: POST /api/orders returned 500
Server 0 orders stored
You can run it yourself in about 3 seconds: npx tsx examples/demo/run.ts in the repo.
A careful tester would catch this in Playwright too, by asserting on the network response. Most tests don't, and agents almost never do.
What Vedha adds to Playwright
Vedha keeps what Playwright does best (auto-waiting, locators, the browser itself) and adds a verification layer to every test step:
-
Every step gets a result:
verified,failed,outcome_unknown,unverifiedorblocked. Never a silent pass. -
Actions that change data are checked. Mark a step as
mutatingand Vedha watches the requests it sent. A 5xx, a rejected request or a page error fails the step even when the on-screen check passed. Where the API follows REST conventions, it also reads the data back to confirm it was stored. -
No duplicate side effects from retries. If a click times out mid-submit, Vedha won't click "Place order" again and create a second order. It reports
outcome_unknownuntil a read-only check settles it. -
Agent explorations become regression tests. A verified flow can be exported as a plain
@playwright/testspec for CI. - Safe defaults for agents. The MCP server won't start without a domain allowlist, treats page text as untrusted data, and redacts secrets you pass to it.
The numbers, and their limits
Every number comes from raw trial files in the repo, and each held-out test set was written before any run on it.
AI agent testing (Playwright MCP vs Vedha MCP, same model, same held-out test tasks):
| Playwright MCP | Vedha MCP | |
|---|---|---|
| Seeded bugs missed (sets 2-4) | 26 | 9 |
| Healthy flows passed (set 4) | 18/18 | 17/18 |
| Median tokens, healthy flows (set 4) | 76,018 | 52,990 (-30%) |
| Median wall time, healthy flows (set 4) | 169 s | 176 s (+4%) |
Scripted tests, no model:
- With typical text/URL-only checks, Vedha caught 60 of 140 seeded bugs; every Playwright and Selenium baseline caught 0.
- A 1,120-trial audit after the last change had 0 false passes and 0 false fails.
- Median overhead: 1.05x hand-written Playwright.
The limits:
- The AI-agent results come from one free model, at 3 trials per task. That's a pilot, not a verdict.
- When an expert writes every check by hand, Vedha and Playwright tie. Vedha's lead is with typical checks and with agents.
- Read-back needs a REST-style read endpoint: the same path, or its singular/plural sibling. GraphQL, separate read endpoints and filtered or per-user collections aren't covered.
- Vedha passed one fewer healthy flow than Playwright MCP on set 4, and it uses more tool calls.
Try it
With Claude Code:
claude mcp add vedha -- npx -y @vedhajs/mcp --allow-domain localhost
Then ask: "test the checkout flow on localhost:3000".
Any MCP client takes the same command (npx -y @vedhajs/mcp --allow-domain localhost). There is also an agent skill (npx skills add HMAKT99/VEDHA), a CLI for scripted tests (npm i -D @vedhajs/cli), and an SDK that wraps an existing Playwright page.
Repo, benchmarks and raw data: https://github.com/HMAKT99/VEDHA (Apache-2.0)
I'd most like to hear about real apps where it gets things wrong: a false failure on a healthy flow, or a bug it should have caught.
Top comments (6)
Playwright is a strong execution layer, but agentic testing also needs an oracle: a way to decide what should be true after the agent acts. Stable fixtures, observable side effects, and assertions over business state keep browser automation from becoming a convincing demo with weak guarantees.
@alexshev , Agreed, the oracle is the hard part. Vedha's take: each step declares what should be true afterwards (expect), and for steps that change data it also checks the side effects on its own (the requests sent, their status, and a read-back of the stored record where the API allows it). The agent doesn't have to remember to assert on business state for the obvious failures. It doesn't replace domain assertions, but it catches the "convincing demo" case where the UI says success and the backend didn't do it. How do you handle the oracle in your setup?
I treat the oracle as a layered contract: UI state is only one signal. For any state-changing flow, the test asserts the response, a read-after-write via the narrowest trusted interface, and the expected audit/event record when one exists. The assertions are versioned with the workflow; when product semantics are unclear, the agent must surface a human review instead of inventing an oracle.
Tokenade, the tool I build, looks at agentic tests from the token side. Each step reloads the page and the transcript, and the price rises with the length of the run. A scripted test pays none of that.
@paul_irolla_6b5ae261d0224 , Good point on cost. Two things in Vedha help there: observations are compact deltas instead of a full page snapshot each step (about 30% fewer tokens than Playwright MCP in the pilot), and once the agent has verified a flow, vedha_promote turns it into a plain Playwright test. From then on you pay the scripted price in CI, not the agent price.
The silent 500 example is the clearest part. Checking a mutating step against what it actually stored beats trusting the success message on the page.