Your AI agent writes Playwright tests that pass while your API returns 500

AI agents now write a lot of Playwright tests. You ask Claude Code or Copilot to "add a test for the checkout", or let an agent explore your app through Playwright MCP and turn what it did into a spec. Healer agents even fix failing tests for you.

Here is the kind of test they write:

import { test, expect } from "@playwright/test";

test("places an order", async ({ page }) => {
  await page.goto("/");
  await page.getByLabel("Name").fill("Ada");
  await page.getByRole("button", { name: "Place order" }).click();
  await expect(page.getByRole("status")).toContainText("Order placed");
});

It's a good test: role-based locators and a web-first assertion. It passes.

It also passes when the order API returns 500 and stores nothing, as long as the page still shows "Order placed". The agent checked what it could see. Nobody told it to check what the server did, and agents almost never add a network assertion on their own.

That matters more with agents than with people. A human who wrote the checkout code might remember the API can fail. An agent generating fifty tests from what's on screen won't, and a healer agent that "fixes" a red test by updating the expected text can make things worse. The suite goes green while the backend is broken.

The one-line fix

Change the import, in the tests the agent wrote or in the template it writes from:

import { test, expect } from "@vedhajs/playwright-test";

Nothing else changes. Same test, same broken app:

  ✓  1 1-playwright.spec.ts › places an order (import from @playwright/test)
  ✘  2 2-vedha.spec.ts › places an order (import from @vedhajs/playwright-test)

    Error: Vedha guard: the test's own checks passed, but the app reported a problem:
      - POST /api/orders returned 500

On a healthy app, both pass.

The failure message is also written for whoever reads it next, human or agent: what the app did, and how to accept it if it was expected. A healer agent gets a real reason instead of a green run.

What it checks

@vedhajs/playwright-test re-exports Playwright's test and expect with an automatic fixture ("guard mode") that watches the browser context while each test runs:

• Fails the test: any 5xx from your app's own site, or a write request (POST/PUT/PATCH/DELETE) that failed outright, even when every expect passed.
• Warns, without failing: a 4xx answer to a write, and uncaught page errors. Validation tests expect a 422 and third-party scripts throw, so these become annotations in the report instead of failures.
• Ignores: requests to other sites (analytics, CDNs).
• Attaches: the findings to the test result as vedha-guard.json.

When a finding is expected, opt out per test or per endpoint:

test.use({ vedhaGuard: "strict" }); // 4xx writes and page errors fail too
test.use({ vedhaGuard: { ignore: ["/api/flaky", /\/v2\//], hosts: ["api.other-domain.com"] } });
test.use({ vedhaGuard: false }); // off, for a test that expects a 500

It is still Playwright's runner, so retries, sharding, reporters and traces work as before.

Making agents use it

Put it where the agent will pick it up:

• Your project instructions (CLAUDE.md, AGENTS.md, Copilot instructions): "Import test and expect from @vedhajs/playwright-test, not @playwright/test."
• Existing specs: agents copy the patterns they find, so once your specs use the import, new ones tend to follow.
• A lint rule that bans importing from @playwright/test in spec files, if you want to enforce it.

When the agent drives the browser itself

Guard mode is for the tests an agent writes. When the agent is the tester, driving the browser live and reporting pass or fail, Vedha also ships an MCP server that checks every step the same way. In a small pilot on held-out tasks (one model, 3 trials per task), the agent missed 9 seeded bugs with Vedha's MCP server against 26 with Playwright MCP. That's a separate setup from guard mode, covered in an earlier post.

Limits

• A hand-written waitForResponse assertion catches this too. Guard mode is the safety net for the many tests, agent-written or not, that don't have one.
• In Vedha's benchmark, tests with typical text and URL checks caught 0 of 140 seeded defects with plain Playwright, and 60 of 140 with Vedha's page-health check: every defect where the server rejected or failed the write. Guard mode applies the same rules to whole tests, but it hasn't been benchmarked separately yet.
• It does not catch a write that returns 200 but stores nothing or stores the wrong value. Vedha's checked steps can read the data back, but that's opt-in and needs more than one line.

Try it

npm install -D @vedhajs/playwright-test

Change the import in one agent-written spec and run it. If a test that used to pass now fails, look at the request it names. Either the agent's test was hiding a bug, or it's an expected error to add to ignore.

Repo, demo and benchmark data: https://github.com/HMAKT99/VEDHA (Apache-2.0)

I'd like to hear what it finds in agent-written suites, and about any false alarm.