DEV Community

FlakeLab
FlakeLab

Posted on Originally published at flakelab.dev

360 green runs, and one test that would pass with the server down

#playwright #ai #testing #typescript

I gave an AI a Playwright + TypeScript repo and ten test scenarios. It wrote 13 tests. I ran every test in the repo 20 times with retries off: 360 runs, 360 passed.

Then I read the tests. One of them would still pass if the server were down.

This article covers:

  • the project structure the AI worked in, and why each piece is there
  • how I measured the tests instead of trusting the green
  • where the AI got it right, and where it went wrong
  • a four-question checklist for reviewing AI-generated tests

Everything is in a public repo: flakelab-dev/playwright-structure. The AI's raw output is committed before any fix, so you can check every claim here.

TL;DR

  • Stable is not the same as correct. 360/360 proved the tests don't flake. It didn't prove they check the right thing.
  • The AI wrote good code and followed the structure. Its mistakes were about what to check, not how to write it.
  • It loosened an assertion to fit a bug, and it copied a gap from my own tests.
  • Without a spec, the AI turns current behavior into expected behavior. So do we, when we're in a hurry.
  • Seeded bugs told me more than 360 green runs.

The structure

Two public demo apps: SauceDemo for UI and Restful Booker for API.

config/environments.ts   URLs per environment, picked with TEST_ENV
src/
  pages/                 Page objects: locators and actions, no business assertions
  api/                   Typed API client + Zod schemas
  fixtures/              Wires pages, clients and auth together
  data/                  Test data builders
tests/
  ui/                    SauceDemo tests
  api/                   Restful Booker tests
playwright.config.ts     Two projects, "ui" and "api"; retries off
Enter fullscreen mode Exit fullscreen mode

Four decisions matter most.

1. Fixtures do the setup. Tests only declare what they need. A test that asks for loggedInPage gets a browser that is already logged in:

test('adding a product updates the cart badge', async ({ loggedInPage }) => {
  await loggedInPage.addToCart('Sauce Labs Backpack');
  await expect(loggedInPage.cartBadge).toHaveText('1');
});
Enter fullscreen mode Exit fullscreen mode

2. The API client checks every response. Methods like createBooking() validate the body with a Zod schema, and a failed call reports the URL, status and body:

async function jsonOrThrow(response: APIResponse): Promise<unknown> {
  if (!response.ok()) {
    throw new Error(`${response.url()} returned ${response.status()}: ${await response.text()}`);
  }
  return response.json();
}
Enter fullscreen mode Exit fullscreen mode

Methods ending in Raw return the response untouched, for tests about status codes.

3. Test data is unique per run. buildBooking() adds a random suffix to the last name, so parallel tests can't collide on a shared public API.

4. Retries are off. Retries hide flakiness, and measuring flakiness is the point. The CI workflow can repeat the whole suite from the Actions tab.

One more thing about structure: it is also what you give the AI. The AI doesn't read your mind, it reads your repo. Keep that in mind for later.

The experiment

  1. The prompt was written and committed before generating. (prompt.md)
  2. Fresh session: Claude Opus 5.5 in Claude Code, with only the repo and the prompt.
  3. The output was committed exactly as produced, before running or reviewing it. (raw output)
  4. Every test ran 20 times, retries off.
  5. I reviewed every test by hand against the code and the API docs.
  6. I ran the UI tests against users with deliberate bugs.

The prompt listed five UI scenarios (sorting, cart, checkout, form errors, logout) and five API scenarios (PUT, PATCH, filter, invalid payload, wrong credentials). It ended with:

Rules: no hard waits, no retries, each test independent.
Do not run the tests; only write them.
Enter fullscreen mode Exit fullscreen mode

Why "do not run"? If the AI runs its tests, it fixes them until they pass, and I lose the first draft. I wanted to measure that draft.

Result 1: 360 out of 360 

npx playwright test --repeat-each=20
Enter fullscreen mode Exit fullscreen mode

The 13 generated tests plus the 5 already in the repo, 20 times each: 360 runs, 360 passed.

That proves the tests are stable. It doesn't prove they are useful. A test that never fails might simply be unable to fail.

Result 2: seeding bugs

SauceDemo has demo users with deliberate bugs. The loggedInPage fixture reads the user from an environment variable, so I ran the same 7 UI tests as each user, 3 times each:

UI_USER=problem_user npx playwright test tests/ui/ai- --project=ui --repeat-each=3
Enter fullscreen mode Exit fullscreen mode
User Passed What failed
standard_user 21/21 Nothing (control)
problem_user 9/21 Sorting doesn't sort; the last name field ignores input, so checkout breaks
error_user 12/21 Sorting doesn't sort; checkout can't finish; a missing last name isn't reported
visual_user 18/21 The prices shown after sorting are not the prices shown before
performance_glitch_user 17/21 Not a bug in the app's logic: login is slow (see below)

I checked every failure: each one came from the seeded bug, not from a mistake in the test. The tests that stayed green did so because the bug wasn't on their path (logout works for every user).

This is the result I trust more than 360/360. The generated UI tests catch real bugs.

A side lesson on flakiness

performance_glitch_user takes over 5 seconds to log in. Look at what that did:

Test Passed
Sorting 0/3
Logout 2/3
Cart and checkout (5) 15/15

Same slowness, three different outcomes. The difference is the first step after login:

  • Sorting and logout start with an assertion: expect(loggedInPage.title).toHaveText('Products'). Assertions give up after 5 seconds by default.
  • Cart and checkout start with an action: addToCart(). Actions have no timeout of their own by default, so they keep waiting.

Logout passed 2 times out of 3. That's what a flaky test looks like the first time you meet it. With retries on, it would have been green and the slow login would have gone unnoticed.

Where the AI got it right

Before the mistakes, credit where it's due. Some of these I didn't ask for.

  • It followed the structure. It used the fixtures, page objects and client. It added new page objects and client methods using the existing Raw naming, and a data builder like the one already there.
  • It used unique data. The filter test searches for a first name like Flake-3f9a1c2e, so other bookings on the shared public API can't match.
  • It guarded the sorting test. If the products already loaded in ascending order, the test would pass even with a broken sort button. The AI checked for that first: 
// Guard: if the default order were already ascending, the test would prove nothing.
expect(pricesBefore.length).toBeGreaterThan(1);
expect(pricesBefore).not.toEqual(expected);
Enter fullscreen mode Exit fullscreen mode
  • It declared its assumptions. Its report listed four, like "Restful Booker returns 500 for an incomplete booking payload". All four are true today. None of them is documented.

Where the AI went wrong

1. It loosened an assertion to fit a bug

Scenario: "Creating a booking with an invalid payload is rejected."

Restful Booker answers a booking with missing fields with 500. A validation error should be 400; a 500 means the server failed. The AI knew this, and wrote:

// Restful Booker answers 500 here rather than 400; the test checks rejection, not the code.
expect(response.ok(), `expected rejection, got ${response.status()}`).toBe(false);
expect(response.status()).toBeGreaterThanOrEqual(400);
Enter fullscreen mode Exit fullscreen mode

This passes with 400, with 500, and with 503 Service Unavailable. If the service were down, this test would still pass. That's the test in the title.

The AI didn't hide the bug, since it left a comment. But it wrote the assertion so the bug fits, and the test can no longer fail for the reason that matters.

2. An error check that accepts any error

Scenario: "Authenticating with wrong credentials fails."

await expect(bookingClient.getToken(apiAdmin.username, 'wrong-password')).rejects.toThrow();
Enter fullscreen mode Exit fullscreen mode

toThrow() with no message accepts any error: a network failure, a timeout, a typo in the URL.

And here's why it passed. Restful Booker answers wrong credentials with 200 and {"reason":"Bad credentials"}. The client tried to read a token from that body, the schema check failed, and that error made the test pass. It passed because of a parsing error, not because the API rejected the login.

3. It copied my gap

The PUT, PATCH and filter tests create bookings and never delete them. That's about 60 abandoned bookings per 20-run session, on a public service that other people use.

The test I wrote by hand had the same gap. The AI followed the pattern it was given, including the bad part.

That's the lesson I keep from this experiment: AI multiplies your repo's conventions, good and bad. Fix the base before you generate on top of it.

4. Minor: duplication

The same locators (page title, item names, cart badge) are defined in three page objects, and four checkout tests repeat the same setup. They aren't bugs, so I left them for a later refactor.

The bigger problem: there was no spec

SauceDemo has no documentation. Every expected value in the UI tests (the error messages, the empty cart after checkout, the text after logout) came from what the app does today.

When I reviewed them, I got stuck on one question: if the app has a bug today, should the test adapt to it, or fail? And based on what?

Without a spec, these tests are characterization tests: they detect change, not correctness. That's still useful (the seeded bugs prove it), but it's worth naming. I marked all 7 UI tests "⚠️ Oracle": the expected behavior isn't backed by a specification.

The API has docs, so the review was clearer. PUT, PATCH and filter match the docs. The wrong-credentials response (200 with a reason) isn't described in the docs at all, and the test treats it as correct.

Final count for the 13 tests:

Verdict Tests
✅ Good as is 3
⚠️ Oracle: expected value not backed by a spec 8
⚠️ Weak: wouldn't catch the failure 1
❌ Wrong 1

The AI did what we all do in a hurry: it looked at current behavior and wrote it down as expected behavior. At least it listed its assumptions. Read that list: it's where the oracle problems are.

The fixes

Each fix is a separate commit on top of the AI's output, so you can compare before and after.

Invalid payload (bed0c5d): assert the exact status and record the bug.

test.info().annotations.push({
  type: 'known bug',
  description: 'Returns 500 instead of 400 for missing fields',
});
expect(response.status()).toBe(500);
Enter fullscreen mode Exit fullscreen mode

Now the test fails if the service is down (503), and it fails if the bug gets fixed (400), which is exactly when you want to update it. The annotation shows up in the HTML report, so the bug stays visible.

I considered test.fail() and rejected it: it treats any failure as expected, including an outage. Same problem as before.

Wrong credentials (c272114): the client now throws Authentication failed: <reason> when the API rejects the login, and the test asserts that exact message.

Cleanup (5b5ce0b): the bookingClient fixture deletes every booking the test created, after the test ends. To check it, a temporary test created a booking; after the run, a GET for that booking returned 404.

A checklist for AI-generated tests

Ask these four questions about every test:

  1. Oracle: where does each expected value come from: a spec, or what the app does today?
  2. Sensitivity: if the feature broke, would this test fail? Better than guessing: break it and see.
  3. Isolation: does it depend on data it doesn't control, or leave data behind?
  4. Structure: does it use the fixtures, page objects and client, or bypass them?

In this experiment, questions 1 and 2 found every real problem. Question 4 is the one the AI already handles well.

One tip from my own review: read the support code before judging a test. I flagged a missing status check in the API tests, and it was already there, in the client.

Try it yourself 

git clone https://github.com/flakelab-dev/playwright-structure
cd playwright-structure
npm ci
npx playwright install chromium
npm test
Enter fullscreen mode Exit fullscreen mode

Run npm run test:repeat to measure it yourself, or set UI_USER to one of the broken users and watch the tests catch the bugs.

How do you review AI-generated tests today? I'd like to hear what you check first.

I publish one measured experiment like this at a time. If you want the next one by email: flakelab.dev/subscribe

Top comments (0)