DEV Community

Babatunde Hammed
Babatunde Hammed

Posted on Edited on

How to Review AI-Generated Playwright Tests (Before They Break in CI)


I asked an LLM to write a Playwright test for a simple login flow. It passed the first time. Then I changed the button text from “Sign in” to “Continue” and re-ran it. The test failed.
The model had used a CSS selector tied to a class name and a fixed sleep. Classic brittle patterns — just generated faster than a junior engineer would have written them by hand.
This is becoming normal. Teams are using AI to draft Playwright tests, and the same failure modes from the pre-AI era are still showing up. The difference is volume and speed. If you don’t review the output, you’re shipping flaky tests into CI on purpose.
Here’s what to look for and how to fix it.

What AI Usually Gets Wrong

Most models default to patterns they see a lot in training data:

  • Position or CSS-based locators — page.locator('.btn-primary'), nth child, or long XPath paths. These break the moment the DOM shifts or a class is renamed.
  • Fixed sleeps — await page.waitForTimeout(2000) or similar. The model is trying to “wait for the page to settle” without knowing how Playwright’s auto-waiting works.
  • Weak or missing assertions — The test clicks through the flow but never checks the outcome the user cares about (e.g. “Order confirmed” is visible).
  • No resilience thinking — It optimizes for “this works on the page I see right now,” not “this will still work after a small redesign.”

The generated code often looks plausible and even passes once. That’s what makes it dangerous.

A Practical Review Checklist

Before you accept an AI-written test, run it through these questions:

  1. Locator strategy — Does it prefer getByRole, getByLabel, or getByText? Or is it using CSS / XPath / nth?
  2. Waiting — Any waitForTimeout, waitForSelector with a hard timeout, or other fixed delays? Prefer web-first assertions (expect(...).toBeVisible()) that retry automatically.
  3. What it actually asserts — Does the test check the user-visible outcome, or does it only confirm that clicks happened?
  4. Redesign resilience — If the button text changes, a wrapper div is added, or elements are reordered, does the test still pass? If you’re not sure, it probably doesn’t.

You don’t need to rewrite the whole test. Spot the weakest locator or wait and fix that first. One solid change often removes the flakiness.

Side-by-Side Example

Here’s a typical AI-generated version of a login test:

test('user can log in', async ({ page }) => {
await page.goto('https://example.com/login');
await page.waitForTimeout(2000);
await page.locator('#username').fill('test@example.com');
await page.locator('div.form-group:nth-child(2) input').fill('password123');
await page.locator('.btn-submit').click();
await page.waitForTimeout(3000);
await expect(page.locator('.success-message')).toBeVisible();
});

Problems: fixed sleeps, CSS and positional locators, and a weak assertion.
Revised version:
test('user can log in', async ({ page }) => {
await page.goto('https://example.com/login');
await page.getByLabel('Email').fill('test@example.com');
await page.getByLabel('Password').fill('password123');
await page.getByRole('button', { name: 'Sign in' }).click();
await expect(page.getByText('Welcome back')).toBeVisible();
});

The button locator still depends on its label, but that's the right trade-off. If the label changes, the failure is one clear line to update, not a flaky timeout.

Practice

Reading a checklist is one thing. Catching these mistakes in real code is another. Locator League has a free Foundations track where you write Playwright locators against real pages and get instant pass/fail feedback, plus a video of the run.

Challenges where you review and fix AI-written tests are coming next. If you want to be told when they open, follow along here.Locator League

Your turn

What's the worst locator or wait an AI has written for you recently? Drop it in the comments. The patterns are usually familiar once you see a few of them.

Top comments (0)