DEV Community

Babatunde Hammed
Babatunde Hammed

Posted on

How to Review AI-Generated Playwright Tests (Before They Break in CI)

#ai #webdev #automation #playwright

I asked an LLM to write a Playwright test for a simple login flow. It passed the first time. Then I changed the button text from “Sign in” to “Continue” and re-ran it. The test failed.
The model had used a CSS selector tied to a class name and a fixed sleep. Classic brittle patterns — just generated faster than a junior engineer would have written them by hand.
This is becoming normal. Teams are using AI to draft Playwright tests, and the same failure modes from the pre-AI era are still showing up. The difference is volume and speed. If you don’t review the output, you’re shipping flaky tests into CI on purpose.
Here’s what to look for and how to fix it.

What AI Usually Gets Wrong

Most models default to patterns they see a lot in training data:

  • Position or CSS-based locators — page.locator('.btn-primary'), nth child, or long XPath paths. These break the moment the DOM shifts or a class is renamed.
  • Fixed sleeps — await page.waitForTimeout(2000) or similar. The model is trying to “wait for the page to settle” without knowing how Playwright’s auto-waiting works.
  • Weak or missing assertions — The test clicks through the flow but never checks the outcome the user cares about (e.g. “Order confirmed” is visible).
  • No resilience thinking — It optimizes for “this works on the page I see right now,” not “this will still work after a small redesign.”

The generated code often looks plausible and even passes once. That’s what makes it dangerous.

A Practical Review Checklist

Before you accept an AI-written test, run it through these questions:

  1. Locator strategy — Does it prefer getByRole, getByLabel, or getByText? Or is it using CSS / XPath / nth?
  2. Waiting — Any waitForTimeout, waitForSelector with a hard timeout, or other fixed delays? Prefer web-first assertions (expect(...).toBeVisible()) that retry automatically.
  3. What it actually asserts — Does the test check the user-visible outcome, or does it only confirm that clicks happened?
  4. Redesign resilience — If the button text changes, a wrapper div is added, or elements are reordered, does the test still pass? If you’re not sure, it probably doesn’t.

You don’t need to rewrite the whole test. Spot the weakest locator or wait and fix that first. One solid change often removes the flakiness.

Side-by-Side Example

Here’s a typical AI-generated version of a login test:

test('user can log in', async ({ page }) => {
await page.goto('https://example.com/login');
await page.waitForTimeout(2000);
await page.locator('#username').fill('test@example.com');
await page.locator('div.form-group:nth-child(2) input').fill('password123');
await page.locator('.btn-submit').click();
await page.waitForTimeout(3000);
await expect(page.locator('.success-message')).toBeVisible();
});

Problems: fixed sleeps, CSS and positional locators, and a weak assertion.
Revised version:
test('user can log in', async ({ page }) => {
await page.goto('https://example.com/login');
await page.getByLabel('Email').fill('test@example.com');
await page.getByLabel('Password').fill('password123');
await page.getByRole('button', { name: 'Sign in' }).click();
await expect(page.getByText('Welcome back')).toBeVisible();
});

Same flow, but the locators target what the user (and assistive tech) sees, and the assertion waits for the actual outcome. This version survives a class rename, a DOM reorder, and a button text change more reliably.

Practice

Reading checklists is useful. Doing the review under pressure is better. On LocatorLeague the Foundations track is free and lets you write and run Playwright locators against real pages with instant pass/fail feedback and a video of the run. The upcoming tracks will include reviewing AI-written tests so you can practice exactly this skill.

Closing

What’s the worst locator or wait an AI has written for you recently? Drop it in the comments — the patterns are usually familiar once you see a few of them.

This is roughly the same length and tone as the previous article. You can publish it as-is on Dev.to (or Hashnode) and cross-post. The link to LocatorLeague sits naturally after the practical content.

Top comments (0)