I started using AI to generate Playwright test scaffolding a few months ago. Feed it a user flow, whether it's a login form or a multi-step checkout, and you get back a working spec in under a minute that runs and mostly passes on the first try.
I asked a model to write a test for a password reset flow, and it built a spec that filled in the email field, clicked submit, and checked for a success message on screen. That passed every time I ran it, but it never checked whether an email actually went out, whether the link inside it worked, or whether that same link still worked after being used once. The page reported success no matter what the backend did, so the test just checked the page's opinion of itself.
Before I touched it, the spec looked like this:
await page.getByLabel('Email').fill(testUser.email);
await page.getByRole('button', { name: 'Send reset link' }).click();
await expect(page.getByText('Check your email')).toBeVisible();
All three lines passed, which is exactly the problem: none of them checked anything that actually mattered. Once I rewrote the assertions, it looked like this:
await page.getByLabel('Email').fill(testUser.email);
await page.getByRole('button', { name: 'Send reset link' }).click();
await expect(page.getByText('Check your email')).toBeVisible();
const resetEmail = await getLatestEmail(testUser.email);
const token = extractResetToken(resetEmail.body);
await page.goto(`/reset?token=${token}`);
await page.getByLabel('New password').fill('NewPassword123!');
await page.getByRole('button', { name: 'Reset password' }).click();
await expect(page).toHaveURL('/login');
await page.goto(`/reset?token=${token}`);
await expect(page.getByText('This link has expired')).toBeVisible();
The expired-token check at the end is what caught a real bug: the first version of the reset endpoint let a token be reused as many times as someone wanted, with no expiration after it was used once.
None of this makes AI bad at writing tests. It's good at the boilerplate, at generating locators and shaping a spec file enough that I'm editing instead of starting from nothing, but it can't decide which outcome the business actually depends on, because that context doesn't live in the DOM. A password reset link that never expires looks identical on screen to one that works the way it should, and the same gap shows up anywhere a UI can report success independent of what the backend actually did, checkout confirmations and invite emails included.
I still generate the first draft of most new specs with a model, but I rewrite the assertions myself, against what the feature is actually supposed to guarantee.
Jeff Thoensen is a Context-Driven QA Engineer focused on automation, API testing, and exploratory testing. Find more at jeffthoensen.com.
Top comments (1)
"the test just checked the page's opinion of itself" is the best one-line description of this failure I have seen. The pattern underneath: AI writes assertions at the same layer it acted on, so a UI action gets a UI assertion, and the actual state change (email sent, token burned, record written) never gets checked. My rule of thumb: every generated test has to cross one layer boundary - act in the UI, assert on the side effect. Yours does exactly that with getLatestEmail.
Two more checks that catch the same class before it lands: break the flow on purpose and watch the test - stub the email service to fail or drop the reset route, and if the spec stays green it was never a test. Cheap to automate as a one-time mutation run per generated suite. And generate from the spec, not from the code: when the model reads the implementation it faithfully asserts current behavior, bugs included; feed it the acceptance criteria plus a couple of real input/output examples instead, and a mismatch surfaces as a red instead of getting blessed.
The expired-token recheck is the part most people skip, glad you kept it.