DEV Community: Jeff Thoensen

My AI-Generated Tests Kept Passing for the Wrong Reason

Jeff Thoensen — Tue, 14 Jul 2026 20:46:16 +0000

I started using AI to generate Playwright test scaffolding a few months ago. Feed it a user flow, whether it's a login form or a multi-step checkout, and you get back a working spec in under a minute that runs and mostly passes on the first try.

I asked a model to write a test for a password reset flow, and it built a spec that filled in the email field, clicked submit, and checked for a success message on screen. That passed every time I ran it, but it never checked whether an email actually went out, whether the link inside it worked, or whether that same link still worked after being used once. The page reported success no matter what the backend did, so the test just checked the page's opinion of itself.

Before I touched it, the spec looked like this:

await page.getByLabel('Email').fill(testUser.email);
await page.getByRole('button', { name: 'Send reset link' }).click();
await expect(page.getByText('Check your email')).toBeVisible();

All three lines passed, which is exactly the problem: none of them checked anything that actually mattered. Once I rewrote the assertions, it looked like this:

await page.getByLabel('Email').fill(testUser.email);
await page.getByRole('button', { name: 'Send reset link' }).click();
await expect(page.getByText('Check your email')).toBeVisible();

const resetEmail = await getLatestEmail(testUser.email);
const token = extractResetToken(resetEmail.body);

await page.goto(`/reset?token=${token}`);
await page.getByLabel('New password').fill('NewPassword123!');
await page.getByRole('button', { name: 'Reset password' }).click();
await expect(page).toHaveURL('/login');

await page.goto(`/reset?token=${token}`);
await expect(page.getByText('This link has expired')).toBeVisible();

The expired-token check at the end is what caught a real bug: the first version of the reset endpoint let a token be reused as many times as someone wanted, with no expiration after it was used once.

None of this makes AI bad at writing tests. It's good at the boilerplate, at generating locators and shaping a spec file enough that I'm editing instead of starting from nothing, but it can't decide which outcome the business actually depends on, because that context doesn't live in the DOM. A password reset link that never expires looks identical on screen to one that works the way it should, and the same gap shows up anywhere a UI can report success independent of what the backend actually did, checkout confirmations and invite emails included.

I still generate the first draft of most new specs with a model, but I rewrite the assertions myself, against what the feature is actually supposed to guarantee.

Jeff Thoensen is a Context-Driven QA Engineer focused on automation, API testing, and exploratory testing. Find more at jeffthoensen.com.

Testing a LiveView App with Playwright: Fixing Navigation Timeouts

Jeff Thoensen — Wed, 27 May 2026 22:07:52 +0000

I was building a Playwright suite against a Phoenix LiveView app for the first time. Tests ran fine in isolation. Overnight, the full suite timed out across the board. Every failure was in a navigation.

What Playwright Waits For by Default

When you call page.goto() or trigger a navigation through a click, Playwright waits for the load event before moving on. That assumes a traditional request/response cycle: the browser makes a request, the server returns a full HTML document, the browser fires load when everything is done.

LiveView doesn't do that. Navigation happens over a persistent WebSocket connection. The URL changes, the page updates, but there's no new document and no load event in the way Playwright expects. So Playwright waits, hits the timeout, and the test fails.

The Fix: waitUntil commit

The waitUntil option controls what signal Playwright waits for before continuing. Setting it to 'commit' tells Playwright to return as soon as the server has responded and the navigation is committed, without waiting for a full page load cycle that isn't coming.

The change applies anywhere a navigation happens — goto and waitForURL both take the option:

// goto
await page.goto('/', { waitUntil: 'commit' });

// waitForURL
await page.waitForURL(/\/library\//, { waitUntil: 'commit' });

This was a 12-commit PR touching every navigation across the suite. Not a complicated change per file, but it had to be consistent. One nav without waitUntil: 'commit' is enough to hang the whole overnight run.

The Second Step: waitForLiveView

waitUntil: 'commit' gets you past the hang, but it returns early. LiveView still needs to establish its WebSocket connection and mount the view. Acting on the page before that happens produces flaky results.

The pattern is to follow every navigation with a waitForLiveView helper that waits for the socket to connect before returning control to the test. It was already in the suite — the missing piece was waitUntil: 'commit' on the line before it.

await page.goto('/', { waitUntil: 'commit' });
await waitForLiveView(page);

// or after a click-triggered navigation
await page.waitForURL(/\/library\//, { waitUntil: 'commit' });
await waitForLiveView(page);

There is one exception worth noting. When the navigation is a plain href link transition rather than a LiveView-driven one, waitForURL governs the transition on its own and waitForLiveView isn't needed. A comment in the diff flags exactly that case: // href link — waitForURL governs this transition, not waitForLiveView. Knowing which navigations go through LiveView and which don't matters when you're deciding where to apply the full pattern.

The actual helper in this suite does more than a basic selector wait. The environment has a brief WebSocket drop around the nightly deploy window, so there's a retry path built in:

export async function waitForLiveView(
  page: Page,
  { retryWithReload = false }: { retryWithReload?: boolean } = {}
): Promise<void> {
  if (!retryWithReload) {
    await page.locator(".phx-connected").waitFor();
    return;
  }
  try {
    await page.locator(".phx-connected").waitFor({ timeout: 10_000 });
  } catch (e) {
    if (!(e instanceof errors.TimeoutError)) throw e;
    await page.reload({ waitUntil: 'commit' });
    await page.locator(".phx-connected").waitFor({ timeout: 10_000 });
  }
}

The default path just waits for .phx-connected. Pass retryWithReload: true when the page is in a clean navigable state and a reload is safe — if the socket connection times out, the helper reloads and tries once more. Avoid it inside open modals, filled forms, or any state a reload would destroy.

The Broader Point

Playwright's defaults are reasonable for the apps they were designed around. When you add a new framework to your suite, those defaults are the first thing worth checking. The assumption baked into waitUntil: 'load' is that navigation means a new document. LiveView changes that contract, and the suite breaks until you adjust for it.

The tool doesn't know what kind of app it's running against. You do. The configuration has to reflect that. More on this kind of thing at jeffthoensen.com.

Jeff Thoensen is a Context-Driven QA Engineer focused on automation, API testing, and exploratory testing. Find more at jeffthoensen.com

Don't Automate Everything

Jeff Thoensen — Sat, 23 Aug 2025 16:45:57 +0000

I went heavy on Playwright early on. Wrote tests for every button and every form that I could. It looked solid until a small UI change broke a dozen of them because I’d used getByText in bad spots. The suite became useless for a while, and we were shipping code without it (so what was the point of all that work?).

Now I focus on automating only the things I don’t want to do manually: setup flows, core user paths, data checks. If I can’t explain why a test exists, I don’t write it. I also check UI mocks early so I’m not surprised when labels change or elements move. It saves time and keeps the suite stable.

I keep my suites small on purpose. Five reliable tests are better than fifty flaky ones, and flaky tests destroy trust in QA fast. I also treat tests like real code. Reviews, refactors, small commits. If you skip that part, the suite turns into a mess quickly.

I use Playwright every day, but I’m not automating to hit numbers. It’s there to save time and keep focus on users, not on constantly broken tests.

jeffthoensen.com

GitHub

LinkedIn

Medium

Substack

Indie Hackers

Hello QA!

Jeff Thoensen — Fri, 27 Jun 2025 14:23:46 +0000

I’ve been in QA for about 10 years, with a background in customer support that gave me a real sense of how bugs hit users.

I care about testing as a way to reduce risk and protect the user experience, not just checking boxes. These days I do a mix of exploratory testing and Playwright automation, but the goal’s the same: find the stuff that actually matters.

If you're into testing that goes beyond coverage reports, we’re probably on the same page.