DEV Community: Jeff Thoensen

When the Mock Is Right and the API Isn't Anymore

Jeff Thoensen — Sat, 25 Jul 2026 17:10:18 +0000

A test suite built against a mock API passed every run for months. The real endpoint's response contract had changed in that time, but the mock hadn't, and nothing caught it because the tests only ever talked to the mock.

The mock returned a users array with name and email on each object. That was accurate when I built the mock, matching the real API's response at the time. Sometime after that, the backend team added a status field and split name into firstName and lastName. The integration hadn't been exercised against the real API since that change went out, so my test suite kept asserting against a version of the response that no longer existed anywhere but in my mock file.

The suite caught nothing, because nothing about it was ever wrong. It was internally consistent: the mock returned the old shape, and the tests checked for the old shape. It stayed green the whole time, testing against a contract that had already changed underneath it.

I added a separate, small set of contract tests that hit the real API directly, on a schedule instead of every commit, and assert that the fields and types the application depends on are still present, rather than asserting on the specific data returned. That suite is slower and talks to a real backend, which is exactly why it isn't run on every push. Its only job is catching the moment a mock and reality drift apart, which the fast mocked suite structurally cannot do.

The mock is still worth having. It's fast, and it isolates frontend logic from backend uptime, which is most of why teams reach for mocking a CRM integration in the first place. What changed is treating the mock as something that needs its own maintenance, tied to the real contract, instead of something written once during initial development and trusted indefinitely afterward.

The Test That Only Failed in CI, Never Locally

Jeff Thoensen — Sat, 25 Jul 2026 16:53:48 +0000

A test checking that a scheduled reminder appeared on the right day passed every time I ran it on my machine and failed every time it ran in CI, with nothing about the code or the test itself different between the two runs. The only thing that was different was the timezone the two machines were running in.

My machine was set to Eastern time, during the part of the year when Eastern runs four hours behind UTC. The CI runner ran in UTC. The test scheduled a reminder for 11 PM on a given day and checked that it showed up under that same calendar date. At 11 PM Eastern, the date comparison ran fine, because the reminder and the check were both being read in the same local time. Four hours ahead in UTC, 11 PM Eastern is already 3 AM the next day, so the reminder that should have shown up "today" was landing on tomorrow instead, and the test correctly failed against a real bug in how the date was being compared.

The bug lived in the feature: the reminder was meant to fire at one specific scheduled instant, not sometime across the user's local calendar day, so comparing it against "today" only makes sense once both sides are read in the same timezone. Comparing a reminder's timestamp against today without ever converting both to a common timezone first happens to work by accident whenever the two machines involved share the same offset from UTC, and stops working the moment they don't.

I fixed the comparison by converting both sides to UTC before comparing, since the reminder represents one fixed instant rather than a day tied to whoever happens to be looking at it. If the feature had actually meant "today on the user's own calendar," converting to UTC would have been the wrong fix, since two users in different timezones can disagree about what day a given instant falls on, and the right move there would have been converting both sides into the user's local timezone instead. Getting that distinction right mattered more than which timezone I ended up picking.

I also set CI to run in a timezone deliberately different from wherever tests get run manually, so a mismatch like this fails immediately instead of waiting for whichever engineer's laptop happens to be in a different zone than the pipeline.

The habit worth keeping from this: any test involving dates is also, whether anyone intended it or not, a test of timezone handling. Code that only works because two clocks happen to agree is running on coincidence, and the only way to find that out before a user does is making sure your test environment doesn't coincidentally match your dev machine's timezone.

Where Exploratory Testing Earns Its Keep

Jeff Thoensen — Sat, 18 Jul 2026 17:25:02 +0000

Happy paths should be automated, and that part isn't up for debate. A suite that runs the same core flows on every commit is what actually keeps a release stable, and there's no version of "we'll just check it manually before every deploy" that scales past a small team. Once that's in place, the question is what's actually left for a person to do once the obvious stuff already runs itself.

I keep a bulk-edit feature as the example I come back to. The automated suite covers selecting rows, editing a field, saving, and confirming the change. Green on every run, and it should stay that way. None of that is where I spend exploratory time, since it's already covered and covered well.

Where I actually spend time is the stuff nobody wrote an assertion for, because nobody thought to yet. I select a few hundred rows instead of a few, since that's closer to how someone treats a bulk-edit feature once they trust it. The save button spun and never resolved, with nothing in the UI to indicate anything had gone wrong. The backend had a request size limit that silently dropped the request past a few dozen rows.

That gap isn't a hole in the automated suite so much as the shape of what automation structurally can't reach on its own. A human has to imagine the scenario before anyone can write a test for it, and a scenario like selecting far more rows than the test data ever used isn't something you arrive at by making the existing suite more thorough. You arrive at it by using the feature the way someone under real conditions would, which usually means pushing past whatever amount the test data was written to cover.

The other place a person earns their keep is judgment calls automation can't make. A test can assert that a field shows an error message. It can't tell you the message is confusing, or that the error appears in a spot nobody would look, or that the flow technically works but takes four more clicks than it should. Those aren't failures a script can detect, since there's no clear pass or fail condition to assert on. Someone has to actually use the thing and notice.

I still treat the automated suite as the floor, not something exploratory testing is meant to replace or double-check. The floor is what keeps regressions from shipping. What a session is actually for is the edges nobody imagined yet, and the parts of the experience only a person can judge.

Jeff Thoensen is a Context-Driven QA Engineer focused on automation, API testing, and exploratory testing. Find more at jeffthoensen.com

My AI-Generated Tests Kept Passing for the Wrong Reason

Jeff Thoensen — Tue, 14 Jul 2026 20:46:16 +0000

I started using AI to generate Playwright test scaffolding a few months ago. Feed it a user flow, whether it's a login form or a multi-step checkout, and you get back a working spec in under a minute that runs and mostly passes on the first try.

I asked a model to write a test for a password reset flow, and it built a spec that filled in the email field, clicked submit, and checked for a success message on screen. That passed every time I ran it, but it never checked whether an email actually went out, whether the link inside it worked, or whether that same link still worked after being used once. The page reported success no matter what the backend did, so the test just checked the page's opinion of itself.

Before I touched it, the spec looked like this:

await page.getByLabel('Email').fill(testUser.email);
await page.getByRole('button', { name: 'Send reset link' }).click();
await expect(page.getByText('Check your email')).toBeVisible();

All three lines passed, which is exactly the problem: none of them checked anything that actually mattered. Once I rewrote the assertions, it looked like this:

await page.getByLabel('Email').fill(testUser.email);
await page.getByRole('button', { name: 'Send reset link' }).click();
await expect(page.getByText('Check your email')).toBeVisible();

const resetEmail = await getLatestEmail(testUser.email);
const token = extractResetToken(resetEmail.body);

await page.goto(`/reset?token=${token}`);
await page.getByLabel('New password').fill('NewPassword123!');
await page.getByRole('button', { name: 'Reset password' }).click();
await expect(page).toHaveURL('/login');

await page.goto(`/reset?token=${token}`);
await expect(page.getByText('This link has expired')).toBeVisible();

The expired-token check at the end is what caught a real bug: the first version of the reset endpoint let a token be reused as many times as someone wanted, with no expiration after it was used once.

None of this makes AI bad at writing tests. It's good at the boilerplate, at generating locators and shaping a spec file enough that I'm editing instead of starting from nothing, but it can't decide which outcome the business actually depends on, because that context doesn't live in the DOM. A password reset link that never expires looks identical on screen to one that works the way it should, and the same gap shows up anywhere a UI can report success independent of what the backend actually did, checkout confirmations and invite emails included.

I still generate the first draft of most new specs with a model, but I rewrite the assertions myself, against what the feature is actually supposed to guarantee.

Jeff Thoensen is a Context-Driven QA Engineer focused on automation, API testing, and exploratory testing. Find more at jeffthoensen.com.

Testing a LiveView App with Playwright: Fixing Navigation Timeouts

Jeff Thoensen — Wed, 27 May 2026 22:07:52 +0000

I was building a Playwright suite against a Phoenix LiveView app for the first time. Tests ran fine in isolation. Overnight, the full suite timed out across the board. Every failure was in a navigation.

What Playwright Waits For by Default

When you call page.goto() or trigger a navigation through a click, Playwright waits for the load event before moving on. That assumes a traditional request/response cycle: the browser makes a request, the server returns a full HTML document, the browser fires load when everything is done.

LiveView doesn't do that. Navigation happens over a persistent WebSocket connection. The URL changes, the page updates, but there's no new document and no load event in the way Playwright expects. So Playwright waits, hits the timeout, and the test fails.

The Fix: waitUntil commit

The waitUntil option controls what signal Playwright waits for before continuing. Setting it to 'commit' tells Playwright to return as soon as the server has responded and the navigation is committed, without waiting for a full page load cycle that isn't coming.

The change applies anywhere a navigation happens — goto and waitForURL both take the option:

// goto
await page.goto('/', { waitUntil: 'commit' });

// waitForURL
await page.waitForURL(/\/library\//, { waitUntil: 'commit' });

This was a 12-commit PR touching every navigation across the suite. Not a complicated change per file, but it had to be consistent. One nav without waitUntil: 'commit' is enough to hang the whole overnight run.

The Second Step: waitForLiveView

waitUntil: 'commit' gets you past the hang, but it returns early. LiveView still needs to establish its WebSocket connection and mount the view. Acting on the page before that happens produces flaky results.

The pattern is to follow every navigation with a waitForLiveView helper that waits for the socket to connect before returning control to the test. It was already in the suite — the missing piece was waitUntil: 'commit' on the line before it.

await page.goto('/', { waitUntil: 'commit' });
await waitForLiveView(page);

// or after a click-triggered navigation
await page.waitForURL(/\/library\//, { waitUntil: 'commit' });
await waitForLiveView(page);

There is one exception worth noting. When the navigation is a plain href link transition rather than a LiveView-driven one, waitForURL governs the transition on its own and waitForLiveView isn't needed. A comment in the diff flags exactly that case: // href link — waitForURL governs this transition, not waitForLiveView. Knowing which navigations go through LiveView and which don't matters when you're deciding where to apply the full pattern.

The actual helper in this suite does more than a basic selector wait. The environment has a brief WebSocket drop around the nightly deploy window, so there's a retry path built in:

export async function waitForLiveView(
  page: Page,
  { retryWithReload = false }: { retryWithReload?: boolean } = {}
): Promise<void> {
  if (!retryWithReload) {
    await page.locator(".phx-connected").waitFor();
    return;
  }
  try {
    await page.locator(".phx-connected").waitFor({ timeout: 10_000 });
  } catch (e) {
    if (!(e instanceof errors.TimeoutError)) throw e;
    await page.reload({ waitUntil: 'commit' });
    await page.locator(".phx-connected").waitFor({ timeout: 10_000 });
  }
}

The default path just waits for .phx-connected. Pass retryWithReload: true when the page is in a clean navigable state and a reload is safe — if the socket connection times out, the helper reloads and tries once more. Avoid it inside open modals, filled forms, or any state a reload would destroy.

The Broader Point

Playwright's defaults are reasonable for the apps they were designed around. When you add a new framework to your suite, those defaults are the first thing worth checking. The assumption baked into waitUntil: 'load' is that navigation means a new document. LiveView changes that contract, and the suite breaks until you adjust for it.

The tool doesn't know what kind of app it's running against. You do. The configuration has to reflect that. More on this kind of thing at jeffthoensen.com.

Jeff Thoensen is a Context-Driven QA Engineer focused on automation, API testing, and exploratory testing. Find more at jeffthoensen.com

Don't Automate Everything

Jeff Thoensen — Sat, 23 Aug 2025 16:45:57 +0000

I went heavy on Playwright early on. Wrote tests for every button and every form that I could. It looked solid until a small UI change broke a dozen of them because I’d used getByText in bad spots. The suite became useless for a while, and we were shipping code without it (so what was the point of all that work?).

Now I focus on automating only the things I don’t want to do manually: setup flows, core user paths, data checks. If I can’t explain why a test exists, I don’t write it. I also check UI mocks early so I’m not surprised when labels change or elements move. It saves time and keeps the suite stable.

I keep my suites small on purpose. Five reliable tests are better than fifty flaky ones, and flaky tests destroy trust in QA fast. I also treat tests like real code. Reviews, refactors, small commits. If you skip that part, the suite turns into a mess quickly.

I use Playwright every day, but I’m not automating to hit numbers. It’s there to save time and keep focus on users, not on constantly broken tests.

jeffthoensen.com

GitHub

LinkedIn

Medium

Substack

Indie Hackers

Hello QA!

Jeff Thoensen — Fri, 27 Jun 2025 14:23:46 +0000

I’ve been in QA for about 10 years, with a background in customer support that gave me a real sense of how bugs hit users.

I care about testing as a way to reduce risk and protect the user experience, not just checking boxes. These days I do a mix of exploratory testing and Playwright automation, but the goal’s the same: find the stuff that actually matters.

If you're into testing that goes beyond coverage reports, we’re probably on the same page.