Make Playwright Email Tests Less Flaky

I like email verification tests when they prove one narrow thing: a real user can finish signup without a hidden messaging bug. I do not like them when they become a slot machine. If your Playwright suite sometimes sees the email, sometimes grabs the wrong one, and sometimes times out for no obvious reason, the test is usually missing structure rather than missing retries.

This is the workflow I keep coming back to for Playwright and Automation teams. It is not fancy, but it is dependable and a bit easier to explain during failure review.

Why these tests fail in different ways

Most flaky email tests fail for one of three reasons:

• two runs read from the same inbox
• the test matches the first email instead of the correct email
• the failure leaves almost no evidence behind

That sounds simple, yet it creates very different symptoms. One build looks green locally and red in CI. Another run passes but verifies the wrong account. A third one times out and gives you zero clue where the delay realy happened.

Before changing the locator or the timeout again, I try to make the test answer a boring question first: "what exact inbox and message was this run supposed to use?"

1. Give each run its own inbox identity

Shared inboxes are the fastest way to create noisy failures. If your suite points every signup test at the same temp mailbox, you are testing message arrival and accidental cross-talk at the same time. That is not a fair test.

I prefer an address pattern that carries run identity inside it, even if the provider is just a temporary inbox used for QA:

const runId = `${Date.now()}-${test.info().parallelIndex}`;
const email = `signup-${runId}@example.test`;

The exact address format does not matter much. What matters is that the run id appears in the inbox name, your test logs, and ideally the application-side metadata too. The same idea shows up in naming inboxes per test run: once every run has a distinct identity, the wierd failures get much less mysterious.

If your team uses a temp mail so service or a temp mailbox provider for staging checks, keep it scoped to one test run and expire it quickly. I have also seen people type temp org mail into setup notes when they mean "some throwaway inbox here"; that is fine as shorthand, but the test itself should still record the actual mailbox it created.

2. Wait for the right email, not any email

The next problem is matching. A lot of tests poll an inbox and click the first message with "Verify" in the subject. That is a brittle shortcut. In busy environments, that approach drifts fast.

Instead, wait on a small contract:

• expected recipient
• expected subject pattern
• expected creation window
• expected text unique to this flow

That contract can stay small and still be strong. For example, if the UI shows the email address back to the user, assert against that exact value before you ever open the inbox. Then look for a message created after the signup action and containing the app or tenant name. This avoids a bunch of fake greens.

Here is the shape I use:

await expect(page.getByText(email)).toBeVisible();

const message = await inbox.waitForMessage({
  to: email,
  subjectIncludes: "Verify your account",
  receivedAfter: startedAt,
  bodyIncludes: "Finish creating your workspace",
});

This does two helpful things. First, it narrows the search space. Second, it tells you what the test believed should happen, which makes failure analysis way more usefull than "timed out after 30s".

If you run email checks in CI, the operational pattern from staging inbox smoke tests is worth borrowing too: keep the inbox logic narrow, observable, and tied to one deploy or one run, not to the whole environment forever. Keeping those concerns seperate helps a lot when a flaky failure only appears under load.

3. Save evidence when the test fails

This is the step teams skip because the happy path already works. Then a red build shows up, and nobody knows if the app failed to send, the inbox API lagged, or the test matched the wrong message.

When a Playwright email test fails, save three bits of evidence:

• the generated email address
• the poll criteria
• the latest inbox snapshot or message list

You do not need to dump everything. A short structured artifact is usualy enough:

test.afterEach(async ({}, testInfo) => {
  if (testInfo.status !== testInfo.expectedStatus) {
    await testInfo.attach("email-debug", {
      body: JSON.stringify({
        email,
        startedAt,
        criteria: {
          subject: "Verify your account",
          bodyIncludes: "Finish creating your workspace"
        },
        inboxSnapshot: await inbox.listRecent(),
      }, null, 2),
      contentType: "application/json",
    });
  }
});

Now the failure review starts from evidence instead of guesswork. That is also why I like storing the inbox name in the same run metadata you use for the rest of the suite. The principle is pretty close to naming inboxes per test run: once artifacts carry the run identity, retries stop erasing the story.

A short checklist I reuse

Before I call an email test "stable enough", I want this checklist to be true:

• each test run gets a unique inbox or alias
• the app logs the email target for the run
• the inbox poll filters by recipient and time window
• the assertion checks a flow-specific string, not just any email
• failures attach enough evidence for another engineer to debug it
• the suite keeps mailbox lifetime short so old mail does not bleed into new runs

It is not a huge list, but it catches most of the avoidable flake I see in QA suites.

Q&A

Should I just increase the timeout?

Only after the matching rules are good. Longer waits can hide shared-inbox bugs and make the suite slower without making it safer.

Is a temporary inbox okay for real projects?

Yes, for staging and test flows. Just keep the scope narrow, avoid real user data, and do not confuse a passing inbox check with full delivery validation.

What if multiple emails are expected in one flow?

Give each message its own assertion contract. If you collapse "verification", "welcome", and "team invite" into one loose poll, the test will get confusing prety fast.

Reliable email tests are mostly about reducing ambiguity. Once the run has one inbox identity, one message contract, and one evidence trail, Playwright failures become much more fixable and much less annoying.