Playwright Inbox Filters for Flaky Signup Tests

Signup email tests often fail in boring ways, but they also pass in sneaky ways. That second case is the one I worry about more in QA. A green check is not very useful when the test matched an old email, picked the wrong inbox result, or proved only that some message existed at some point.

When I review flaky Playwright suites, the root cause is rarely "Playwright is random". It is usualy a weak mailbox contract. The suite polls too broadly, the alias is not unique enough, or the assertion looks only at a subject line. I have even seen helper comments that still say tempail mail, which usually means the inbox logic grew quick and nobody tightened the semantics after the first demo.

Two posts I like pointing teams to are this idempotent verification flow write-up and these CI approval email checks. They solve different problems, but both push toward one habit I care about: make the email evidence specific enough that another engineer can replay the lookup without guessing.

Why signup email tests pass for the wrong reason

Most flaky signup checks are not truly flaky. They are under-specified.

I usually see one of four patterns:

• the test searches by subject only
• the inbox contains stale mail from a prior run
• retries produce two valid-looking messages
• parallel workers share the same recipient pattern

Any one of those can produce a false pass. The suite sees "Verify your email" and moves on, even though the current browser session did not generate that message. That is why I prefer a disposable temporary email workflow where each run owns its own inbox identity, not just its own assertion.

The inbox filter contract I use in Playwright

My default contract has three filters and one hard failure rule:

1. The test generates a unique recipient for this run.
2. The app carries a run identifier through the email payload.
3. The inbox query filters by recipient, run identifier, and created-after time.
4. The helper fails if more than one message matches.

That sounds stricter than many teams start with, but it reduces triage time a lot. If the test fails, I can answer what we expected, what we searched for, and why the returned message did or did not qualify. If the test passes, I know the evidence was not fuzzy.

For teams that need a quick disposable inbox during local debugging, I sometimes use a free temporary email tool just to inspect payload shape and delivery timing before I lock the checks into automation. The important part is not the provider, it is the narrow lookup contract around the provider.

A small helper that keeps evidence scoped

Here is the minimal shape I reach for:

import { test, expect } from "@playwright/test";

test("signup sends a verification email for this run", async ({ page }, testInfo) => {
  const runId = `signup-${testInfo.workerIndex}-${Date.now().toString(36)}`;
  const email = `qa+${runId}@example.test`;
  const createdAfter = new Date().toISOString();

  await page.goto("/signup");
  await page.getByLabel("Email").fill(email);
  await page.getByLabel("Password").fill("SuperSecret123!");
  await page.getByRole("button", { name: "Create account" }).click();

  await expect(page.getByText("Check your inbox")).toBeVisible();

  const message = await inbox.waitForMessage({
    to: email,
    runId,
    createdAfter,
    timeoutMs: 30000
  });

  expect(message.subject).toContain("Verify your email");
  expect(message.html).toContain(runId);
  expect(message.to).toBe(email);
});

The helper is simple on purpose. Recipient plus runId plus time window is much safer than a broad subject search. If the lookup returns zero matches, I investigate trigger timing or delivery. If it returns many, the query is too loose. If it returns one with the wrong runId, I know stale mail leaked into the result set. That narrows the search realy fast.

What to inspect when the test still flakes

When this pattern still goes red, I inspect the boring metadata first:

• generated recipient
• runId stored by the app
• created-after boundary
• number of candidates returned before filtering
• final matched message ID

That list sounds small, but it answers most QA questions in a few minutes. If you can log those values into the Playwright report or trace annotations, debugging gets much calmer. Microsoft’s Playwright docs also recommend traces for investigating failures because they preserve DOM, network, and action context in one place (source).

I also try to keep retries honest. If the first attempt already triggered delivery, the second attempt should not silently reuse that evidence. Either version the run identifier per attempt, or make duplicate matches a loud failure. Otherwise, the suite gets green while the product behavior drifts a bit every week.

A quick QA checklist

Before I call a signup email test reliable, I want these boxes checked:

• recipient is unique per run
• email payload contains a run identifier
• inbox query uses recipient, runId, and time boundary
• duplicate matches fail on purpose
• logs show enough detail to replay the lookup

If you already have those in place, the next stability gains are usualy about product timing, not test randomness.

Q&A

Should I use one inbox per worker or one inbox per test?

For signup flows, I lean toward one recipient per test run. One inbox per worker is better than sharing, but it still lets retries and reruns smear evidence together.

Is a longer timeout enough?

Sometimes, but only after the filters are correct. A bigger timeout can hide a weak assertion for a while, and that is not the kind of green build I trust much.

Do I always need a run identifier inside the email?

Not always, but it is the cleanest way to prove ownership when the same environment is sending similar verification messages all day.