Playwright Email Tests Need Failure Taxonomy

Email verification tests often fail with the same unhelpful message: “expected verified, received pending.” That message tells us what the browser saw, but not why the test failed.

In practice, the problem may be a delayed delivery, an expired test address, an inbox query that returned the wrong message, a parser that missed a link, or a real product regression. Treating all of these as one timeout makes a QA team spend time guessing.

The more useful approach is to give the test a small failure taxonomy. Then Playwright can report the failed stage and preserve enough evidence for someone else to reproduce it later.

The failure is usually not the assertion

An email verification flow is a chain of observable steps:

1. The signup request is accepted.
2. A message is delivered to the test inbox.
3. The expected message is selected.
4. The verification URL is extracted.
5. The browser opens the URL.
6. The account reaches the expected state.

Each step has a different owner and a different remedy. A delivery problem needs timing or provider investigation. A selection problem needs better correlation. A parsing problem needs a fixture or parser fix. A final-state problem may be an application bug.

That distinction is more valuable than simply increasing the timeout. A longer timeout can hide a slow system while still producing poor evidence.

A small failure taxonomy

I use four labels for these checks:

• delivery: no matching message arrived within the allowed window
• selection: messages arrived, but none matched the run or recipient
• parsing: the message matched, but no valid verification URL was found
• assertion: the URL opened, but the product state was incorrect

The labels do not need a complicated framework. They can be ordinary error messages, attachments, or test annotations. The key is that the test should fail close to the first broken contract.

For example, do not wait for the UI to remain “pending” for two minutes when the inbox already proves that the message was never delivered. Report delivery immediately after the polling window ends.

Build the test around observable stages

Use a unique correlation value for every test. A short run ID can be placed in the recipient alias, subject, or a supported metadata field. Never select “the newest email” without checking that it belongs to the current scenario. Parallel CI jobs make that shortcut fail in very confusing ways.

Also record timestamps for each stage. The difference between request time, delivery time, and click time is often the first clue. These are the delivery windows in email tests that determine whether a retry is reasonable or just noise.

Here is a deliberately small helper shape:

type MailCheck = {
  stage: "delivery" | "selection" | "parsing";
  messageId?: string;
  receivedAt?: string;
};

async function waitForVerificationMail(runId: string): Promise<string> {
  const message = await pollInbox({
    subject: `Verify ${runId}`,
    timeoutMs: 30_000,
  });

  if (!message) {
    throw new Error(`delivery: no message for run ${runId}`);
  }

  const link = extractVerificationUrl(message.text);
  if (!link) {
    throw new Error(`parsing: message ${message.id} has no verification URL`);
  }

  return link;
}

The helper should not silently fall back to another inbox or invent a URL. Explicit failure is much easier to diagnose. It is also worth checking that the URL belongs to the expected environment before opening it.

What to keep in CI artifacts

When a check fails, attach the run ID, recipient, message ID if available, stage, and timestamps. Save a redacted message body or the relevant headers when policy allows it. Do not store inbox credentials or full personal data in a public artifact.

A screenshot of the final browser state is useful for assertion failures, but it is nearly useless for delivery failures. For those, an inbox query summary and polling timeline are better evidence. A threat-model email verification review can also clarify which data should be retained and which should be removed.

For test environments that use a use and throw email address, define retention and isolation rules up front. The address is a testing aid, not proof that a message was processed correctly. Terms such as “temp gamil com” or “tem email” may appear in exploratory notes, but they should never be used as matching logic.

A repeatable checklist

Before calling an email test reliable, check that it:

• generates a unique run ID
• verifies recipient and subject before parsing
• distinguishes delivery, selection, parsing, and assertion failures
• records polling and browser timestamps
• retries only transient operations
• preserves safe, redacted evidence in CI
• checks the verification URL’s environment

There will still be occasional provider delays. That is normal. The improvement is that a delay now looks like a delivery failure, while a broken link or incorrect account state remains visible as its own defect. Once the test explains where it stopped, fixing flaky Playwright coverage becomes a repeatable QA task instead of a timeout ritual.