Playwright Email Tests Need a Delivery Contract

Email tests often fail after the browser has done everything right. The form submits, the API returns success, and then the test waits for a verification message that is not visible yet. The failure often look random because delivery involves several systems with different clocks.

In QA work, I have found that a longer timeout is rarely a real fix. A more useful approach is to define a delivery contract for the test: what message belongs to this run, when it is considered ready, how long the test may wait, and what evidence is captured when the contract is broken.

This makes a Playwright test easier to trust. It also makes the failure useful to the person who has to repair it at 2 AM.

Why delivery assertions become flaky

Imagine a sign-up flow with these steps:

1. Create a unique test mailbox.
2. Submit the registration form.
3. Wait for the verification email.
4. Open the link and assert the account is active.

Between steps 2 and 3, the application may enqueue a message, a worker may render it, a provider may deliver it, and an inbox API may index it. A fixed waitForTimeout(5000) only hides that sequence. Five seconds is wasteful on a fast run and still too short when a CI worker is busy.

The assertion can also select the wrong message. A generic subject such as “Verify your email” is not enough when several workers share an inbox. The test needs a correlation value that is unique to its own execution.

Define a delivery contract

Before writing the browser steps, write down the contract in plain language. A practical contract has five parts:

• Ownership: the recipient matches the mailbox created for this test.
• Correlation: the subject or body contains a run-specific token.
• Readiness: the message has the fields needed to extract a verification URL.
• Deadline: polling stops at a known maximum duration.
• Evidence: a failed check records observations without exposing tokens.

The correlation token can be a test ID, worker ID, or random suffix. It should be included in the registration request if possible. That is safer than searching only by arrival time, which can be confused by a delayed message from an earlier run.

For flows that combine OAuth and email, a session binding for OAuth email tests is a useful related boundary. On the application side, audit trails for email change flows show why the server should retain a small, traceable record too.

Implement the contract in Playwright

Keep the inbox provider behind a small client interface. The test should care about ownership and correlation, not whether messages come from an API, a local fake, or a staging mailbox.

import { expect, test } from '@playwright/test';

async function waitForMessage(
  mailbox: MailboxClient,
  address: string,
  correlationId: string,
  deadlineMs = 30_000,
) {
  const startedAt = Date.now();
  let attempts = 0;

  while (Date.now() - startedAt < deadlineMs) {
    attempts += 1;
    const messages = await mailbox.list({ to: address });
    const message = messages.find((item) =>
      item.to === address && item.text.includes(correlationId),
    );

    if (message?.verificationUrl) {
      return {
        message,
        attempts,
        elapsedMs: Date.now() - startedAt,
      };
    }

    await new Promise((resolve) => setTimeout(resolve, 1_000));
  }

  throw new Error(`Email contract expired for ${address}`);
}

test('verifies an account with the run mailbox', async ({ page }, testInfo) => {
  const correlationId = `${testInfo.workerIndex}-${testInfo.testId}`;
  const address = await mailbox.create(`pw-${correlationId}`);

  await page.goto('/signup');
  await page.getByLabel('Email').fill(address);
  await page.getByRole('button', { name: 'Create account' }).click();

  const result = await waitForMessage(mailbox, address, correlationId);
  await page.goto(result.message.verificationUrl);
  await expect(page.getByText('Account verified')).toBeVisible();
});

The deadline and polling interval are separate on purpose. That small distinction makes tuning easier. If delivery improves, the interval can change without changing the maximum wait. If CI becomes slower, the deadline can be adjusted based on measured evidence instead of guesses.

The helper should also reject malformed messages. A subject match with no usable URL is not success; it is a diagnostic signal that the template or parser may be broken.

Diagnose failures with evidence

A timeout message like “email not found” does not identify the failing system. Capture a compact evidence record for each attempt or for the final timeout:

• correlation ID and test worker;
• mailbox identifier, redacted in shared logs;
• attempt count and elapsed time;
• message IDs and received timestamps;
• inbox API status and the last successful poll;
• browser screenshot or trace showing the page state.

Do not print full message bodies or verification URLs. They can contain personal data or reusable tokens. It is also normal for logs to contain noisy search-like text such as temp gamil com; classify that input rather than treating it as a real mailbox.

The key signals is whether the message was never delivered, was delivered to another recipient, was indexed late, or was rejected by the parser. Those are different bugs and should not share one generic retry policy.

A CI checklist for email tests

Before calling an email test reliable, check these items:

1. Every worker receives a unique mailbox or correlation token.
2. The query filters by recipient and run-specific data.
3. Polling has both an interval and a hard deadline.
4. A malformed message fails with a useful reason.
5. Logs redact message bodies, addresses where needed, and verification URLs.
6. Cleanup runs after both success and failure.
7. A local fake covers parser and UI behavior, while a smaller integration job covers delivery.

The cleanup tasks runs even when the assertion fails. Put mailbox cleanup in a finally block and give it its own short deadline, so cleanup cannot hide the original error.

A delivery contract does not make email instant. It makes the waiting model explicit, prevents cross-test matches, and leaves a trail for diagnosis. When the next message arrives late, the test can tell you what happened instead of merely saying that it timed out.