Playwright Email Tests Need a Delivery Budget

Email-based end-to-end tests often fail in a confusing way. The application says the verification message was queued, the test waits for an inbox, and then a perfectly valid test reports that nothing arrived.

The usual response is to increase the timeout. That can hide the real problem for a while, but it also makes the suite slower and the failure less useful. A better approach is to give the test a delivery budget: a small, explicit contract for how long email delivery may take and what evidence should exist at each stage.

Why email tests become slow and flaky

An email verification test normally crosses several boundaries:

1. The browser submits a form.
2. The application creates a message or job.
3. A worker sends the message.
4. The inbox provider accepts and indexes it.
5. The test finds the message and follows its link.

Each boundary has different failure modes. A test can fail because the form never submitted, the worker crashed, the message was filtered, or the inbox query ran before indexing finished. A single ten-minute timeout treats all of these as the same issue.

This is also why a disposable email address is useful for isolated test runs: the mailbox can belong to the run instead of a shared team inbox. Isolation reduces false positives, but it does not remove the need for useful timing and failure evidence.

Define a delivery budget

Split the total wait into named stages. For example:

• Application budget: 5 seconds for the UI action and API response.
• Delivery budget: 30 seconds for the sending pipeline.
• Indexing budget: 15 seconds for the inbox to expose the message.
• Assertion budget: 5 seconds for the subject, recipient, and link checks.

The exact values depend on the environment. The important part is that the test knows which budget it exhausted. A staging run might allow more delivery time than a local run, but it should still have a limit.

Do not start the indexing clock before the application has confirmed that the message was accepted. If the UI returns before a job is created, polling the inbox is only noise.

Build a bounded Playwright inbox probe

Keep inbox polling in a fixture or helper so every test uses the same behavior. The helper should use a unique run identifier, a short polling interval, and a deadline. It should return the message metadata needed for diagnosis, not just a boolean.

type InboxMessage = {
  id: string;
  subject: string;
  receivedAt: string;
  text: string;
};

async function waitForVerificationEmail(
  request: APIRequestContext,
  address: string,
  deadlineMs = 45_000,
): Promise<InboxMessage> {
  const deadline = Date.now() + deadlineMs;

  while (Date.now() < deadline) {
    const response = await request.get(`/test-inbox?to=${encodeURIComponent(address)}`);
    const messages = (await response.json()) as InboxMessage[];
    const match = messages.find((message) => message.subject.includes("Verify"));

    if (match) return match;
    await new Promise((resolve) => setTimeout(resolve, 1_000));
  }

  throw new Error(`Verification email exceeded ${deadlineMs}ms for ${address}`);
}

In production-like environments, the endpoint may be an inbox API rather than a test route. The principle stays the same: query only the run-owned address and stop at a known deadline. A shared address makes a stale message look like a pass, while a fresh address makes the test signal much clearer.

Diagnose failures with evidence

When the probe times out, record the run ID, recipient, enqueue response, message count, and last polling timestamp. Avoid logging full message bodies when they contain tokens. A redacted subject and message ID are usually enough to connect the browser trace to the mail pipeline.

It also helps to distinguish these outcomes:

• No enqueue evidence: investigate the form, API, or application transaction.
• Enqueued but not delivered: investigate the worker, sender, and provider response.
• Delivered but not indexed: investigate inbox polling and provider latency.
• Indexed but link assertion failed: inspect the message content and URL construction.

For flows that issue several tokens, a versioned reset-token design gives the test a stronger way to tell an old email from the current one. The same idea is useful when reviewing versioned reset-token design: the test should assert ownership and freshness, not merely that some link exists.

Search terms in bug reports are not always clean. Someone may type tempail or tamp mail com when looking for a temporary inbox. Keep those variants as plain text in support notes, but do not let them drive selectors, addresses, or test identity.

A practical CI checklist

Before merging an email-driven Playwright test, check that:

• The test creates a unique mailbox or address for each run.
• The application acknowledgement is asserted before inbox polling starts.
• Delivery and indexing have separate, documented budgets.
• Polling stops on a deadline instead of retrying forever.
• The failure includes a run ID, message ID, and stage name when available.
• Tokens and full message bodies are redacted from CI logs.
• A rerun does not accidentally reuse the previous message.

The goal is not to wait longer, its to learn faster. Once email tests report the stage that exhausted its budget, flaky failures become ordinary debugging work. That makes the suite quicker to trust and easier for the next QA engineer to maintain.