Playwright Email Tests Need a Failure Budget

An email test that waits for 60 seconds and then says “message not found” is not really a diagnosis. It is only a symptom. The same timeout can mean that delivery is slow, the test opened the wrong inbox, an old message was selected, or the application never sent anything at all.

I have found it more useful to give each email test a small failure budget. That budget is not only a timeout. It is a set of explicit questions the test must answer before it fails: did the browser trigger the request, did the inbox receive anything after that moment, and did the message belong to this test run?

This approach works well with a tempmail disposable inbox, but the important part is the evidence contract around it. The provider is just one component. The test should make the boundary between delivery noise and a product defect visible.

Why a timeout is not a diagnosis

Consider a signup test with this final assertion:

await expect(page.getByRole("link", { name: "Verify email" })).toBeVisible();

When it times out, the report usually tells you that the link was not visible. It may not tell you whether the inbox was empty, whether a message arrived with a different subject, or whether the parser ignored a perfectly valid message.

That missing detail creates a familiar loop:

1. increase the timeout
2. rerun the test
3. see it pass once
4. call the next failure flaky

The test becomes slower without becoming more trustworthy. A timeout should be the last line of a small investigation, not the whole investigation. Teams also make the results harder to read when they mix a real delivery delay with a wrong inbox configuration. The failure messages are similar, but the fixes are seperate.

Define the failure budget before writing the test

Before adding a polling loop, write down the states that matter. For a verification email, I normally use four:

• triggered: the browser completed the action that should send the email
• observed: the inbox returned one or more messages after that action
• accepted: one message passed subject, recipient, and timestamp checks
• opened: the link was extracted and used successfully

Each state gets evidence. Store the trigger time, inbox identifier, message ids, received times, and rejection reasons. If no message arrives, that points toward delivery or wiring. If messages arrive but none are accepted, the matching rule is suspect. If a message is accepted but the browser step fails, the application or link parser deserves attention.

That distinction is the failure budget: every red run spends a little diagnostic information, and the test should leave enough behind to explain where it was spent. It makes a test slower to design, but much faster to repair later.

A Playwright helper that makes the budget visible

Keep the polling and evidence collection in one helper. The test itself should describe the user flow, not repeat inbox rules in every spec.

type InboxEvidence = {
  inbox: string;
  triggeredAt: string;
  seen: Array<{ id: string; subject: string; receivedAt: string }>;
  rejected: string[];
};

async function waitForVerification(
  mail: MailClient,
  inbox: string,
  triggeredAt: Date,
): Promise<{ url: string; evidence: InboxEvidence }> {
  const evidence: InboxEvidence = {
    inbox,
    triggeredAt: triggeredAt.toISOString(),
    seen: [],
    rejected: [],
  };

  const deadline = Date.now() + 45_000;
  while (Date.now() < deadline) {
    const messages = await mail.list(inbox, { since: triggeredAt });

    for (const message of messages) {
      evidence.seen.push({
        id: message.id,
        subject: message.subject,
        receivedAt: message.receivedAt,
      });

      if (!message.subject.includes("Verify your email")) {
        evidence.rejected.push(`${message.id}: wrong subject`);
        continue;
      }

      if (!message.html.includes(inbox)) {
        evidence.rejected.push(`${message.id}: recipient proof missing`);
        continue;
      }

      return { url: extractVerificationUrl(message.html), evidence };
    }

    await new Promise((resolve) => setTimeout(resolve, 1_000));
  }

  throw new Error(JSON.stringify(evidence, null, 2));
}

The exact mail client will differ, but three details are worth keeping. The query starts at the browser action, not at inbox creation. Matching checks ownership instead of trusting the newest message. And the thrown error contains structured evidence that CI can attach to the report.

For a broader example of separating CI email checks from application failures, the same principle applies: isolate the boundary and record what crossed it. A note that says fake e mail com or tem email is not useful evidence by itself; the run still needs ids and timestamps.

How to triage a red run

I read the evidence in a fixed order:

1. No messages seen: check the trigger request, mail routing, and delivery latency.
2. Only old messages seen: check the since boundary and inbox isolation.
3. Fresh messages rejected: inspect the subject, recipient proof, and parser assumptions.
4. A message was accepted but the UI failed: inspect the extracted URL, session state, and browser navigation.

This order prevents a common mistake: changing the polling timeout when the real problem is cross-test contamination. Parallel workers can also collide when they share one inbox. A fresh address per worker, or a provider-supported lease, gives the evidence a cleaner owner.

It is worth comparing this with less guessy signup email assertions. The goal is not to make every assertion more strict. The goal is to make each strict check explain what it protects.

A repeatable checklist

Before merging an email test, I check:

• the trigger timestamp is recorded after the relevant browser action
• the inbox is unique to the worker or test case
• polling has a bounded deadline and a clear interval
• every candidate message gets a rejection reason
• the accepted message proves ownership and freshness
• CI preserves the evidence when the test fails
• the browser assertion is separate from inbox diagnosis

That last item matters. A test can have perfect inbox evidence and still reveal a broken verification page. Keeping the layers seperate makes the next action obvious, and it avoids retrying a genuine product failure alot.

Q&A

Should I retry the inbox poll?

Yes, when delivery has normal variance, but keep the deadline bounded. A retry should collect more evidence, not erase the first observation.

Do I need this for every email test?

Start with signup, password reset, and invitation flows. They are expensive to debug and often run in parallel. Simple notification checks can use a lighter helper until they show a pattern of failures.

What if the provider returns only the newest message?

Keep the limitation visible in the evidence and strengthen ownership checks. It is not ideal, but an honest limitation is more usefull than a green test that matched an older message.

The best email test is not the one that never waits. It is the one that can explain a wait, spend its failure budget carefully, and point the QA engineer toward the next check.