Playwright Email Polling Needs Fresh Evidence

Email assertions are often the last step in an otherwise healthy Playwright flow. The page submits, the API returns 202, and then the test waits for a verification message. When that wait fails, the report may only say that a timeout was reached.

That message is not enough to diagnose the problem. The test might have used the wrong throwaway email address, matched an earlier message, or started polling before the delivery job existed. A retry can pass while hiding the original failure.

This post defines a small evidence contract for email polling. It does not claim one best throwaway email; it makes each Playwright attempt explain what it saw, accepted, and where the wait ended.

Why a passing retry can still be a bad test

Suppose a signup test creates an inbox and submits a form, then asks for the latest message. The first attempt times out during slow delivery; the retry sees a quick message and passes.

The retry did not prove that the first attempt was a transient delay. It may have used a different throwaway email address, worker, or old message with the right subject. Without attempt-level evidence, the team debates whether the product or test is flaky. That takes alot longer than reading a structured failure.

I want each attempt to record four boundaries:

• the inbox or alias assigned to the run
• the time the product action was triggered
• the messages observed after that time
• the reason the selected message belonged to this test

For related operational thinking, deploy evidence for email alerts is a useful reminder that an email result needs context around the event that produced it. For authentication flows, a threat model for verification email adds another helpful question: what exactly does possession of the message prove?

The evidence contract for email polling

The contract can be a plain object saved as a test attachment. Keep it small enough to scan in CI, but detailed enough to distinguish delivery, matching, and application failures.

type EmailPollEvidence = {
  attempt: number;
  inbox: string;
  triggeredAt: string;
  candidates: Array<{
    id: string;
    subject: string;
    receivedAt: string;
    decision: "accepted" | "rejected";
    reason: string;
  }>;
  acceptedMessageId?: string;
};

The triggeredAt value is important. Filtering by message time is safer than always taking the newest message, especialy when a shared test environment receives unrelated mail. The acceptance reason matters too: “matched subject” is weaker than “matched subject and contained the run-specific token.”

If your test notes contain searches such as temp gamil com or tem email, keep those as plain text in diagnostics. They can be useful clues about copied setup instructions, but they should not become link anchors or supported integration names.

A Playwright helper with bounded polling

The helper below uses a deadline instead of an unlimited loop. It also stores every candidate decision, so a timeout leaves a useful receipt.

async function waitForVerificationEmail(
  mail: MailClient,
  inbox: string,
  runToken: string,
  attempt: number,
) {
  const triggeredAt = new Date();
  const deadline = Date.now() + 45_000;
  const evidence: EmailPollEvidence = {
    attempt,
    inbox,
    triggeredAt: triggeredAt.toISOString(),
    candidates: [],
  };

  while (Date.now() < deadline) {
    const messages = await mail.list(inbox);

    for (const message of messages) {
      if (new Date(message.receivedAt) < triggeredAt) continue;

      if (!message.subject.includes("Verify your email")) {
        evidence.candidates.push({
          id: message.id,
          subject: message.subject,
          receivedAt: message.receivedAt,
          decision: "rejected",
          reason: "subject did not match",
        });
        continue;
      }

      if (!message.text.includes(runToken)) {
        evidence.candidates.push({
          id: message.id,
          subject: message.subject,
          receivedAt: message.receivedAt,
          decision: "rejected",
          reason: "run token was missing",
        });
        continue;
      }

      evidence.candidates.push({
        id: message.id,
        subject: message.subject,
        receivedAt: message.receivedAt,
        decision: "accepted",
        reason: "subject and run token matched",
      });
      evidence.acceptedMessageId = message.id;
      return { message, evidence };
    }

    await new Promise((resolve) => setTimeout(resolve, 1_000));
  }

  throw new Error(JSON.stringify(evidence, null, 2));
}

There are a few deliberate choices here. The run token is stronger than a subject-only match. The deadline prevents a stuck provider from hanging the worker. Retaining rejected candidates improves observabilty without dumping full message bodies into CI.

In a real suite, I would also deduplicate candidates by message id. Otherwise one message returned by several polls can make the evidence look noisier than it is. A small helper is worth the few extra lines, mabye more than increasing the timeout.

How to classify failures

When the helper fails, start with the evidence rather than rerunning immediately:

1. No candidates after the trigger time: investigate delivery, the selected inbox, and whether the request created a send job.
2. Candidates with the wrong subject: investigate templates, locale, and environment configuration.
3. Correct subject but missing run token: investigate message routing or whether parallel tests share an inbox.
4. Accepted message but a later assertion fails: investigate the link, token, or application state after email delivery.

This keeps a Playwright timeout from becoming a generic “email issue.” The failure is more specific, and retries can be compared instead of replacing the first attempt.

A repeatable CI checklist

Before calling an email test reliable, I check the following:

• every attempt has an isolated inbox or a unique run token
• polling begins after the triggering action
• message time is checked against the attempt boundary
• subject matching is combined with ownership evidence
• the helper has a hard deadline
• candidate decisions are attached to the test report
• retry attempts preserve earlier evidence

If the environment requires a temporary address, document its privacy and retention limits as part of the fixture contract. A free throwaway email service may suit a low-risk staging check, but production-like data and long-lived accounts deserve a controlled test mailbox. Make that boundary obvious in setup code.

Q&A

Should every email test use a unique inbox?

Ideally, each parallel attempt gets an isolated inbox or a unique address alias. If that is not possible, a run token plus strict time filtering is the minimum I would accept.

Is a longer timeout the right fix for slow delivery?

Only when the evidence shows normal delivery is slower than the current deadline. A longer wait does not fix a wrong inbox, stale-message match, or missing ownership check.

Should I attach the full email body to CI?

Usually no. Store the message id, subject, timestamps, decision, and a redacted diagnostic excerpt. Full bodies can contain personal data and are rarely needed to prove the polling decision.

Reliable email tests are not tests that never wait. They are tests that make the wait bounded, the match explainable, and the failure useful on the first read. That is what lets QA fix the right layer instead of simply pressing retry again.