Playwright Tests Need an Evidence Ledger

Email verification tests often get labeled as timing problems. The test waits for a message, the message arrives, and the assertion still fails. A retry passes, so the team adds a longer timeout and moves on.

That fix can hide the real problem: the test has no record of which message it expected or why that message was selected. In a QA suite, “the newest email” is not evidence. It is only a guess that happened to work once.

An evidence ledger gives each Playwright run a small, inspectable history. It connects the signup action, the email request, the message that was observed, and the final assertion. This makes failures less mysterious, whether the inbox is provided by tempmailso or by an internal mail catcher. It also helps when a temp org mail address is reused by several test workers.

The failure that looks like a Playwright timing bug

Consider two parallel tests using the same address:

1. Test A submits a signup form.
2. Test B submits the same form with different data.
3. Two verification messages arrive close together.
4. Both tests read the newest message and look for a link.

One test may pass with the other test's link. The next run may fail, even though the application behaved correctly. A slow CI runner makes the order change more often, but the root cause is missing ownership of the evidence.

The same issue appears with a single test when an old message is left in the inbox. A selector such as subject contains "Verify" is useful, but it does not prove that the message was created after this run's signup action.

Record an evidence ledger for every test

The ledger does not need a database. A fixture can keep a small array in memory and attach it to the test report when the test ends.

type EvidenceEvent = {
  at: string;
  step: string;
  runId: string;
  messageId?: string;
  details?: Record<string, string>;
};

const evidence: EvidenceEvent[] = [];

function record(event: Omit<EvidenceEvent, "at">) {
  evidence.push({ ...event, at: new Date().toISOString() });
}

Give each test a run ID before it creates an account. Record the input email, the time the verification request was sent, the polling selector, and the message ID returned by the inbox adapter. Do not put full message bodies or verification tokens into a normal CI log; store a redacted artifact if deeper inspection is required.

The important field is not the timestamp by itself. It is the relationship between events. A useful sequence looks like this:

run-created -> signup-submitted -> email-requested
           -> message-observed -> link-asserted -> cleanup-complete

If message-observed has no email-requested event, the test used stale evidence. If two runs point at the same message ID, the test data boundary is broken. These are much more actionable findings than “expected link, received nothing.”

Correlate messages instead of reading the newest email

The inbox adapter should accept a correlation key and a lower time boundary. For example:

await inbox.waitForMessage({
  address,
  after: requestStartedAt,
  headers: { "x-test-run": runId },
  subject: "Verify your account",
});

If the mail provider cannot preserve a custom header, put a short run suffix in the plus-address or in a unique signup field. The exact mechanism varies, but the rule stays the same: the test must be able to explain why this message belongs to this run.

Polling should also have a stop rule. Use a deadline, a small interval, and a final diagnostic event. An endless loop makes a failed test expensive and leaves less information for the next investigation. For a related CI pattern, see replayable email checks in GitHub Actions.

Do not make the polling helper silently swallow provider errors. Distinguish message_not_found, provider_unavailable, and invalid_correlation. Each error suggests a different next action. Retrying a provider outage may help; retrying an invalid correlation only creates more noise.

Turn the ledger into a failure diagnosis

When a test fails, print a compact ledger summary next to the Playwright trace:

run=checkout-184
requested_at=10:42:11.204Z
observed_message=msg-771
observed_at=10:42:13.009Z
correlation=missing
result=message_rejected

This summary tells the reviewer that an email did arrive, but it was not safe to use. That is a different bug from a missing email, and it should go to a different owner.

Keep cleanup in the ledger too. A mailbox that was not released can contaminate the next test and make the suite look more flaky than it is. A dry-run contract for automation runbooks is a useful companion idea when you want to prove the workflow without sending real actions: a dry-run contract for automation runbooks.

Q&A: practical boundaries for CI email tests

Should every test create a new mailbox?

Not always. A unique address or lease per worker can be enough, provided messages have a correlation key and cleanup is observable. Shared inboxes are acceptable only when the adapter enforces ownership.

Is a longer timeout ever the right fix?

Yes, when delivery latency is genuinely higher than the deadline. First record the provider response and polling duration, though. Otherwise a timeout change can hide a selector bug or an old-message collision.

What belongs in a Playwright trace?

Keep browser steps, run IDs, message IDs, and redacted status fields. Tokens, full email bodies, and private addresses should be masked. The trace should help QA diagnose the failure without becoming a second secret store.

A repeatable QA checklist

Before calling an email test reliable, check that:

• every run has a unique ID;
• the request time is recorded before polling;
• the message selector includes correlation data;
• old messages are rejected, not merely skipped;
• polling has a deadline and typed failure reasons;
• the evidence ledger is attached to the test report;
• cleanup records success or explains why it failed.

The ledger is a small addition, but it changes the debugging question from “why did Playwright miss the email?” to “which evidence did this test accept, and why?” That question is more precise, more repeatable, and much kinder to the person investigating CI at 2 a.m.