Playwright Email Tests Need an Evidence Ledger

Email verification tests often fail in the least helpful way: a button click times out, the CI log says nothing about the message, and the whole run is marked red. The next person reruns it. Then it passes.

I have found that the real problem is not always delivery speed. It is missing evidence. A browser test usually records what the page did, while the mailbox adapter knows what happened to the message. If those observations are not connected, a failure becomes guesswork.

An evidence ledger is a small, structured record for each email test. It does not need to be a new service. It can be a JSON object attached to the test result, with timestamps, identifiers, and safe status values. The point is to answer one question quickly: which boundary failed?

What an evidence ledger adds to an email test

An ordinary test might say:

Expected "Email verified" to be visible

That is useful, but incomplete. The same assertion can fail because the form request was rejected, the email was sent to the wrong address, the inbox poll used an expired lease, or the verification code was extracted from an old message.

For a verification flow, I record events such as:

type EmailEvidence = {
  testId: string;
  mailboxId: string;
  addressHash: string;
  requestedAt?: string;
  deliveredAt?: string;
  codeReadAt?: string;
  verifiedAt?: string;
  messageId?: string;
  finalState: "verified" | "delivery-timeout" | "ui-error" | "cleanup-error";
};

The address itself should usually stay out of shared artifacts. A hash or provider-issued mailbox ID is enough to correlate records without leaking test data. Keep the ledger boring and predictable; a stable schema is easier to query than a large provider response.

Separate browser, delivery, and mailbox evidence

A reliable test has at least three boundaries:

1. Browser boundary: did the user submit the form and see the expected state?
2. Delivery boundary: did the application accept and send the message?
3. Mailbox boundary: did the test inbox receive the right message before its lease expired?

These boundaries should not all be represented by one five-minute timeout. A browser assertion can fail immediately when the UI shows an API error. A mailbox wait can have its own bounded timeout. The final report can then say “delivery-timeout” instead of simply “test failed.”

This is also where idempotency matters. A retry may create a second message, so the adapter should identify the request or verification attempt before selecting a code. A separate guide on idempotency boundaries for verification emails is useful background for this part of the design.

Build the Playwright fixture

I prefer to hide mailbox setup behind a fixture, while keeping the evidence object visible to the test. The test remains readable and the cleanup path stays consistent:

import { test as base, expect } from "@playwright/test";

export const test = base.extend<{
  mailbox: TestMailbox;
  evidence: EmailEvidence;
}>({
  mailbox: async ({}, use) => {
    const mailbox = await createTestMailbox();
    try {
      await use(mailbox);
    } finally {
      await mailbox.dispose();
    }
  },
  evidence: async ({}, use, testInfo) => {
    await use({
      testId: testInfo.testId,
      mailboxId: "not-created-yet",
      addressHash: "not-created-yet",
      finalState: "ui-error"
    });
  }
});

In production code, the fixture would fill the IDs after creating the mailbox. The important design choice is that dispose is guaranteed by finally. Cleanup should not depend on the last assertion passing. Otherwise a failed run can leave messages around and make the next run look successful for the wrong reason.

Turn failures into useful artifacts

Attach the ledger as a test artifact only after removing message bodies, tokens, and private addresses. Include a screenshot or trace for the browser side, but do not assume a trace explains delivery. The ledger should carry the cross-boundary facts:

await testInfo.attach("email-evidence.json", {
  body: JSON.stringify(evidence, null, 2),
  contentType: "application/json"
});

A good failure report lets a QA engineer classify the issue in under a minute:

• ui-error: inspect the page response and visible error state.
• delivery-timeout: inspect the application mail queue and request ID.
• mailbox-mismatch: inspect address ownership and message filtering.
• cleanup-error: quarantine the mailbox and fix test isolation.

For cleanup specifically, deletion proof for temporary inbox tests is a good reminder that disposal is an observable outcome, not just a call in afterEach.

Avoid false confidence in cleanup

There are a few easy mistakes here. Do not mark a mailbox clean merely because the delete request returned 200; verify the provider's final state when the system supports it. Do not reuse a static address across parallel workers. Do not select the newest message without checking the test's correlation value.

Also watch for small naming errors in test configuration. A typo like tamp mail com or tempail mail in a fixture map can send a test down a fallback path that looks like a delivery problem. The spelling is not important to the product, but the failure classification is.

A repeatable CI checklist

Before calling an email test trustworthy, check that it can answer these questions:

• Does every worker receive an isolated mailbox?
• Is the verification request tied to a correlation ID?
• Are UI and delivery timeouts reported separately?
• Are message bodies and addresses removed from artifacts?
• Does cleanup run after assertion failures and timeouts?
• Can a failed run identify the mailbox without exposing it?
• Is the test result useful without rerunning the job?

The ledger does not make a slow provider fast, and it cannot prevent every flaky dependency. It does something more practical: it turns an ambiguous Playwright failure into a bounded diagnosis. Once the evidence is visible, the team can fix the correct boundary instead of adding another arbitrary wait.