Playwright Email Tests Need Triage Snapshots

Email assertions usually fail with weaker evidence than UI assertions. When a button click fails, Playwright gives you a trace, screenshot, DOM state, and timing history. When a signup email fails, many teams only keep one line: "message not found." That is not enough to debug if the issue was delivery lag, wrong inbox ownership, stale data, or a broken template.

After cleaning up a few too many flaky suites, I stopped treating email checks as a special case. I now give them the same triage standard as UI failures: every failed wait should produce a small snapshot of what the test knew at that moment. It sounds boring, but it saves a lot of back-and-forth later.

Why email failures are harder to triage than UI failures

A browser failure is local to the worker most of the time. An email failure crosses systems:

• the app has to enqueue or send the message
• the provider has to accept it
• the inbox poller has to search the right mailbox
• the test has to prove the message belongs to this scenario

If any of those layers is vague, the suite becomes noisy. I often see people retry first and inspect later, which feels efficient but honestly makes the signal worse. A retry can pass while hiding the fact that the first poll used the wrong inbox or searched too early.

This is also where disposable temporary email setups can get messy. In incident notes, someone usually pastes a rushed phrase like tem email while searching logs, and that tiny detail tells you the same thing every time: the team is triaging under pressure. Your artifacts should be clear enough that tired people can still find the story.

For reliability work, I like to think in terms of replayable evidence packs. If an email check fails, I want one compact bundle of facts, not five scattered log streams and two guesses.

What a triage snapshot should capture

My baseline snapshot has six fields:

1. the test scenario ID
2. the inbox address and inbox owner
3. first poll time and last poll time
4. how many matching messages were seen
5. the newest matching subject and received timestamp
6. the trace or artifact path for the worker

That list is intentionally small. If you capture too much, nobody reads it. If you capture too little, the failure becomes another Slack debate. I have found that a short JSON blob plus the Playwright trace is normaly enough to tell whether the bug is in delivery, routing, or assertions.

I also keep the helper tiny. The more configuration your email harness needs, the easier it is to forget one bit during a rushed refactor. This is one of those cases where small context builders for debugging helpers really do beat giant shared utilities.

A Playwright pattern for collecting the snapshot

Here is the pattern I use in TypeScript:

type EmailSnapshot = {
  scenarioId: string;
  inbox: string;
  owner: string;
  firstPollAt: string;
  lastPollAt: string;
  matchesSeen: number;
  newestSubject?: string;
  newestReceivedAt?: string;
  tracePath: string;
};

async function waitForEmailWithSnapshot(args: {
  scenarioId: string;
  inbox: string;
  owner: string;
  tracePath: string;
  timeoutMs: number;
}) {
  const firstPollAt = new Date().toISOString();
  let matchesSeen = 0;
  let newestMessage: { subject: string; receivedAt: string } | undefined;

  try {
    return await pollInboxUntilFound(args);
  } catch (error) {
    const snapshot: EmailSnapshot = {
      scenarioId: args.scenarioId,
      inbox: args.inbox,
      owner: args.owner,
      firstPollAt,
      lastPollAt: new Date().toISOString(),
      matchesSeen,
      newestSubject: newestMessage?.subject,
      newestReceivedAt: newestMessage?.receivedAt,
      tracePath: args.tracePath,
    };

    await test.info().attach("email-triage.json", {
      body: Buffer.from(JSON.stringify(snapshot, null, 2)),
      contentType: "application/json",
    });

    throw error;
  }
}

The code is simple on purpose. The useful bit is not the helper name. The useful bit is the contract:

• every failed wait leaves a readable artifact
• the artifact ties the inbox to one scenario
• timestamps show whether the poll window was realistic
• the trace path points the next debugger to the rest of the run

If your inbox provider returns message metadata, add message IDs too. I would not add full HTML by default, though. It bloats artifacts fast and makes privacy review harder than it needs to be.

The checklist I use before calling a test flaky

Before I label an email test as flaky, I ask four boring questions:

• Did the snapshot prove we polled the intended inbox?
• Did the send action happen before polling started?
• Did the newest message timestamp land inside the timeout window?
• Did the trace show the app reaching the state that should trigger mail?

That checklist catches a lot more than people expect. Some "email failures" are actually setup bugs, race conditions, or bad cleanup from prior runs. Some are real backend regressions. The point is you can separate those paths faster, which makes triage a bit less miserable.

One more thing: keep the snapshot attached on green retries too if your framework allows it selectively. When a failure disappears on retry, that first artifact is often the only honest record of what went wrong.

Q&A

Should I capture a snapshot on every poll?

Usually no. Capture one final snapshot on failure and maybe one light success artifact on pass. Per-poll artifacts get noisy very fast.

Do screenshots help for email failures?

Sometimes, but traces and inbox metadata help more. A screenshot of the app rarely explains why a message was missing.

What is the smallest version of this idea?

Start with scenario ID, inbox, first poll, last poll, and newest received timestamp. Even that tiny set of data makes QA triage much more grounded, and a little less guessy.