Playwright Traces for Flaky Email Tests

When an email test flakes in CI, the first reaction is often to add one more retry and move on. I get why teams do it. Delivery is async, queues can wobble, and nobody wants a release blocked by a slow message. But retries alone rarely explain why the failure happened, and that is the part that keeps coming back next week.

What has worked better for me in Playwright is using traces as the main debugging artifact, not just as a pretty recording after the fact. A good trace lets me line up browser actions, server timing, and the inbox lookup in one place. That makes it much easier to tell whether the test failed in the app, the worker, or the message fetch. It sounds simple, but it saves a lot of messy guesswork.

Why Playwright traces help more than another retry

Retries can hide a timing issue without removing it. If the inbox query is broad, or if a previous run left old messages behind, a rerun can pass for the wrong reason. That is especially easy to miss when the assertion only checks the subject line.

Playwright traces help because they preserve the order of events:

• the click that triggered the email
• the network call that returned success
• the UI state shown to the user
• the polling sequence that searched for the message

That sequence matters more than people expect. I have seen teams blame the mail provider when the real issue was an app action firing twice, and I have also seen the opposite: the UI looked fine, but the worker was delayed enough that the test queried too early. The trace makes those cases easier to separate.

If your auth flow depends on email links, the discipline from replay-safe magic link checks is worth borrowing too. The inbox step is only trustworthy when the message can be tied to the current run instead of "whatever arrived last".

The three timestamps I compare first

Before I touch a timeout, I compare three timestamps:

1. when the browser submitted the action
2. when the backend acknowledged the request
3. when the inbox API first returned the matching message

If those numbers drift in the wrong order, the bug is usually obvious enough. For example, if the backend returns quickly but the inbox takes 40 seconds, I start with the worker or provider path. If the browser shows a success state before the request actually completes, the UI contract is probly too optimistic. If the inbox returns an older message almost instantly, the lookup is not scoped tightly enough.

This is also where old test notes can trip people up. I still find scratch strings like temp org mail in repos, which is a decent sign someone tested disposable inbox ideas but never cleaned up the matching rules. Those leftovers are not the bug by themselves, but they often point to a system that evolved without a strong assertion contract.

The article on shared inbox noise reduction hits a similar point from the Node.js side: isolation is cheaper than investigating mixed messages later.

A small Playwright pattern for run-scoped email checks

My default pattern is boring on purpose:

• generate a unique runId per test
• include that runId in the user action or API payload
• persist it into email metadata that the inbox service can query
• fail if the inbox returns a message with a different id

Here is a compact Playwright example:

import { test, expect } from "@playwright/test";

test("password reset email belongs to this run", async ({ page, request }) => {
  const runId = `pw-${Date.now().toString(36)}`;
  const inbox = `reset-${runId}@example.test`;

  await page.goto("/forgot-password");
  await page.getByLabel("Email").fill(inbox);
  await page.getByRole("button", { name: "Send reset link" }).click();

  await expect(page.getByText("Check your inbox")).toBeVisible();

  const startedAt = Date.now();

  while (Date.now() - startedAt < 30_000) {
    const response = await request.get("/test-inbox/messages", {
      params: { inbox, runId }
    });

    if (response.status() === 200) {
      const message = await response.json();
      expect(message.runId).toBe(runId);
      expect(message.subject).toContain("Reset your password");
      return;
    }

    await page.waitForTimeout(1500);
  }

  throw new Error(`No email found for runId ${runId}`);
});

Two details matter here. First, the lookup is scoped by runId, not by subject. Second, the wait loop is explicit, so the trace shows every poll clearly. That sounds minor, but it makes triage way less fuzzy when you open the artifact later.

What the trace viewer tells me when CI fails

When this test fails in CI, I open the trace and check a short list:

• Did the submit click happen once or twice?
• Did the response confirm the action, or was the UI ahead of reality?
• Did polling begin too soon for the system's normal delivery time?
• Did the inbox query use the same runId the app created?
• Did the returned message metadata belong to another run?

I like this flow because it keeps the conversation grounded in evidence. Instead of "email tests are flaky again", I can usually say something more precise, like:

• the frontend fired the action twice after a disabled-state regression
• the worker lagged behind normal SLA in CI
• the inbox API sorted by newest message before filtering by run id

That is the kind of detail that gets a fix scheduled instead of another hand-wavey timeout bump. It also makes postmortems less dramatic, which is nice because nobody wants a 20-minute meeting over a test that was just matching the wrong email.

Quick Q&A for teams adopting this flow

Do I still use retries?

Yes, but I treat retries as transport tolerance, not proof of correctness. The proof still comes from run-scoped matching and traceable timing.

Should every email test capture a trace?

For local runs, not always. For CI failures and for high-value auth or billing flows, yes absolutely. Storage is cheaper than another round of blind debugging, and the time savings are realy noticeable after a few incidents.

What if my inbox service cannot filter by runId?

Then I would fix that before adding more tests. Without a queryable correlation field, you are asking QA to guess. That tends to look stable right up until parallel CI gets busy.

Playwright traces do not magically fix flaky email tests. What they do is make the failure legible. Once the test can prove which run a message belongs to, the trace becomes a reliable map instead of a post-failure souvenir, and debugging gets a lot less annoying.