Playwright Email Tests Need Delivery Windows

When email assertions get flaky, teams often blame the mailbox first. Sometimes that is fair. More often, the real gap is that the test never defined how long delivery is allowed to take, what counts as late, and which message belongs to the current action. Without that contract, Playwright is left guessing around asynchronous behavior, and QA ends up rerunning builds instead of learning from them.

I started treating these checks as delivery-window tests rather than inbox checks. That small wording change helped a lot. It pushed me to record when the app triggered the email, how long the worker queue was allowed to take, and when the message actually landed. Once that data is present, a failure is much easier to classify and usualy much easier to fix.

Why email tests fail when delivery time has no contract

An end-to-end test can pass while still proving the wrong thing. If the assertion is just "an email arrived", then a stale message, a retried job, or a previous scenario can satisfy the test. This is even easier to miss when notes mention fuzzy terms like tempail or fake e mail com and nobody documents the expected arrival window.

In practice, I keep seeing four failure patterns:

• the app triggers the right template, but the worker finishes later than the UX promise
• the inbox poller grabs an older matching message
• the suite waits too long, so a slow path looks acceptable
• the suite waits too little, so teams patch over normal queue jitter with retries

That is why I like writing down a delivery window in the test case itself. If signup email delivery should complete in 15 seconds for a normal environment, say that. If password reset mail can take 30 seconds because it passes through extra checks, say that too. A vague "eventualy arrives" is not a quality bar, its a shrug.

This is the same reason I pay attention to session boundaries for test inboxes. A message is only useful evidence when you can prove it belongs to one scenario and one timing expectation.

Define a delivery window before you poll

The pattern I use is simple:

1. Record the trigger time right after the product action.
2. Define an acceptable arrival window for that scenario.
3. Poll only for messages created after the trigger.
4. Fail with timing evidence, not just with "message not found".

For example, a verification email test might state:

• expected arrival: within 20 seconds
• warning threshold: after 12 seconds
• hard failure: after 20 seconds
• ownership rule: recipient plus scenario id plus arrival time after trigger

That structure gives QA something concrete to discuss with backend teams. A delivery window is not the same as a timeout. The timeout is only the test mechanic. The delivery window is the product or platform expectation behind it. Mixing those two ideas together is where a lot of brittle automation begins, actualy.

I also like connecting this to adjacent operational checks. If your team already validates async alerts, the ideas in rollback email verification map pretty cleanly: one event, one expected email, one clear time range.

A Playwright helper that records arrival evidence

The helper I want is boring on purpose. It should not hide the timing math. It should expose it:

type DeliveryWindow = {
  expectedByMs: number;
  warnAfterMs: number;
};

type DeliveryEvidence = {
  triggeredAt: number;
  matchedMessageId: string | null;
  matchedReceivedAt: string | null;
  elapsedMs: number;
  warning: boolean;
};

async function waitForDeliveryWindow(
  inbox: TestInbox,
  recipient: string,
  subjectIncludes: string,
  triggeredAt: number,
  window: DeliveryWindow
): Promise<DeliveryEvidence> {
  const started = Date.now();
  const message = await inbox.waitForMessage({
    recipient,
    subjectIncludes,
    afterTimestamp: triggeredAt,
    timeoutMs: window.expectedByMs
  });

  const receivedAt = message ? Date.parse(message.receivedAt) : null;
  const elapsedMs = Date.now() - started;

  return {
    triggeredAt,
    matchedMessageId: message?.id ?? null,
    matchedReceivedAt: message?.receivedAt ?? null,
    elapsedMs,
    warning: elapsedMs > window.warnAfterMs
  };
}

Two details are doing most of the work here:

• afterTimestamp prevents an older email from winning the match
• the returned evidence includes elapsed time, so slow passes are visible instead of invisible

That second part matters more than teams expect. If a test passes in 19.8 seconds against a 20-second window, I do not want the result to look identical to a test that passed in 2 seconds. Both are green, but one of them is trying to tell you something.

I usually store the evidence next to the Playwright trace or test attachment. That way a CI failure comes with enough context to answer three QA questions fast:

• Did the email arrive inside the agreed window?
• Did the message belong to this scenario?
• Was the problem delivery, matching, or the product trigger itself?

Without those answers, people add bigger waits and call the run stable. It may become greener, sure, but not more honest.

What to review when the window keeps slipping

When delivery-window failures start clustering, I review these before touching the timeout:

1. Does the app emit one send event per user action, or can duplicates happen?
2. Does the test isolate mailbox ownership strongly enough?
3. Are background jobs sharing the same queue as slower non-critical mail?
4. Are "green but slow" runs being tracked anywhere?
5. Is the product promise still realistic for the environment under test?

That checklist sounds basic, but it catches a lot of differnt root causes. Sometimes the queue is overloaded. Sometimes the test is reading the wrong mailbox. Sometimes the UX promise was written for local development and never updated after real anti-abuse steps were added. All three produce "flaky email test", but they need very differnt fixes.

Q&A

Should every scenario have its own delivery window?

Mostly, yes. Signup, reset, invite, and OTP flows often have differnt backend paths. Reusing one generic timeout for all of them tends to make at least one path misleading.

What if I use a temporary email account generator in CI?

That is fine, as long as the test still defines ownership and timing. The inbox provider can help isolate messages, but it should not become the only explanation for why a check passed.

Is a slow pass really a failure signal?

I think so. Not every slow pass means a bug, but it is a useful QA signal. If you log it and review it, you can catch delivery regressions before they become full outages or noisy flaky runs.

Reliable email automation comes from explicit expectations. Once the suite names the allowed delivery window and saves evidence for each match, Playwright stops feeling random and starts giving you something you can debug.