Inbox contracts for stable Playwright tests

Reliable browser tests usually fail for boring reasons, not dramatic ones. The page loads a little slower, the worker sends a verification message twice, or the test reads the wrong inbox state and decides the product is broken. When that keeps happening, I stop blaming Playwright first. I look at the contract around the inbox.

This is especially true in QA work where signup, OTP, and password reset flows are part of a release gate. Teams often say they are testing email, but really they are testing hope: open a mailbox, wait a bit, and assume the newest message is the right one. That is how flakiness sneaks in. It also leads to strange tickets with phrases like temp mail com, temp mail so, or even tempail mail, because nobody is fully sure which inbox path the test actually touched.

Why email checks still flake in good Playwright suites

I see three repeat offenders:

• The test does not define which message belongs to which scenario.
• The poll loop waits for "any email" instead of the expected email.
• The failure output does not preserve enough evidence to debug fast.

The result is a suite that passes on Tuesday, fails on Wednesday, then passes again after a rerun. That is not useful signal. According to the Google Testing Blog, flaky tests reduce trust in automation and create real productivity drag across engineering teams. That lines up with what most QA groups feel day to day, even if they never phrase it that formal.

Before changing waits or adding retries, I would read a post like git bisect for email flakes. The idea is simple: treat an email failure like a system regression, not like random weather.

What an inbox contract should define

An inbox contract is just a few rules that every test follows. Nothing fancy, but it needs to be explicit.

At minimum, I want the contract to define:

• The scenario id or correlation id attached to the email event.
• The recipient mailbox expected for that scenario.
• The subject or event type the test is waiting for.
• The timeout budget and polling interval.
• The evidence saved when the assertion fails.

That contract matters more than the inbox provider. If you swap providers later, the test strategy stays stable. If the strategy is vague, every provider will look unreliable eventualy.

I also like to isolate privacy-sensitive checks early. The article on privacy reviews with email sandboxes makes the same point from a different angle: a cleaner mailbox boundary gives you cleaner evidence and fewer accidental leaks.

A Playwright pattern that keeps evidence

My preferred setup is to create a mailbox fixture per test case, then wait for a message that matches a narrow filter. I do not assert on "latest message". I assert on "message for this scenario". That small wording change fixes a lot.

import { test, expect } from "@playwright/test";

test("user verifies email", async ({ page }) => {
  const scenarioId = `signup-${Date.now()}`;
  const email = await inbox.createAddress({ scenarioId });

  await page.goto("/signup");
  await page.getByLabel("Email").fill(email.address);
  await page.getByRole("button", { name: "Create account" }).click();

  const message = await inbox.waitForMessage({
    recipient: email.address,
    scenarioId,
    subjectIncludes: "Verify your account",
    timeoutMs: 20000
  });

  expect(message.html).toContain("Verify your account");
});

The important part is not the exact helper. The important part is the contract in the helper arguments. Recipient, scenario id, expected subject, and timeout are all visible. When the test fails, I want the report to save the polling summary, matched headers, and final inbox snapshot. Without that trail, people guess. Guessing is how flaky tests stay alive for months.

One more detail that helps a lot: keep the mailbox fixture independent from UI assertions. If the page step fails, I still want inbox evidence. If the inbox step fails, I still want the page trace. That split makes failure analysis much less messy, and a bit more humane for the person on triage duty at 6 PM.

A short checklist for QA teams

When I review an email test workflow, this is my quick pass:

1. Does each test use a mailbox or alias scoped to one scenario?
2. Does the waiter match on business intent, not just arrival time?
3. Are timeout budgets written down and consistent across the suite?
4. Does failure output include message metadata and polling attempts?
5. Can the team reproduce the same path locally without changing code?

If two or more answers are no, I would not call the suite stable yet. I might call it workable, maybe even "good enough for today", but not stable. There is a differnce.

Quick Q&A

Should I just add retries?

Retries can reduce noise, but they also hide weak contracts. Use them after you improve matching and evidence, not before.

Is this only a Playwright problem?

No. Cypress, API tests, and CLI smoke tests run into the same issue. Playwright just makes the seams visible because end-to-end flows are honest about timing.

What should I save on failure?

Save the scenario id, recipient, poll timestamps, matched subjects, and the final rendered page state. That bundle is usualy enough to tell whether the app, the worker, or the test harness drifted.

Stable automation does not come from waiting longer. It comes from asking the inbox a more precise question. When your tests know exactly which message they are looking for and preserve evidence when they miss it, the whole suite gets calmer, and the QA conversation gets a lot less weird.