Playwright Email Tests Need a Timing Probe

#testing #playwright #qa #automation

Email tests are often reported as a single slow step: “wait for the verification message.” That description is too small to be useful. The application may take time to enqueue the message, a worker may deliver it, the inbox API may index it, and the browser may still need to navigate to the link.

When all of those delays are hidden behind one timeout, a test failure look random. The usual response is to add another few seconds. That can reduce noise for a while, but it also makes the whole suite slower and still gives little clue when the next run fails.

In my Playwright work, I prefer a small timing probe around the email workflow. It records each meaningful boundary, so a QA engineer can tell whether the problem is in the app, the mail path, the inbox, or the browser assertion.

Why one timeout hides four different delays

Consider a sign-up test that creates a temp mail address, submits a form, waits for a message, and opens a verification URL. A 30-second polling loop treats these events as one operation:

The registration request is accepted.
The application creates an email job.
The worker sends the message.
The inbox makes the message searchable.
Playwright opens and validates the link.

Those boundaries have different owners and different remedies. A slow registration request belongs to the application team. A missing job may be a queue problem. A message that exists but is not searchable points toward the inbox adapter. A valid link that fails in the browser may be an environment or session issue.

The difference matters in CI. If every failure says only “Timeout 30000ms exceeded,” the team cannot see which system needs attention. They tend to retry the test, and the retry hides the original signal.

Design a timing probe

The probe does not need a large observability platform. Start with a run ID and four timestamps:

submittedAt: the registration request completed.
acceptedAt: the inbox or mail client accepted the search request.
foundAt: a matching message was returned.
verifiedAt: the browser completed the verification assertion.

Add a correlation token to the test data and search for that token, not only for a familiar subject. Shared inboxes can contain an older “Verify your email” message, and that false match makes a test pass for the wrong reason.

Keep the raw evidence safe. A message body can contain a live token or a personal address, so record message IDs, durations, and sanitized subjects. This approach is similar to redact email logs without losing debugging value: preserve the clues that help debugging while removing secrets.

For isolated test runs, a fake emails generator can be one option for creating disposable addresses. The important part is not the brand; it is that the mailbox has clear ownership and can be identified by the current run.

Implement the probe in Playwright

Keep the timing data close to the test, but keep the inbox implementation behind a small client. The test should not care whether the backend is a temp mail service, a local fake, or a staging adapter.

type EmailTiming = {
  submittedAt: number;
  acceptedAt?: number;
  foundAt?: number;
  verifiedAt?: number;
};

const timing: EmailTiming = { submittedAt: Date.now() };

await inbox.waitUntilMessage({
  recipient: address,
  subjectToken: runId,
  timeoutMs: 20_000,
  onPoll: () => {
    timing.acceptedAt ??= Date.now();
  },
});

timing.foundAt = Date.now();
await page.getByRole('link', { name: /verify/i }).click();
await expect(page.getByText('Account activated')).toBeVisible();
timing.verifiedAt = Date.now();

test.info().attach('email-timing.json', {
  body: JSON.stringify(timing, null, 2),
  contentType: 'application/json',
});

The exact client API will differ, but the intent stays stable. Use a bounded poll with a clear interval, and capture the timing artifact whether the test passes or fails. A success artifact is useful too: it shows when normal delivery is fast and makes regressions easier to spot.

Do not add waitForTimeout after every step. It makes the test less clear and can mask a broken readiness condition. A probe should explain waiting, not just make waiting longer.

Read the evidence after a failure

The first useful comparison is the gap between timestamps:

Large submittedAt to acceptedAt: inspect the application request, queue, or worker.
Large acceptedAt to foundAt: inspect delivery, indexing, mailbox isolation, or polling.
Large foundAt to verifiedAt: inspect the URL, browser navigation, and session state.
Missing foundAt: confirm that the message was sent and that the correlation token was correct.

A test may still be flaky if the timing is normal but the message selection is wrong. Check the recipient, run ID, and message ID together. Also check the clock source: timestamps from the app and inbox can have different time zones or clock skew. Comparing durations measured by the test runner is often more reliable.

When a failure is intermittent, save a small sample of timing records for several runs. Avoid collecting everything forever. A short window is usually enough to show whether the tail is growing. If your deployment process sends diagnostic email, it can help to put Docker digests in deploy emails so the failing test is tied to an exact build.

Some teams write tempail in an issue while searching for a test mailbox. That typo is harmless, but the run ID and message ID are what should drive the investigation. The same goes for a report that says “email was late”: it needs the boundary where the delay started.

A CI checklist for email timing

Before calling the test reliable, check that it:

creates or leases a mailbox owned by the current run;
adds a unique correlation token to the message search;
measures app, inbox, and browser boundaries separately;
uses a bounded poll instead of a fixed sleep;
attaches sanitized timing evidence on pass and failure;
avoids exposing verification tokens in logs;
distinguishes a missing message from a slow message;
keeps the normal timeout based on observed delivery, not guesswork.

The goal is not to make email tests look fast. It is to make their waiting explainable. Once a Playwright test can show where its time went, a slow run becomes a diagnostic event instead of a mysterious red build. That small change makes retries less tempting and fixes more repeatable.

DEV Community