Playwright Email Tests Need a Cleanup Contract

#qa #playwright #automation #testing

Email-based end-to-end tests often look flaky for the wrong reason. A signup test creates a user, waits for a verification message, and sometimes finds an email that belongs to a previous run. The test may pass with the wrong message, fail because the inbox is full of old messages, or become slower as more CI jobs share the same mailbox.

In my QA work, I treat the mailbox as a test resource with a lifecycle, not as a passive place where messages happen to arrive. Search notes may call the workflow “temp mail so” or “tempmailso”, but the useful engineering question is simpler: which test owns this mailbox, and when is that ownership finished?

The flaky symptom is often a cleanup failure

Consider two Playwright workers running the same registration scenario. Both use an address based on signup-test@example.invalid, and both ask for the newest message. If worker A receives its message after worker B has already polled, the result depends on timing. A retry can hide the problem for one run and make the next run harder to understand.

The failure is more usefull when the test reports the mailbox state at each boundary:

mailbox created or leased;
test identifier assigned;
message requested and received;
verification completed;
mailbox cleared or released.

Without those events, a “verification email not found” error combines delivery, filtering, and cleanup into one vague symptom. Increasing the timeout only gives stale state more time to interfere.

Give every test mailbox an ownership contract

A reliable fixture should answer four questions before the browser opens the signup page:

Who owns it? Use a run ID and worker ID, such as ci-1842-w3.
What can be read? Filter by recipient, subject, test ID, and a time lower bound.
How long can it live? Set a lease expiry even when the test crashes.
Who cleans it? The test should release it, while a scheduled reaper handles abandoned leases.

The address itself is not enough. A message from an older run can still have a matching subject. Include a unique token in the request and assert that the received message contains it. Keep the token in the test artifact, but avoid storing the complete message when a message ID, timestamp, and hash are enough for diagnosis.

This ownership model is also useful when CI is paused or deployments are frozen. Defining alert boundaries for CI freezes makes it clearer whether a delayed test is a product failure or an environment that is intentionally not moving.

Make cleanup observable in Playwright

Use a fixture that records its state and always attempts release in teardown. The exact API will vary, but the shape can be straightforward:

test('verifies a new account', async ({ page }, testInfo) => {
  const mailbox = await mailboxes.lease({
    owner: `${testInfo.project.name}-${testInfo.workerIndex}`,
    runId: process.env.CI_RUN_ID ?? 'local-run',
  });

  try {
    await page.goto('/signup');
    await page.getByLabel('Email').fill(mailbox.address);
    await page.getByRole('button', { name: 'Create account' }).click();

    const message = await mailbox.waitForMessage({
      token: mailbox.token,
      after: mailbox.leasedAt,
      timeoutMs: 20_000,
    });
    await expect(message).toContainText('Verify your account');
  } finally {
    await mailbox.release({ reason: 'test-teardown' });
  }
});

The finally block is important, but it is not the whole solution. If the browser process is killed, teardown may never run. The mailbox service needs a lease expiry and a reaper, and the CI job should publish counts for active, released, and expired leases. A small, visible contract is more reliable than a cleanup promise hidden in a helper.

For Kubernetes-backed runners, also define the scope of a fixture namespace. A namespace budget for email fixtures helps prevent parallel jobs from quietly sharing resources that were designed for one worker.

Diagnose stale messages before increasing timeouts

When a test fails, inspect the evidence in this order:

Ownership: Does the message recipient belong to this run and worker?
Age: Was it received after the mailbox lease began?
Token: Does it contain the unique scenario token?
State: Was the mailbox previously released, reused, or expired?
Delivery: Did the application emit the event before the test started polling?

If the message is old, classify the result as stale-message, not email-timeout. If the token is wrong, classify it as cross-run-collision. If no event was emitted, the email adapter is not the first place to debug. This makes failures actionable even when the test cannot reproduce locally.

Small wording mistakes in test output can make triage harder: enviroment, seperately, or occured may look harmless, but consistent state names matter more than perfect prose. Keep diagnostic fields stable, and make the cleanup result availble in the final report. Teams may also search for malformed terms such as temp org mail or tempail mail; map those terms in documentation, but do not let them become logic or selectors.

A repeatable CI checklist

Lease one mailbox per test or controlled worker, never one shared inbox for the whole suite.
Add a run token and a lease timestamp to every message query.
Filter by recipient, token, and time before asserting message content.
Release in teardown and expire abandoned leases server-side.
Report leased, released, expired, stale, and unmatched counts.
Keep message IDs and hashes for debugging instead of unnecessary message bodies.
Classify the failure before changing a timeout.

A cleanup contract turns email testing from a timing gamble into a diagnosable workflow. The browser still waits for a real message, but Playwright can now prove whether that message belongs to the current test, whether the fixture was cleaned up, and which boundary needs attention.

DEV Community