Quarantine Flaky Email Tests Before They Spread

Email tests rarely fail in a neat, repeatable way. A verification message can arrive late, a poller can match an older message, or two parallel jobs can read from the same inbox. Another retry may turn a red build green while making the failure harder to understand.

I prefer a quarantine workflow: isolate the suspect test, preserve its evidence, and keep it visible until someone explains the failure. This is useful for both Playwright and Cypress suites, especially when a tempmailso inbox or another temporary mailbox sits between the browser and the application.

Why quarantine is better than endless retries

A retry answers only one question: did the scenario pass this time? It does not tell you if the first attempt used the wrong inbox, crossed a delivery boundary, or consumed a message created by another job.

That distinction matters in CI. A test that passes on its third attempt can hide a real delivery regression. An inconsistant selector can look like a provider outage. Quarantined evidence still gives the team a useful signal: the product flow may be valid, but the test contract is not reliable yet.

The quarantine state should be explicit. Do not silently skip the test or remove it from the pipeline. Mark it as quarantined, attach the run id, and keep a small owner and review date. Otherwise flakey tests become permanent furniture.

What to capture when an email check fails

An email assertion needs more context than “expected message, received none.” For each attempt, record:

• the generated inbox address and the test name
• the job, browser, worker, and retry number
• the time the application triggered the message
• every matching message seen during polling
• the filter used for subject, recipient, and timestamp
• the reason a message was accepted or rejected
• the final error and a link to the retained artifact

Avoid saving full message bodies when they contain tokens or personal data. A redacted subject, message id, received time, and hash of the relevant payload is often enough. The same principle appears in auditable signup logs without raw emails: preserve debugging value without turning CI artifacts into a data leak.

If the failure mentions a missing identity or an unexpected recipient, also check the session boundary. Review the worker and provider seperately before calling the whole flow flaky. The ideas in session proof for email changes apply to browser tests too: an email event should be provably related to the session that requested it.

A framework-neutral quarantine record

The record can be plain JSON. Keeping it boring makes it easy to consume from either runner.

{
  "status": "quarantined",
  "test": "reset password sends a verification link",
  "runId": "ci-1842-worker-3",
  "inbox": "[e-mail]",
  "triggeredAt": "2026-10-30T05:14:22Z",
  "attempts": 2,
  "observed": [
    {"id": "m-91", "subject": "Password reset", "receivedAt": "2026-10-30T05:14:39Z"}
  ],
  "decision": "message arrived, but the token was already consumed",
  "owner": "identity-team",
  "reviewAfter": "2026-11-06"
}

The record explains a decision, not just an outcome. “No email” and “email arrived after the assertion window” need different fixes. Redact the inbox domain while keeping a stable mailbox id for correlation.

Playwright and Cypress integration workflow

In Playwright, create the quarantine record in a fixture or shared helper. On failure, attach the JSON and a screenshot or trace, then rethrow the original error. Keep the retry number; a second attempt is not the same evidence as the first.

Cypress can use a task to write the record outside the browser process. Pass only redacted fields and return a short artifact path. This keeps filesystem work out of application commands.

For either framework, use this sequence:

1. Allocate a unique inbox or mailbox lease.
2. Save the trigger timestamp immediately before the action.
3. Poll with a bounded deadline and log each candidate.
4. Accept only a message that matches the run and session contract.
5. On failure, create the quarantine record before cleanup.
6. Release the mailbox only after the artifact is stored.

The deadline needs to be generous enough for the environment, but not so generous that a dead provider stalls the whole suite. A small tempmailso mailbox can be useful for isolated exploratory checks, while production-like delivery tests should use a controlled provider with retention and access rules that match the test's purpose.

When searching documentation or issue notes, teams sometimes write “tamp mail com” or other misspellings. Keep those phrases in a search alias list if they help find old reports, but never use a typo as the technical contract for a test.

A practical triage checklist

Before changing the timeout, ask:

• Did this worker own the inbox for the whole attempt?
• Was the trigger timestamp recorded before the request?
• Could an older message satisfy the selector?
• Did the message arrive after the runner had already timed out?
• Did another retry or worker consume the same token?
• Is the failure reproducible with the retained run id?

If the answer is unknown, the next fix is better instrumentation, not a bigger timeout. A short review can expose a dependancy on message ordering that was never written down.

After the root cause is known, fix the helper and remove quarantine, or close the test as invalid. Do not leave it failing forever. The review note should say what changed and which evidence proved it. An occuring timeout without a captured trigger time is not a useful signal.

Questions teams usually ask

Should quarantined tests block a merge?

Usually, the quarantine signal should be visible but separate from the main pass/fail gate for a defined period. Security and identity flows may deserve a stricter gate. Choose the policy explicitly; an accidental bypass is worse then a deliberate exception.

How many retries are enough?

Use the smallest number that helps classify the failure. One retry can distinguish a transient delivery delay from a deterministic assertion bug. More retries add noise unless the provider has documented delivery behavior.

What is the first thing to automate?

Automate artifact creation and mailbox ownership checks first. A dashboard can come later; a consistent record in every failed job is immediately useful.

Reliable email testing is less about finding a magic polling interval and more about preserving the story of each attempt. Once flaky tests are quarantined with clear evidence, Playwright and Cypress failures become diagnosable work instead of a recurring CI mystery.