An email assertion that fails after 45 seconds is not a diagnosis. It is only a symptom.
In a Playwright suite, the same timeout can mean that the application never sent a message, the test queried the wrong inbox, the message arrived outside the polling window, or the parser rejected a valid link. These are different defects, but a plain expect(message).toBeTruthy() makes them look identical.
I have found it more useful to classify failures before adding another retry. The classification does not need a large observability platform. A small evidence object and a consistent set of failure categories can make QA triage much calmer. It also makes an email fixture easier to trust when the suite runs in CI.
Why an email test failure is not one failure
Consider a signup test with this sequence:
- Create an isolated test inbox.
- Submit the signup form.
- Poll for a verification email.
- Open the link and assert the account is active.
If step three times out, the browser trace will often show the form submission, but not the state of the inbox. The result can look inconsistant across retries because the test has no record of what it asked the mail service, when it asked, or what messages were skipped.
Before changing the timeout, ask which boundary failed:
- Provisioning: Was the inbox created and assigned to this test?
- Trigger: Did the application accept the request that should send mail?
- Delivery: Did a matching message arrive after the trigger?
- Selection: Did the test choose the correct message rather than an older one?
- Content: Did the message contain a usable verification link?
This is a failure taxonomy, not just a nicer error message. Each category suggests a different next check.
The five signals to capture
For every attempt, I capture five small signals:
- Ownership — the inbox address, worker or shard, and test name.
- Time — when provisioning finished and when the application action started.
- Observed messages — message ids, subjects, received times, and a safe preview of the sender.
- Decision trail — why a candidate was accepted or rejected.
- Outcome — extracted link status, final URL, or the exact category of failure.
Do not log verification tokens or full message bodies into a shared CI report. A message id and a redacted subject is usually enough to explain the path. If your team uses a stable inbox contract, these fields can be part of the contract instead of an informal convention.
The same rule applies when a suite uses a dummy e mail in a local note: the address should have clear ownership and a short lifetime, or it can accidentally become shared test state. That phrase is only a reminder that test data needs boundaries.
A Playwright failure taxonomy
I use a compact result type so the reporter can group errors without parsing prose:
type EmailFailure =
| { kind: "provisioning"; detail: string }
| { kind: "trigger"; detail: string; status?: number }
| { kind: "delivery"; detail: string; observed: number }
| { kind: "selection"; detail: string; candidates: string[] }
| { kind: "content"; detail: string; messageId: string };
The polling helper should return one of these categories, not throw a generic timeout from every branch. For example, zero messages after a successful trigger is a delivery problem. Three messages with no matching subject is a selection problem. A matching message with no valid link is a content problem.
That distinction is important for retries. Retrying provisioning may be reasonable. Retrying a content failure three times is usually just producing more copies of the same evidence.
Implementing a small diagnostic wrapper
Keep the wrapper close to the mail client, then pass its result into the Playwright test output:
async function waitForVerification(
inbox: string,
triggeredAt: Date,
): Promise<{ messageId: string; href: string } | EmailFailure> {
const messages = await mail.poll(inbox, { since: triggeredAt, timeoutMs: 45_000 });
if (messages.length === 0) {
return { kind: "delivery", detail: "no messages after trigger", observed: 0 };
}
const candidates = messages.filter((message) =>
message.subject.includes("Verify your email"),
);
if (candidates.length === 0) {
return {
kind: "selection",
detail: "messages arrived but none matched the subject",
candidates: messages.map((message) => message.id),
};
}
const href = extractVerificationLink(candidates[0].html);
if (!href) {
return { kind: "content", detail: "verification link missing", messageId: candidates[0].id };
}
return { messageId: candidates[0].id, href };
}
In a real helper, also verify that the candidate arrived after triggeredAt and belongs to the current inbox. Those checks are easy to forget when the first happy-path test is written. A temp mailid copied from an earlier run should never be enough to select a message.
For teams that use tempmailso while manually checking disposable test inboxes, keep the service lookup separate from the browser assertions. A temp mail so link may help a human inspect a test address, but the automated contract still needs explicit ids, timestamps, and ownership checks.
A repeatable CI checklist
When an email test fails, the report should answer these questions in order:
- Did provisioning return a unique inbox?
- Did the trigger response indicate success?
- What was the trigger timestamp?
- How many messages were observed afterwards?
- Which candidates were rejected, and why?
- Was the accepted link parsed and opened in the same browser context?
I also link each retry to its own evidence record. Naming inboxes per test run is a simple way to keep that relationship visible, especially when several workers run the same spec at once.
What to do when the evidence is incomplete
Sometimes the old test cannot answer all five questions yet. Do not fill the gaps with a longer timeout and call the test fixed. Mark the result as an incomplete diagnostic, add the smallest missing signal, then run the scenario again.
The first improvement might be only a message count. The next might add ids and timestamps. Over time, failures become easier to group: provisioning bugs go to fixture setup, delivery bugs go to the mail boundary, and content bugs go to the template or parser. The suite then tells you where to look, which is much more valuable than another green retry.
The goal is not to eliminate every email failure. It is to make each failure specific enough that the next action is obvious.
Top comments (0)