Make CI Email Checks Explain Their Failures

Email verification tests are easy to write and surprisingly hard to debug. A test can say “verification email not found” while the real problem is a delayed worker, the wrong inbox, a stale message, or a request that never reached the mail service.

I have found that the most useful improvement is not another retry. It is making every check leave behind a small explanation of what it saw. Once CI can answer which inbox, which request, which messages, and how long it waited, a flaky-looking test becomes a normal engineering problem.

The failure is usually not the email

An email check normally crosses several boundaries:

1. The browser submits the signup form.
2. The API creates a verification request.
3. A queue or worker sends the message.
4. The inbox provider makes it readable.
5. The test finds the right message and follows its link.

Each boundary can be healthy while the complete check still fails. A generic timeout hides that difference. It also encourages a familiar fix: increase the timeout and hope the next run behaves better.

A better mental model is to treat the check as a tiny distributed-system probe. The assertion is only the last step. The useful product of the test is the evidence collected along the way.

A small evidence contract

Before changing the test, define the fields that should exist for every run:

type EmailCheckEvidence = {
  runId: string;
  inbox: string;
  requestId?: string;
  startedAt: string;
  waitedMs: number;
  messagesSeen: number;
  matchedMessageId?: string;
  failureReason?: "submit" | "delivery" | "match" | "link";
};

The runId should be unique per test. Do not use a shared inbox when parallel workers can read or delete each other’s messages. A dedicated disposable email address can be useful for isolated test data, provided the mailbox is treated as untrusted input and test data contains nothing sensitive.

The requestId is especially valuable. Capture it from the signup response or a response header, then log it with the browser test. If the message is absent, you can ask the API and worker logs about the same request instead of searching through unrelated events.

Implement the check

Polling should have a deadline, a visible interval, and a final snapshot. Here is a small Playwright-shaped example:

const evidence = {
  runId: crypto.randomUUID(),
  inbox,
  startedAt: new Date().toISOString(),
  waitedMs: 0,
  messagesSeen: 0,
};

const started = Date.now();
let message;

while (Date.now() - started < 30_000) {
  const messages = await inboxClient.list({ to: inbox });
  evidence.messagesSeen = messages.length;
  message = messages.find((item) => item.requestId === requestId);
  if (message) break;
  await new Promise((resolve) => setTimeout(resolve, 1_000));
}

evidence.waitedMs = Date.now() - started;
evidence.matchedMessageId = message?.id;

if (!message) {
  evidence.failureReason = evidence.messagesSeen ? "match" : "delivery";
  await testInfo.attach("email-check.json", {
    body: JSON.stringify(evidence, null, 2),
    contentType: "application/json",
  });
  throw new Error(`Verification email not found for ${evidence.runId}`);
}

The important detail is that the loop matches on a stable identifier, not only on a subject line. Subjects are often reused, and parallel tests can make a text-only match pass for the wrong user. For more practical ideas, these Playwright email checks are a useful related read.

What to save in CI

Save the evidence JSON on every failure. If the message exists, save a redacted copy of its headers and the selected link. Never put tokens, full message bodies, or mailbox credentials into public CI logs.

I also save three timestamps: submission, worker acceptance, and inbox observation. Their gaps tell a quick story:

• A large submission-to-acceptance gap points to the API or queue.
• A large acceptance-to-observation gap points to delivery or inbox polling.
• A short gap with no match points to filtering or correlation logic.

When workers run in parallel, parallel inbox isolation matters as much as the polling code. Isolation makes the evidence trustworthy, not merely verbose.

A practical failure checklist

When a CI email check fails, inspect these in order:

1. Is the generated inbox unique for this run?
2. Did the submit request return the expected status and request ID?
3. Did the worker accept the job, or did it reject it before sending?
4. How many messages were visible during polling?
5. Did the matcher use a request ID, recipient, and time window?
6. Was the link present but invalid or already consumed?
7. Is the tempail mail test fixture still configured the same way as the inbox client?

This order keeps a delivery problem from being misdiagnosed as a browser problem. It also gives the person fixing the failure a next action, which is what a good test should do.

Final thought

Reliable email automation is less about waiting longer and more about preserving context. Give each run an identity, correlate the message to the request, isolate inboxes, and attach a small redacted evidence file. The test becomes easier to trust because its failures explain themselves.