Email Checks Need a Failure Budget

#automation #devtools #testing #cicd

Email verification tests are easy to underestimate. The browser clicks the signup button, a message is sent, and the test waits for it. When the inbox is slow, though, the whole workflow can become a vague ten-minute argument with a timeout.

I like treating this as a small reliability problem instead of a special email problem. A test has a limited amount of time and retry capacity. That is its failure budget. Once the team writes the budget down, an intermittant check becomes easier to explain and less expensive to maintain.

This applies whether the test uses a temporary inbox, a sandbox mailbox, or a helper that creates an address for each run. Even a quick search for a temp mail for facebook test account can lead to the same engineering question: what evidence proves that this run used the right inbox and waited for the right message?

The hidden cost of waiting for email

An inbox check usually has three clocks:

the application clock, when the email is queued
the delivery clock, when the provider makes it visible
the test clock, when polling gives up

If those clocks are mixed together, a timeout says very little. Maybe the app never queued the message. Maybe the message arrived after the final poll. Maybe the test read an older message. Teams then increase the timeout, which makes CI slower while keeping the bug blurry.

The problem gets worse with retries. A retry can create a second signup or a second email, so the test may pass by finding the wrong message. Duplicate signup email behavior deserves its own assertion, not a hidden side effect of retrying.

Define a failure budget

Start with a simple contract for one email step:

const emailBudget = {
  totalMs: 30_000,
  pollEveryMs: 2_000,
  maxAttempts: 2,
  subject: "Confirm your account"
};

The exact numbers depend on your system. The useful part is that they are explicit. totalMs covers delivery waiting, while maxAttempts controls how much extra work CI is allowed to do. A retry should not silently double the whole test duration.

I also record a startedAt timestamp before the action that triggers the email. Every inbox query filters for messages newer than that boundary. This prevents a previous run from spending the budget on a stale result, a easy mistake when test addresses are reused.

Record evidence before retrying

Before the next attempt, save a small receipt. It does not need to contain the message body or private user data. These fields are usually enough:

run ID and test name
inbox identifier, redacted where needed
trigger timestamp
poll number and elapsed milliseconds
matching message IDs or subjects
final reason for retry or failure

That receipt changes the debugging conversation. Instead of “email is flaky,” you can say “the app queued the message, the inbox returned no matching ID for 30 seconds, and the second attempt saw two messages.” That is a much smaller problem.

For parallel CI, give each worker a unique inbox lease. Delivery windows in Playwright email tests are especially important here: a fast poll is not a substitute for knowing when a provider can reasonably expose the message.

A small CI implementation

Keep the polling loop boring and bounded:

async function waitForEmail(inbox, budget, startedAt) {
  const deadline = Date.now() + budget.totalMs;
  let polls = 0;

  while (Date.now() < deadline) {
    polls += 1;
    const messages = await inbox.list({after: startedAt});
    const match = messages.find((m) => m.subject === budget.subject);
    if (match) return {match, polls, elapsedMs: Date.now() - startedAt};
    await new Promise((resolve) => setTimeout(resolve, budget.pollEveryMs));
  }

  throw new Error(`No matching email after ${polls} polls`);
}

In production code, add cancellation and structured errors. For a test helper, the mental model matters most: filter by time, poll at a known interval, and stop at a known deadline. The logs should show the same fields on both pass and fail, otherwise the green runs hide useful baseline data.

When to spend the budget

Not every failure deserves a retry. Retry when the receipt shows that the trigger succeeded but delivery was still inside a known transient window. Do not retry immediately when the API rejected the request, the inbox lease is invalid, or the subject contract changed. Those are deterministic failures and another attempt only burns CI minutes.

A get temporary email helper can make setup convenient, but it cannot decide whether the product should accept that address. Keep inbox provisioning, delivery observation, and product policy as separate steps. That separation makes tests more honest and makes future changes less surprizing.

Practical checklist

Before merging an email-based workflow, check that:

the trigger and first poll have timestamps
old messages are excluded
each worker has an isolated inbox
the total wait and retry count are bounded
failure receipts avoid sensitive message content
retries happen only for documented transient cases
the test explains whether queueing or delivery failed

The goal is not to make every email arrive instantly. It is to make slow delivery a normal, measurable state. Once the failure budget and receipt are in place, automation feels much more approachable: the test either finds a new message in its window, or leaves enough evidence for the next fix.

DEV Community