CI Email Tests Need a Run Manifest

Email assertions are often the least explainable part of an otherwise tidy CI pipeline. A browser test can tell you that a verification link was not found, but that sentence leaves out the useful details: which inbox did the run own, when did the signup happen, and which messages were considered?

I have started treating each email test like a small deployment. It gets an identity, a start time, a cleanup deadline, and a receipt. The result is a run manifest: a compact record that follows the test from inbox creation to pass or failure.

This is a simple idea, but it makes a fake emails generator or temporary inbox tool much more useful. The tool creates the mailbox; the manifest explains what happened around it.

Why an email test needs a manifest

Most flaky email tests have a familiar shape:

• a test creates an address
• the app sends a message
• a polling helper looks for the newest email
• the job fails with a timeout

The timeout is not enough evidence. Perhaps the message arrived in another inbox. Perhaps the test matched an older verification email. Perhaps the worker sent the message after the polling deadline. Those cases look almost identical in a short CI log.

The first improvement is to give every run a stable identifier, such as signup-1842-attempt-1. Then attach the identifier to the test record, inbox label, log lines, and saved artifacts. A teammate should be able to move from a failed assertion to the exact mailbox without searching through several jobs.

I keep this record seperate from the full email body. The manifest is safe to show in ordinary build output, while message bodies and tokens belong in restricted artifacts.

The four fields that make a run explainable

A useful manifest does not need to become a new database. Four groups of fields cover most debugging sessions:

1. Ownership: run ID, test name, inbox address, and the worker or shard that created it.
2. Timing: created-at, trigger time, polling deadline, and the time the message was observed.
3. Matching: message IDs seen, subject checks, sender checks, and the ID of the accepted message.
4. Lifecycle: current state, cleanup deadline, and a reason when the run is rejected.

For example, a redacted record might look like this:

{
  "runId": "signup-1842-attempt-1",
  "test": "new user verification",
  "inbox": "[e-mail]",
  "triggeredAt": "2026-10-12T02:18:04Z",
  "pollDeadline": "2026-10-12T02:18:49Z",
  "seenMessageIds": ["m-991", "m-992"],
  "acceptedMessageId": "m-992",
  "status": "passed",
  "cleanupAt": "2026-10-12T02:23:04Z"
}

Do not put a verification token or the complete link in this normal log. A hash of the token, or just the link host and path pattern, is enough to prove that extraction happened.

A small manifest helper for CI

The helper can be boring on purpose. It should record decisions, not hide them behind a large abstraction.

type EmailRunManifest = {
  runId: string;
  test: string;
  inbox: string;
  triggeredAt: string;
  pollDeadline: string;
  seenMessageIds: string[];
  acceptedMessageId?: string;
  status: "pending" | "passed" | "failed" | "expired";
  cleanupAt: string;
  notes: string[];
};

function startEmailRun(test: string, inbox: string, ttlMs = 5 * 60_000) {
  const now = Date.now();
  const runId = `${test}-${now}`;

  return {
    runId,
    test,
    inbox,
    triggeredAt: new Date(now).toISOString(),
    pollDeadline: new Date(now + 45_000).toISOString(),
    seenMessageIds: [],
    status: "pending" as const,
    cleanupAt: new Date(now + ttlMs).toISOString(),
    notes: [],
  } satisfies EmailRunManifest;
}

The important part is not the TypeScript. It is the sequence: create the record before triggering the application, append every candidate message, then close the record in a finally block. If cleanup is dependant on the assertion passing, failed runs slowly turn into test pollution.

In a parallel job, include the CI shard in the run ID. Two tests with the same human-readable name should still be easy to tell apart.

How to use the receipt after a failure

When a test fails, print a one-line summary and save the full manifest as an artifact:

email-run signup-1842-attempt-1 failed: seen=2 accepted=0 deadline=45s cleanup=scheduled

That line answers more than a generic timeout. If seen=0, investigate delivery, routing, or the application worker. If messages were seen but accepted=0, inspect subject, sender, or ownership rules. If an accepted message exists but the browser still failed, the problem likely sits in link extraction or navigation.

This is also where small human notes help. A rushed ticket might say tepm mail com or “email missing”; the manifest gives the next person something concrete to verify instead of another guess.

I like to link the manifest from the job summary, alongside a cron-friendly email smoke test. For deployment-driven systems, deployment context for rollback emails is a useful related pattern: operational messages are easier to trust when they carry the context that produced them.

Cleanup is part of the contract

An inbox lease without cleanup is only half an automation design. Set an expiry when the run starts, attempt cleanup after pass and failure, and keep a short sweeper for jobs that were killed before their finally block ran.

The sweeper should be idempotent. It may run twice, and it should not fail the current build because a resource was already removed. Record expired when the deadline passes, rather than silently deleting the evidence. This makes the difference between an enviroment problem and a product problem much clearer.

Also decide wether old messages are ignored by timestamp or removed with the inbox. Timestamp filtering is usually safer during diagnosis, but only if the message provider's clock and your CI clock are close enough. Log that choice in the manifest.

A practical checklist

Before calling an email test reliable, check that it:

• creates a run ID before the application action
• records the owned inbox and shard
• records the trigger and polling deadline
• stores every candidate message ID
• explains why a message was accepted or rejected
• avoids printing tokens in normal logs
• schedules cleanup even when assertions fail
• saves a redacted receipt as a CI artifact

This pattern does not make delivery instant, and it will not fix a broken mail worker by itself. It does something more useful for day-to-day engineering: it turns a vague failure into a bounded investigation. Once the test can explain its own run, debugging email in CI feels much less like guesswork.