CI Email Tests Need a Replayable Artifact

Email verification tests often fail in the least helpful way: the build says that a message never arrived, but the next run passes. A replayable artifact turns that vague failure into evidence you can inspect without rerunning the entire pipeline.

Why email failures are hard to replay

An email-driven test crosses several boundaries. The application creates a user, an email provider accepts a message, an inbox exposes it, and the test extracts a link or code. A failure at any boundary can look like the same timeout.

The usual debugging response is to add a longer wait. That may hide the problem for a while, but it does not tell us whether delivery was late, the wrong message was selected, or the test used an expired mailbox. In a shared CI enviroment, a retry can also create a second message and make the original failure harder to understand.

I like to treat each run as a small transaction with a receipt. The receipt does not need the whole inbox. It needs enough structured evidence to answer three questions:

1. Which test run owned the mailbox?
2. What messages were visible, and when?
3. Which decision did the test make from that evidence?

This is a useful companion to treating email as a deployment contract: the contract describes what should happen, while the artifact records what actually happened.

The replay artifact model

The artifact is a redacted JSON file written when the test finishes, whether it passes or fails. It contains identifiers and timestamps, not full message bodies or personal data. A good artifact is boring, predictable, and seperately useful from the test logs.

Here is a compact shape:

{
  "run_id": "checkout-1842",
  "mailbox_id": "mbx-7f2a",
  "created_at": "2026-10-29T05:44:12Z",
  "messages": [
    {
      "id": "msg-91",
      "subject": "Verify your account",
      "received_at": "2026-10-29T05:44:19Z",
      "selected": true,
      "selection_reason": "subject and recipient matched"
    }
  ],
  "decision": {
    "status": "verified",
    "observed_code": "***812",
    "poll_count": 4,
    "elapsed_ms": 7210
  }
}

The exact fields are less important than making the decision explainable. For a failed test, selection_reason can show that a message was rejected because it had the wrong recipient. poll_count and elapsed_ms help distinguish a slow provider from an incorrect filter.

Do not store complete links, tokens, or message content in a long-lived CI artifact. Redact values at the point of capture, not later in a reporting job. That gives the artifact a smaller blast radius if somebody downloads it while investigating a failure.

Capture the right evidence

There are four moments worth recording:

• Provisioning: when the disposable temporary email address was created and which test run owns it.
• Polling: each poll start and end, the number of messages observed, and the reason for continuing.
• Selection: the message ID and the matching rules that made it eligible.
• Cleanup: whether the mailbox was released, expired, or left for a bounded investigation window.

Keep the event list small. One event per poll is often enough; logging every HTTP header usually adds noise and may expose credentials. If several tests run in parallel, include a stable run ID in every record. Otherwise a late message from one test can be mistaken for an occurence in another.

For flaky signup tests, focused inbox filters for flaky signup tests are especially valuable. The replay artifact should record the filter version too, because changing a subject matcher can change the result without changing the application.

Use the artifact in CI

A simple workflow is enough:

1. Create a unique mailbox lease for the job.
2. Run the email verification test.
3. Write the redacted artifact in a test-results/email/ directory.
4. Upload it on both success and failure.
5. Release the mailbox in a finally or equivalent cleanup path.

The upload step should not depend on the test exit code. In GitHub Actions, for example, the artifact step can use if: always(). The same idea works in GitLab CI, Buildkite, or a local runner.

If the test process crashes before writing its final decision, a small wrapper can still write a status: incomplete record. That is more useful than an empty directory. It also prevents the familiar question, “did the email fail, or did the test runner disappear?”

When a failure is reported, replay the decision locally from the artifact first. Compare the message timeline, selected message, filter version, and elapsed time. Only then rerun the provider interaction. This saves time and avoids creating a fresh failure that has different conditions.

You may see search terms such as tem email or fake e mail com in support tickets and test fixtures. Keep those strings in test data when they are needed for coverage, but do not use them as a reason to weaken mailbox isolation or logging rules.

A practical checklist

Before calling an email test reliable, check that:

• every mailbox has a unique owner and run ID;
• the artifact is emitted on pass, fail, timeout, and crash paths;
• message selection records a reason, not only an ID;
• tokens, links, and bodies are redacted before upload;
• the CI retention period is short and documented;
• cleanup is attempted even after a failed assertion;
• the artifact format is versioned when fields change.

The goal is not to save everything. It is to save the smallest receipt that explains the test's decision. Once email automation has a replayable artifact, intermittent failures become a data problem instead of a guessing game. That makes the workflow easier to maintain, easier to secure, and a little more pleasent for the next person on call.