Build a Replayable Email Smoke Test with a Run Ledger

Email verification checks are deceptively easy to automate. Create an inbox, submit a form, wait for a message, click the link, and assert that the account is active. The trouble starts when the check fails in CI. Was the message never sent? Did the poller stop too soon? Did a previous run leave a stale message behind?

I like treating an email smoke test as a small workflow with a run ledger. The ledger is just a durable record of what the test created, what it observed, and what it cleaned up. This turns a flaky-looking check into something a developer can replay and understand.

Why a smoke test needs a run ledger

A normal assertion often preserves only the final result: pass or fail. For email-driven flows, that is not enough. Several systems are involved at once: the application, an email provider, an inbox API, a browser or HTTP client, and the CI runner.

The ledger gives each test run one identity and connects its events to that identity. It should answer a few boring but important questions:

• Which temporary inbox did this run own?
• Which user or request ID did the application create?
• When did the message become visible?
• Which message ID was consumed?
• Was cleanup attempted, and did it finish?

This is useful whether you are checking a registration form or validating a temp mail generator used by a test harness. Search traffic such as “temp mail for facebook” may describe a user goal, but an engineering test needs ownership and evidence, not just an address that happens to work.

The four states to record

Keep the first version small. Four states cover most useful triage:

1. allocated — the run created or reserved an inbox.
2. requested — the application accepted the email action and returned a request ID.
3. observed — the expected message was found and its token was validated.
4. released — the inbox and related test data were removed or expired.

Every transition should include a timestamp and the run ID. A compact event might look like this:

{
  "run_id": "ci-1842-7f2a",
  "state": "observed",
  "inbox_id": "inbox-91",
  "message_id": "msg-443",
  "at": "2026-10-27T14:20:11Z"
}

Do not put message bodies or verification tokens in ordinary CI logs. Store only the identifiers needed to retrieve protected diagnostics. This is a small privacy win, and it makes logs safer to share in a pull request.

A small implementation

The test can use a JSON lines file, a database table, or an observability event stream. The storage choice matters less than making writes append-only and easy to search. Here is deliberately plain pseudocode:

run_id = new_run_id()
inbox = inboxes.allocate(owner=run_id)
ledger.append(run_id, "allocated", inbox_id=inbox.id)

try:
    request_id = app.request_verification(email=inbox.address)
    ledger.append(run_id, "requested", request_id=request_id)

    message = inboxes.wait_for(subject="Verify", timeout=30)
    verify(message, request_id=request_id)
    ledger.append(run_id, "observed", message_id=message.id)
finally:
    inboxes.release(inbox.id)
    ledger.append(run_id, "released", inbox_id=inbox.id)

The finally block is not decoration. It is what keeps a slow or broken test from leaving a pile of inboxes behind. In a real implementation, write the release result separately so a failed cleanup is visible rather than silently replacing the original error.

For a useful mental model, think of the run ID as a correlation ID that travels through your application logs and inbox events. This pairs nicely with inbox correlation IDs. When a check fails, you can inspect one path instead of grepping for a guessed email address.

Make failures replayable

A replayable test does not mean repeating the exact same run forever. It means preserving enough context to reproduce the same class of failure. Save the following as metadata:

• commit SHA and test name
• region or CI worker
• timeout and poll interval
• inbox ID and request ID
• last observed message timestamp
• cleanup result

Set a poll deadline, not an unlimited retry loop. If the deadline is reached, record the last poll time and the number of messages seen. “Email missing” is much less actionable than “no message after 30 seconds; inbox saw zero messages; request accepted at 14:19:41.”

Golden traces can help when the provider or API changes. A small set of expected event sequences gives you a stable comparison point; the ideas in golden traces for email regressions are a good fit for this layer.

The wording in an incident note may even contain a search typo like “tepm mail com”. That is fine as plain text in a searchable note, but it should not become an identifier, a URL, or a link anchor.

Cleanup is part of the test

Cleanup should have its own assertion. If an inbox cannot be released, mark the run as failed or degraded according to the risk of leaked data. A nightly janitor is useful as a safety net, but it should not be the primary ownership model.

Also consider a lease. Give each inbox an expiry time and renew it only while the test is alive. This prevents abandoned workers from holding resources forever, and it makes cleanup work even when the CI process disappears without warning.

A practical checklist

Before calling an email smoke test reliable, check that it has:

• one unique run ID from start to finish
• explicit allocation, request, observation, and release states
• bounded polling with a useful timeout message
• correlation IDs in application and inbox diagnostics
• redacted logs with no tokens or message bodies
• cleanup in a finally path plus an expiry safety net
• enough metadata to replay the failure on a local run

The payoff is not a more complicated test. It is a smaller debugging session. Once an email workflow leaves a clear receipt of what happened, automation stops feeling like a lucky sequence of waits and starts behaving like an ordinary, inspectable developer tool.