Email Automation Needs a Dry-Run Contract

Email automation often fails before the email provider is actually the problem. A test creates a user, requests a verification message, waits for a mailbox, and then reports only “timeout.” The next developer reruns it with a longer timeout. That helps sometimes, but it doesnt explain what the test intended to do or which resource was left behind.

I like treating email automation as a small workflow with a dry-run contract. Before the test sends anything, it can describe the mailbox it wants, the messages it expects, and the evidence it must collect. The same description can then drive CI checks, cleanup, and failure triage.

Why email automation needs a dry-run mode

A dry run is not just a flag that skips a network call. It is a preview of the side effects that would happen:

• reserve a mailbox or create a test identity;
• send a verification request;
• wait for a message with a known correlation key;
• consume the message and validate the token;
• release the temporary resources.

This is useful when a test uses a temp email generator or a burner email generator. The runner can show its plan before it spends time waiting for delivery. It can also catch missing configuration, duplicate run IDs, and an invalid cleanup policy earlier.

The goal is a simple mental model: intent first, side effects second, evidence last.

The contract: intent, lease, and evidence

Each email test run should carry three pieces of information.

Intent says what the test is trying to prove. For example, signup_verification expects one verification message for one user.

Lease says how long the mailbox and message are valid. A lease is better than an unbounded “wait until it arrives” rule. A related example is this guide about lease expiry in Playwright email tests.

Evidence says what the runner will save when the flow finishes: message ID, delivery timestamp, matched subject, and the final state. Do not save the full message when a token hash or a short assertion is enough.

These fields make a temporary mailbox behave like a managed test resource instead of an invisible dependency.

A small run manifest

The manifest can be JSON, YAML, or an ordinary struct in your test code. JSON is handy because CI logs and small developer tools can inspect it without extra adapters.

{
  "run_id": "ci-1842-signup",
  "case": "signup_verification",
  "mailbox": {
    "kind": "temporary",
    "lease_seconds": 180
  },
  "expected": {
    "messages": 1,
    "subject_contains": "Verify your account"
  },
  "cleanup": "always"
}

The dry-run output should include the resolved values, not only the requested ones. If the adapter returns a mailbox address, record a redacted form such as qa-****@example.test. If a provider cannot supply a lease, fail the dry run rather than silently creating a resource with unknown lifetime.

For OTP-oriented tests, a snapshot of the inbox can be useful evidence. This article on inbox snapshots for OTP tests shows the same idea from a Playwright perspective.

Turning the manifest into CI checks

The runner can validate the manifest before it contacts the provider:

def validate_manifest(manifest):
    required = ["run_id", "case", "mailbox", "expected", "cleanup"]
    missing = [key for key in required if key not in manifest]
    if missing:
        raise ValueError(f"missing manifest fields: {', '.join(missing)}")

    if manifest["mailbox"]["lease_seconds"] <= 0:
        raise ValueError("mailbox lease must be positive")
    if manifest["cleanup"] != "always":
        raise ValueError("email tests must declare cleanup=always")

Then the executor can use the same object for each state transition:

planned -> mailbox_reserved -> message_requested
        -> message_received -> assertion_passed -> cleaned

Every transition should be idempotent. If the CI worker retries after a network timeout, it should look up the existing run_id before reserving another mailbox. Otherwise one flaky step can create alot of abandoned addresses and make the next failure harder to read.

Failure states that help humans

“Email failed” is not a useful state. I prefer a small set of explicit outcomes:

• configuration_error: the dry run could not resolve a provider or lease;
• mailbox_reserved: setup worked, but no message request was made;
• delivery_timeout: the request exists, but no matching message arrived;
• assertion_failed: a message arrived but had the wrong content;
• cleanup_failed: the test result is known, but a resource still needs attention.

The final report should include the run ID, lease expiry, and message correlation key. It should not print the token or an unredacted address. Those details are informations that can turn a normal CI log into a privacy problem.

Practical checklist

Before adding another retry, check the contract:

• Can a developer preview all external side effects?
• Does every resource have a positive lease and an owner?
• Is the run ID stable across retries?
• Can the runner distinguish no message from wrong message?
• Is cleanup attempted in a finally-style path?
• Does the report explain what occured and what remains?

If a workflow uses a temporary email account generator, keep that dependency behind the mailbox adapter. The test should care about intent and evidence, not a vendor-specific API. That boundary makes it easier to switch providers, run a local fake, or diagnose a CI-only problem.

A dry-run contract is a small addition, but it changes the conversation around failures. Instead of asking “why did email break?”, the team can ask “which transition did not happen, what evidence do we have, and did cleanup finish?” That is a much better starting point for reliable automation.