Replay Email Evidence in GitHub Actions

Email-dependent CI checks are often treated like a single assertion: send a message, poll an inbox, click a link, pass or fail. That model is fast when everything works and nearly useless when it does not.

The useful question is not “did the email test fail?” It is “what did this run observe, and can I replay that observation?” A small evidence bundle in GitHub Actions turns a noisy API failure into a short debugging session.

The failure pattern

Suppose a signup job creates a user, calls an email API, and waits for verification. A retry might find an old message. A shared inbox might contain another branch’s mail. Or the provider may accept the request while delivery is still pending. The final assertion hides all three cases.

This gets worse when teams use a create temporary mail flow or a throwaway email generator for test identities but do not record which address belonged to which run. The inbox is isolated in theory, yet the CI logs can’t prove it.

Start with a run identifier and carry it through every layer:

RUN_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
TEST_EMAIL="ci+${RUN_ID}@example.test"

curl -fsS -X POST "$EMAIL_API/messages" \
  -H 'content-type: application/json' \
  -d "{\"to\":\"$TEST_EMAIL\",\"run_id\":\"$RUN_ID\"}" \
  > email-submit.json

Never put a verification token in a normal log line. Save the response body as a restricted artifact, or redact the token before printing. This is a small habit, but it saves time when a failed run needs to be shared with someone outside the feature team.

Store evidence in the job

Capture four small files, even when the test passes:

1. request.json — the safe request inputs and run ID.
2. submit.json — the email API receipt, including provider status.
3. poll.jsonl — timestamped polling results and message IDs.
4. assertion.txt — the final reason for pass or failure.

A polling record should say what was checked, not just dump a response:

{"at":"2026-09-07T14:20:10Z","message_id":null,"state":"pending"}
{"at":"2026-09-07T14:20:16Z","message_id":"m_4821","state":"matched","subject_ok":true}

Keep the retention period short and the artifact name predictable. Developers should be able to find evidence in seconds, and old test mail should not become a permanent data store. One common mistake is calling it a temp org mail address in a note while the actual fixture uses a different naming rule; consistent labels matter more than clever labels.

For inbox matching, record the predicate too: recipient, subject prefix, run ID, and minimum message timestamp. These fields make a false positive visible.

Replay the check locally

The fastest fix is usually a replay against saved evidence, not a full rerun of the entire pipeline. Make the parser accept a fixture directory:

python scripts/check_email.py \
  --submit .artifacts/submit.json \
  --poll .artifacts/poll.jsonl \
  --expected-run "$RUN_ID"

The command should return a distinct exit code for “no matching message,” “wrong message,” and “malformed provider response.” That distinction is more actionable than a generic timeout. It also lets you add regression fixtures for provider quirks without sending another message.

If your test uses Playwright, pair the browser trace with the inbox record. The guide on inbox filters for flaky signup tests is a useful companion for making the message selection explicit. Before an agent or parallel job starts, freeze email test plans so the expected contract stays stable.

A compact workflow

In GitHub Actions, the shape can stay simple:

- name: Run email API check
  run: ./scripts/run-email-check.sh

- name: Upload email evidence
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: email-evidence-${{ github.run_id }}-${{ github.run_attempt }}
    path: .artifacts/email/
    retention-days: 3

The if: always() line is the important shortcut. Evidence that only exists on success is not evidence; it is decoration. In practice, this workflow makes failures less guessy and avoids rerunning unrelated build steps just to inspect one missing message.

Checklist

• Give every test run a unique email correlation ID.
• Isolate the inbox or use a strict recipient and timestamp filter.
• Save the API receipt, poll history, and assertion reason.
• Redact tokens and keep artifacts for a short period.
• Make the checker replayable from local fixtures.
• Upload evidence when the job fails as well as when it passes.

The payoff is modest but real: a flaky email check becomes a bounded investigation. GitHub Actions still tells you that the build failed, but the artifact tells you why.