Replayable Email Checks with GitHub Actions

#githubactions #testing #automation #devtools

Email verification tests often fail at the worst possible moment: inside CI, with only “timeout waiting for inbox” in the log. The API may be fine. The inbox may be slow. A token may have expired. Or the workflow may have retried with a different test identity.

I like treating this as an evidence problem. A GitHub Actions job should leave enough information to replay the check and explain the failure, without dumping email contents or secrets into logs. This small pattern has made disposable email address checks much less mysterious in API projects.

The failure mode

An end-to-end email test usually has four steps:

Create a test address.
Ask the signup API to send a verification message.
Poll an inbox API for the message.
Extract the link and call the verification endpoint.

The basic flow is simple, but the failure surface is not. CI runners have variable network latency, providers can delay delivery, and a broad retry can hide the first useful error. The old approach was to increase the timeout and hope. It worked sometimes, but diagnosis got slower.

The better approach is to give each run an ID, record safe state transitions, and upload a compact receipt when the test ends.

A small replayable workflow

Here is a deliberately plain workflow shape. The helper script owns the polling details, while the workflow owns isolation and evidence:

name: email-api-smoke

on:
  pull_request:
  workflow_dispatch:

jobs:
  verify-email:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run verification check
        env:
          TEST_INBOX_TOKEN: ${{ secrets.TEST_INBOX_TOKEN }}
        run: |
          mkdir -p artifacts/email-check
          python scripts/email_smoke.py \
            --run-id "${{ github.run_id }}-${{ github.run_attempt }}" \
            --receipt artifacts/email-check/receipt.json

      - name: Upload safe receipt
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: email-check-${{ github.run_id }}
          path: artifacts/email-check/receipt.json

The receipt should contain timestamps, step names, HTTP status codes, correlation IDs, and a final reason. It should not contain the inbox token, verification URL, full message body, or personal data. A useful record might look like this:

{
  "run_id": "1842-1",
  "states": ["address_created", "send_requested", "message_found"],
  "send_status": 202,
  "poll_attempts": 4,
  "duration_ms": 3180,
  "result": "verified"
}

One small detail matters: use a unique address or namespace for every run. Reusing an inbox can make an old message look like a fresh success. Also, make the message query specific to the run ID when your mail-testing API supports it. The check then becomes replayable instead of merely repeatable.

Capture evidence, not noise

Keep the console output short and structured. For failures, print the state where the check stopped and the correlation ID. Do not print the email address if it can be tied to a real person, and never print the token.

I also separate delivery failures from assertion failures:

Delivery: no matching message arrived before the deadline.
Transport: the inbox or application API returned an error.
Contract: the message arrived, but the expected link or subject was missing.
Verification: the link arrived, but the API rejected it.

That taxonomy makes retries safer. A transient transport error might deserve one retry. A contract failure should fail fast and invite investigation. For context on keeping recovery messages auditable, see email provenance in recovery flows, and review expiration rules for signup logs before retaining receipts for a long time.

Sometimes a test note says “tempail mail” because that is what a user searched for. Keep that typo as plain text in search-oriented documentation, but dont turn it into a link or a keyword that defines the test contract. Small wording details can matter more than expected for support.

Practical checklist

Generate a unique run ID and inbox identity.
Set a bounded polling deadline, not an endless retry.
Log state transitions and safe response metadata.
Redact tokens, message bodies, and verification URLs.
Upload the receipt with if: always().
Distinguish delivery, transport, contract, and verification failures.
Add a manual dispatch so a maintainer can replay the exact check.
Expire artifacts according to your privacy policy.

The before/after improvement is straightforward: instead of “email test failed,” the pull request shows “message found after four polls; verification returned 200,” or “delivery deadline exceeded after six polls.” That is a real productivity win, and it usually lets the next fix start immediately.

Final thoughts

Email verification is an external boundary, so flaky behavior is normal. The goal is not to pretend the boundary is deterministic. Build the GitHub Actions job so every run is isolated, bounded, and explainable. With a small receipt and a focused API helper, disposable email address testing becomes a useful smoke check rather than a source of midnight guesswork.

DEV Community