AWS CI Email Fixtures Need a Promotion Gate

Email fixtures are easy to add to a CI job and surprisingly hard to operate after that job becomes part of a release process. A test can pass while the fixture is still readable, a retry can reuse an old inbox, or a cleanup step can fail after the green check has already been reported.

For cloud pipelines, I treat an email fixture as a short-lived infrastructure resource. It gets an owner, a run ID, a timeout, and an explicit cleanup result. The deployment promotion gate then checks that contract. This is a small change, but it prevents a lot of debugging by guesswork.

The promotion problem

Consider a Docker-based integration test running in GitHub Actions. The test creates an address, requests a verification email, and waits for a message. The application is deployed to AWS, while the test runner might use a container on a hosted worker. When the test is retried, the second attempt can see the first attempt's message. When the job is cancelled, the fixture can remain available.

Those are not only test failures. They are evidence-quality failures. A green result that cannot explain which mailbox, deployment, and attempt it used is not strong evidence for promotion.

Email polling also has a timing boundary. I keep the polling timeout and retry count in the test contract, instead of hiding them in a helper. The notes on email timing budgets in Playwright are a useful reminder that waiting is part of the test design, not just a sleep added at the end.

Define the fixture contract

Before the test starts, create a run-scoped fixture with metadata similar to this:

{
  "run_id": "gh-1842-attempt-2",
  "owner": "checkout-integration",
  "created_at": "2026-10-04T02:10:00Z",
  "expires_at": "2026-10-04T02:25:00Z",
  "environment": "staging"
}

The important fields are not the exact names. The fixture must be uniquely connected to one pipeline attempt. The application should use the generated address only for that attempt, and the test should reject a message that does not carry the expected correlation token.

I also record whether the fixture was created, consumed, expired, or cleaned. That state belongs in the test evidence bundle. It should not be inferred from a final exit code.

For an AWS deployment, keep the fixture configuration outside the image. Inject the endpoint and a short-lived test token through the CI job or a protected staging secret. Never bake a mailbox credential into a Docker layer. Image history is an awkward place to discover that mistake.

Make the pipeline prove cleanup

The cleanup path needs to run on success, failure, and cancellation. In GitHub Actions, use a final job with if: ${{ always() }} and make it consume the fixture manifest produced by the test job:

cleanup-email-fixtures:
  if: ${{ always() }}
  needs: integration
  runs-on: ubuntu-latest
  steps:
    - uses: actions/download-artifact@v4
      with:
        name: fixture-manifest
    - run: ./ci/cleanup-email-fixtures.sh fixture-manifest.json

The script should be idempotent. If the resource is already gone, record already_absent and continue. If the cleanup service is unavailable, return a failure that blocks promotion; do not turn it into a warning just to keep the dashboard green.

For Kubernetes jobs, put the run ID in the namespace or pod labels and give the job a TTL. Labels make orphan searches possible, while the TTL is only a backstop. A cleanup owner still needs to inspect leftovers. Docker gives you process isolation, not lifecycle management.

The failure report should distinguish at least four cases: the message never arrived, the wrong message was read, the fixture expired, and cleanup failed. The failure taxonomy for email tests is a good companion for making those results actionable.

Promote only with evidence

I use a promotion gate with three checks:

1. The test result is successful and includes the fixture run ID.
2. The evidence bundle shows the expected message correlation token and timestamps.
3. Cleanup completed, or the resource was already absent and that state was recorded.

The gate can be a small script that reads JSON rather than parsing human-readable logs:

test "$(jq -r '.status' evidence.json)" = "passed"
test "$(jq -r '.cleanup.status' evidence.json)" = "cleaned"
test "$(jq -r '.fixture.run_id' evidence.json)" = "$GITHUB_RUN_ID"

In practice, I allow a separate already_absent cleanup status when a retry has safely removed the resource. The policy should be explicit. Otherwise one engineer treats it as success and another treats it as a leak.

Some old runbooks contain search-shaped strings like tepm mail com or tempail. I leave those as plain text when they help identify legacy test data, but they must never become credentials, fixture selectors, or link anchors. That distinction saved me from chasing a typo as if it were a real endpoint.

A compact checklist

Before making an AWS CI/CD job a promotion dependency, check:

• Every fixture has a pipeline run ID, owner, environment, and expiry.
• Retries create a new fixture or prove why reuse is safe.
• Message polling has a visible timeout and correlation token.
• Docker images contain no fixture credentials.
• Kubernetes labels and TTLs make abandoned jobs searchable.
• Cleanup runs after success, failure, and cancellation.
• Cleanup is idempotent and reports already_absent clearly.
• Promotion reads machine-readable evidence, not only a green job badge.

The goal is not to make email tests elaborate. It is to make their result trustworthy. Once the fixture lifecycle is part of the contract, AWS deployments get a clear receipt: what ran, what it observed, and whether the temporary data was removed. That is a much better basis for promotion than “the test passed” and a hope that the cleanup step did its job.