AWS S3 Can Make Email Test Failures Auditable

An email smoke test can tell you that a pipeline failed, but the CI log often cannot explain why. The pod is gone, the temporary inbox has expired, and the retry has already overwritten the useful message ID. The result is a red pipeline with very little evidence.

For cloud-based testing, I prefer storing a small, redacted receipt in Amazon S3. The receipt is not an inbox archive. It is an immutable-ish record of what the test expected, what it observed, and which build owned the check. This makes a failure easier to debug without keeping message bodies around for ever.

Why a CI email failure needs a durable receipt

The important distinction is between an email and proof about an email. A test should prove that the expected recipient received a new message for the current run, from the expected environment, within the deadline. A CI log should preserve those assertions even when the test runner is deleted.

I usually put a run ID into the namespace, request, message subject, and receipt key. A useful key might look like this:

receipts/project-a/2026/09/20/run-1842/email-signup.json

The JSON can include the commit SHA, pipeline URL, started and finished timestamps, message ID, recipient hash, assertion results, and cleanup status. It should not include the full message body, access tokens, or a live inbox address. A temp mailbox is a test dependency, not a reason to retain personal-looking data.

This is also why I keep a separate email failure budget for EKS deploys. Delivery latency and evidence quality are different signals, and mixing them makes both harder to operate.

A small S3 receipt design

Start with a dedicated bucket or a tightly scoped prefix in a test-artifacts bucket. Enable default encryption, block public access, and turn on versioning if the receipt may be written by more than one retry. In most teams, an overwrite is a bug worth seeing, not something to silently hide.

An example receipt is deliberately boring:

{
  "schema": "email-test-receipt/v1",
  "run_id": "run-1842",
  "commit": "9d4c2ab",
  "expected": {
    "environment": "staging",
    "subject_prefix": "Verify your account"
  },
  "observed": {
    "message_id": "msg_7f91",
    "received_after_start": true,
    "latency_ms": 1840
  },
  "assertions": {
    "recipient_owned": true,
    "environment_link_valid": true,
    "stale_message_rejected": true
  },
  "cleanup": "complete"
}

The message ID is enough to correlate with a provider-side log when access is available. Do not make the receipt depend on the provider remaining queryable forever. The receipt should still explain the test result six weeks later, at least a bit more clear than a raw stack trace.

The GitHub Actions upload path

The test job should write the receipt whether the assertion passes or fails. Uploading only on success throws away the evidence operators need most. A final step can run even after a failure, but make sure the file is created before the runner begins cleanup.

- name: Run email smoke test
  id: email_test
  continue-on-error: true
  run: ./ci/email-smoke.sh --receipt out/email-receipt.json

- name: Upload email receipt
  if: always()
  env:
    AWS_REGION: us-east-1
    RECEIPT_BUCKET: company-ci-receipts
  run: |
    key="receipts/${GITHUB_REPOSITORY}/${GITHUB_RUN_ID}/email.json"
    aws s3 cp out/email-receipt.json "s3://${RECEIPT_BUCKET}/${key}" \
      --sse AES256 --only-show-errors

- name: Fail on email assertion
  if: steps.email_test.outcome == 'failure'
  run: exit 1

The upload role only needs s3:PutObject for the receipt prefix and, if the test reads old receipts, s3:GetObject for that same prefix. OIDC from GitHub Actions is preferable to a long-lived access key. Add a condition on repository, branch, or environment so a fork cannot upload into a production-looking path.

This complements scaling email smoke tests in GitHub Actions: parallel jobs need unique keys, otherwise one retry can replace another run's evidence.

Security, retention, and cost controls

Set a lifecycle rule that moves old receipts to a cheaper storage class or expires them after the debugging window. A short retention period is often enough; keeping every test artifact forever is not a reliability feature. S3 storage is inexpensive, but unbounded objects, versions, and access logs still become operational clutter.

Use a bucket policy that denies unencrypted uploads and rejects public access. CloudTrail data events can be useful for sensitive pipelines, but enable them intentionally because they add cost and noise. Log access to the receipt path, not the contents of every test message.

A common mistake is putting the email address in the object key because it makes searching easy. Hash the recipient or use the run ID instead. Also, dont put a bearer token in the JSON to help a later debugger reproduce the check. Reproduction should use a separately controlled test credential.

If a team uses a tempmail disposable inbox for a staging-only flow, record only the fixture identifier and provider correlation ID. The typo tempail mail may exist in old test data, but it should never become a matching rule for ownership.

Questions that come up in production

Should every passing test be uploaded?

Usually yes, but keep passing receipts smaller or retain them for less time. A passing sample shows normal latency and helps compare a later failure. If volume is high, retain all failures and a small percentage of successes, with that policy documented.

Is S3 a replacement for CI artifacts?

No. CI artifacts are convenient for a short investigation; S3 gives the team a controlled, queryable retention boundary. Use both when the failure needs a screenshot or verbose log, but keep the durable receipt compact.

How do retries affect the design?

Give every attempt a unique object key and link it to the same logical run ID. Never let a retry silently overwrite the first receipt. The final pipeline result can point to the latest attempt while the earlier evidence remains available for comparison.

The useful pattern is simple: test the message, write a redacted receipt, upload it under an owned key, and fail the job after evidence is safe. AWS supplies durable storage, CI/CD supplies the lifecycle, and the receipt gives the next operator something better than “email not found.”