AWS CI: Docker Checks for Test Email Failures

An AWS CI job can report green while the email step was never trustworthy. The Docker container is running, the application endpoint answers, and the test still cannot find its verification message. That is not a small test annoyance. It means the pipeline is missing a dependency signal.

I have had better results treating test email as an operations dependency. The job should know whether its inbox is reachable before it starts the expensive browser or API checks. A disposable email generator can create isolated test data, but it does not automatically make the delivery path observable. The container, inbox, and CI job need a clear contract.

Why container-green is not pipeline-green

Docker reports a container as healthy when the configured command succeeds. It does not know that an email provider accepted a message, that the correct mailbox received it, or that the message can be read by the test runner.

That creates a familiar failure shape:

ECS task starts -> app health check passes -> test requests verification email
                                      -> inbox wait times out

The timeout is often blamed on Playwright or on a slow AWS network. Sometimes it is. More often, the test has no evidence for which boundary failed. Was the worker down? Did the message go to the wrong address? Did the inbox API return a rate limit? A generic expected email within 30 seconds message is not enough to answer that.

Define the email dependency

Before changing retries, write down the minimum contract for one test run:

• The test gets a unique mailbox or lease.
• The application sends to that exact address.
• The inbox reader can authenticate without exposing a token in logs.
• The reader records message ID, received time, and subject, but not the whole body.
• The lease is cleaned up after the run, even when the test fails.

For a test team evaluating a temporary email address, this boundary is more important than choosing a catchy provider name. The address is a fixture, not a place for real user data. Keep it separate from production mail and make the ownership of each mailbox obvious.

I also keep the expected message identity in the test receipt. The receipt can say signup-verification / run-4812; it should not copy an OTP or password into CloudWatch. This has made incident review much quicker, becuase we can compare the application event with the inbox event without creating another secret store.

Add a Docker health check

The health check should test the narrowest dependency that proves the container is ready for email tests. It should not send a real verification email on every Docker probe. A lightweight inbox capability endpoint is usually enough.

HEALTHCHECK --interval=20s --timeout=5s --start-period=30s --retries=3 \
  CMD wget -qO- http://localhost:8080/internal/test-mail/ready || exit 1

The endpoint can verify configuration and perform a safe, authenticated capability check against the test inbox service. Return a useful status without echoing credentials:

{
  "mail_reader": "ready",
  "mailbox_scope": "ci-run",
  "checked_at": "2026-10-23T02:00:00Z"
}

Do not make a third-party inbox call on a liveness probe. If the provider is briefly slow, Kubernetes or ECS may restart a perfectly usable container. Use readiness for dependency availability and liveness for whether the process itself is stuck. Mixing those signals caused needless restarts in one of my earlier pipelines.

Expose useful evidence in AWS

In AWS, send structured events for each boundary rather than one final test failure. A small event set is enough:

mailbox.leased
verification.requested
verification.accepted
inbox.message_seen
verification.asserted

Include the CI run ID, mailbox lease ID, and message ID where available. Avoid the email address if it is not needed for correlation; a short hash is often safer. Set a retention period for these logs, and redact message bodies at the application boundary.

When the job fails, the operator should be able to see whether the last event was verification.requested or inbox.message_seen. That distinction turns a vague email timeout into either an application delivery issue or an inbox reader issue. It also pairs well with email receipts for Playwright retries and provenance for recovery emails.

Retry without making the failure worse

Retry only the operation that is safe to retry. Polling an inbox for an existing message is usually safe. Requesting a new verification email may not be: it can create multiple messages, rotate the token, and hide the original failure.

Use a bounded poll with an explicit deadline:

deadline=$((SECONDS + 45))
while (( SECONDS < deadline )); do
  if inboxctl find --run-id "$CI_RUN_ID" --event verification.accepted; then
    exit 0
  fi
  sleep 3
done
echo "verification message not observed before deadline" >&2
exit 1

The command above is only useful if inboxctl returns non-zero for a real reader error and a different status for “not found yet.” Otherwise a network failure can look like a normal wait. That distinction was easy to miss, but it saves a lot of wasted reruns.

If a retry must send again, use an idempotency key derived from the CI run and test case. Never let a flaky mailbox check create an unlimited stream of messages.

A practical CI checklist

Before relying on Docker-based email tests in AWS, check that:

1. Readiness proves the inbox reader can operate, while liveness only protects the process.
2. Each run owns a unique mailbox or lease.
3. Logs correlate the run, lease, and message without storing secrets.
4. Inbox polling distinguishes “not received” from “reader failed.”
5. Retries are bounded and idempotent.
6. Cleanup runs after both success and failure.
7. A tempail mail search term has not accidentally become a production dependency.

This arrangement is not complicated, but it changes where the debugging starts. The CI job no longer asks only whether a Docker container is alive. It can show which email boundary is ready, which message was expected, and where the handoff stopped. That is the kind of small reliability improvement that makes AWS pipelines much less noisy.