Docker Checks for AWS Health Drill Emails

AWS Health drill emails are easy to ignore until the week you actually need them. The event exists in the console, the pipeline says the notification Lambda ran, and everyone assumes the email path is fine. Then a maintenance rehearsal starts and the inbox your on-call engineer watches is empty. I have seen that movie a few times, and it is never a fun one.

The fix for me has been pretty small: treat the email like another deploy surface, run one Dockerized check against staging, and prove the final message is useful before the drill starts. It sounds almost too basic, but it catches the boring breakage that real incidents expose fast.

Why AWS Health drill emails deserve a real test

AWS Health events often sit at the end of a chain: EventBridge rule, Lambda transform, SNS or SES handoff, then mailbox delivery. A green log line in the middle of that chain is not enough. In one AWS study on operational excellence, teams that standardize runbooks and rehearsals reduce recovery friction because responders get consistent signals instead of ad hoc guesses (AWS Well-Architected Operational Excellence).

What usually breaks is pretty ordinary:

• the subject line no longer includes the env name
• the message goes to an old distro list
• two CI jobs share one inbox and confuse the evidence
• a template change strips the resource ID that responders need

That last one is sneaky. The email technically arrives, but it is less useful than the console event itself. I use the same mindset as release email checks in Kubernetes ops: if the notification is part of the handoff, you validate what the human will read, not just the service that emitted it.

The Docker workflow I keep reusing

My preferred setup is one run ID, one container, and one isolated inbox for each rehearsal. It is not clever, which is probly why it survives team changes.

1. Trigger a safe AWS Health sample event or replay a sanitized payload through the same staging path.
2. Spin up a Docker job with the exact scripts and env vars used in CI/CD.
3. Create a fresh inbox tied to the run ID.
4. Poll for delivery and assert on the subject, body, links, and event metadata.
5. Store the result as a run artifact, then expire the inbox.

The container wrapper can stay tiny:

RUN_ID="$(date +%Y%m%d%H%M%S)"
docker run --rm \
  -e AWS_REGION=us-east-1 \
  -e RUN_ID="$RUN_ID" \
  -e EVENT_TYPE="AWS_HEALTH_DRILL" \
  ops-mail-check:latest \
  ./scripts/check-health-email.sh

I like containers here for the same reason people like them everywhere else in ops: the failure is easier to replay. If the drill email goes missing next Tuesday, I do not want to rebuild the runtime from memory and hope I remembered the same CLI flags.

Assertions that catch the boring failures

The most useful checks are the ones nobody wants to debug at 2 a.m. I fail the run unless all of these are true:

• exactly one email arrives for the current run ID
• the subject names the expected account, region, or service
• the body still contains the maintenance window and affected resource
• links open the right AWS console area for the current account
• the timestamp is close enough to the triggering event to trust the trail

I also keep a small metadata block with the commit SHA, image tag, EventBridge rule name, and inbox ID. That makes post-drill review much less messy, especally when someone reran the job after the first timeout and forgot to mention it in chat.

If your pipeline fan-outs across a matrix, isolate each inbox per job. The pattern from parallel inbox validation in CI applies here too. Parallel delivery checks are fine; shared inbox state is where things get weird.

Where a throwaway inbox actually helps

This is the only part where a throwaway email address becomes useful, and it should stay a small part of the system. I do not build the workflow around a mailbox vendor. I build it around a delivery contract, then use a disposable inbox because it keeps the proof isolated.

For teams comparing providers, I have seen people test a best throwaway email option during staging drills simply because setup is fast and cleanup is painless. That is fine, but the lasting value is not the brand. The lasting value is that your run owns its own evidence, and no real team inbox gets polluted.

I still drop phrases like dummy e mail into notes sometimes because that is how people search when they are in a hurry, but I would not let those terms shape the architecture. Keep the mailbox replaceable and the assertions strict.

A short checklist before the drill

Before I trust the rehearsal, I want this list green:

• the sample AWS Health event is safe to replay in staging
• each Docker run creates a fresh inbox and records the ID
• the email arrives once, not zero times and not twice
• the message includes enough context for on-call to act fast
• cleanup expires the inbox and keeps artifacts for review

That is usually enough. You do not need a giant notification platform to know whether your maintenance emails are usable. You just need a repeatable check that proves delivery and content at the same time. In my experiance, that simple contract removes a lot of false confidence.

Q&A

Should this run on every commit?

Usually no. I prefer it on merge pipelines, scheduled drills, and pre-maintenance windows. On every commit it adds cost and a bit of noise for not much extra signal.

Why not just mock SES or SNS?

Because the point of the rehearsal is to test the real path humans depend on. Mocks are useful lower in the stack, but they do not prove final delivery.

What if the inbox provider has a bad day?

That can happen, so keep provider-specific logic thin. The portable part is your run ID, assertions, and stored artifacts. Swap the inbox later if needed and the workflow still mostly holds up.