ECS Rollbacks Need Email Correlation IDs

I learned this one the annoying way. An ECS deploy rolled back, the alert email arrived, and half the team still spent 20 minutes asking whether the message belonged to the failed rollout or to an older retry. The mail was real, but the evidence around it was weak.

What finally worked was adding one correlation ID that followed the deploy from the pipeline, into the container release metadata, through the notification event, and into the email subject and headers. That sounds small, but it changed rollback triage a lot. Instead of arguing over screenshots, we could tie the message back to one release attempt and move on.

Why rollback emails become guesswork

A lot of teams still treat rollback mail like a courtesy signal. If an email lands, someone assumes the rollback was detected and routed correctly. In practice, that breaks down pretty fast:

• retries reuse the same subject line
• test and production alerts share one mailbox
• the notification body forgets which task definition failed
• a manual resend muddies the timeline

The result is noisy ops. I have seen scratch notes with terms like tem email and dummy e mail in on-call docs because people were trying to remember which mailbox or alias was "safe" for a quick test. Thats usually a sign the pipeline is doing too much hand-waving and not enough recording.

This is also why I like reading patterns outside the cloud bubble once in a while. The QA angle in email retries without false positives maps well to infra work: if your signal cannot prove which run it belongs to, retries become misleading instead of helpful.

The correlation ID pattern that fixed it

The pattern is simple and pretty cheap to maintain:

1. Generate a deploy-scoped correlation ID in CI/CD.
2. Inject it into the ECS deployment metadata and notification payload.
3. Include it in the email subject and a custom mail header.
4. Record the same ID in CloudWatch logs and your rollback event store.
5. Alert on missing or mismatched IDs, not just on "email sent."

For us, the useful format was:

service=payments env=prod deploy=8f31c2a attempt=2

That string is boring on purpose. Operators can read it fast, grep it fast, and paste it into CloudWatch, SNS logs, or a Slack thread without cleanup. Fancy UUIDs work too, but short deploy-aware IDs are easier when teh pager already went off.

The other key point is scope. The ID should belong to the deploy attempt, not only to the mail event. If you attach it too late, the notification system becomes the source of truth and your rollout record stays fuzzy.

A small ECS implementation

You do not need a giant platform project for this. A shell step and a tiny mail payload change gets most of the value.

set -euo pipefail

DEPLOY_SHA="${GITHUB_SHA::7}"
DEPLOY_ATTEMPT="${GITHUB_RUN_ATTEMPT:-1}"
CORRELATION_ID="payments-prod-${DEPLOY_SHA}-${DEPLOY_ATTEMPT}"

aws ecs update-service \
  --cluster prod-apps \
  --service payments \
  --force-new-deployment \
  --region us-east-1

aws sns publish \
  --topic-arn "$ROLLBACK_TOPIC_ARN" \
  --subject "Rollback detected: ${CORRELATION_ID}" \
  --message "{\"service\":\"payments\",\"correlation_id\":\"${CORRELATION_ID}\",\"state\":\"rollback_detected\"}"

On the mail side, I like adding the ID in two places:

• subject line for humans
• custom header for machines

If you use SES or another provider that preserves custom headers, the next step is easy. Your parser or evidence collector can index X-Correlation-ID and attach the exact message to the rollback record. That ended up being more durable than trying to infer intent from timestamps alone, which was always a bit janky.

Container teams can borrow one more idea from test automation here. The fixture CLI for smoke tests article is about APIs, but the mindset is solid: create small, repeatable setup inputs so every run leaves predictable evidence. Rollback mail should be treated the same way.

Operational checks that matter

Once the correlation ID exists, I would verify four things before trusting the workflow:

1. The ID is created before the deploy starts.
2. The same ID appears in deployment logs, rollback events, and the email.
3. A resend cannot silently replace the original context.
4. Retention is short enough that old evidence does not look current.

This matters more than polishing the HTML template. During incidents, nobody cares if the email looked slick. They care whether the message points to the right container image, cluster, and attempt number.

I also stopped letting shared inboxes act like audit systems. They are decent for spot checks, but poor as a long-term source of truth. If the mailbox is the only place your team can confirm a rollback, the workflow is still fragile, full stop. It also gets noisey when multiple services share the same notification path.

One more practical tip: log the ECS task definition revision in the same record. Rollbacks often look identical at the subject line level, but the revision number shows whether you reverted to :142 or :141. That tiny detail saves surprizing amounts of time in postmortems.

Q&A

Why not rely on CloudWatch timestamps only?

Because timestamps tell you when something happened, not whether two systems are talking about the same deploy attempt. Correlation IDs connect the events instead of asking humans to guess.

Should every email include the full deploy context?

No. Keep the body short and link to the richer record. The email should help the operator route and verify the event, not become a second dashboard.

Does this only help AWS teams?

Nope. I used ECS here because that is where I felt the pain, but the pattern works anywhere you can stamp one deploy-scoped identifier across release events, logs, and outbound mail. The implementation is small, the payoff is real, and it keeps rollback triage a lot less messy.