ECR to Kubernetes: Keep a Deploy Receipt

The release was green, the Kubernetes rollout said complete, and the on-call engineer still could not answer one basic question: which image was actually running?

That gap usually starts with a harmless-looking Docker tag such as :main or :latest. Amazon ECR has the immutable digest, but the pipeline report keeps only the tag. Kubernetes then records a deployment revision, while the incident ticket contains a commit SHA copied from a different screen. Each system has part of the story. None of them has the receipt.

I have had better results treating every container release as a small evidence bundle. The goal is not more logging for its own sake. It is to make rollback and incident review possible without reconstructing the deploy from memory. This sounds simple, but it get overlooked when a team is moving quickly.

The image tag is not a deployment receipt

Tags are useful for humans, but they are weak evidence. A mutable tag can point at a new digest later, and a CI retry may push a second image under the same name. If the production manifest says api:main, you know what someone intended to deploy, not necessarily what the kubelet pulled.

The minimum useful identity has three parts:

• the source commit SHA
• the ECR image digest
• the Kubernetes workload and namespace that accepted the image

Add the workflow run ID and deployment timestamp too. Those fields make the record searchable when releases become more noisey than expected. I also include the actor or service principal, because an emergency manual push should not look identical to a normal pipeline.

What the receipt should prove

A deploy receipt should answer these questions in one screen:

1. What source revision produced the image?
2. What exact digest was pushed to ECR?
3. Which environment was targeted?
4. Did Kubernetes observe that digest on ready pods?
5. What command or workflow can roll back to the previous known-good digest?

The receipt should be boring and specifc. A JSON artifact works well because it can be attached to the CI run and indexed by a small internal tool. For example:

{
  "service": "checkout-api",
  "commit": "8f31c2a",
  "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/checkout-api",
  "digest": "sha256:example-digest",
  "environment": "production",
  "workflow_run": "1842",
  "namespace": "checkout",
  "deployment": "checkout-api",
  "observed_at": "2026-09-27T11:10:00Z"
}

Do not put secrets, registry tokens, or full environment dumps in it. The receipt is for identity and decision making, not a second log archive.

Build the receipt in CI

After the Docker build, push the image, then resolve the digest from the registry response instead of guessing it from the tag. Pass that digest to the deployment step and save the same value as an artifact.

The deployment input should look like this:

IMAGE="123456789012.dkr.ecr.us-east-1.amazonaws.com/checkout-api"
DIGEST="sha256:example-digest"

kubectl -n checkout set image deployment/checkout-api \
  api="${IMAGE}@${DIGEST}"
kubectl -n checkout rollout status deployment/checkout-api --timeout=5m

Using image@sha256:... is the important part. It removes ambiguity at the point where Kubernetes creates pods. The pipeline can still publish a friendly tag for browsing, but the workload should use the digest.

Keep run artifacts that explain a CI result next to the receipt, not in a separate dashboard that expires sooner. When a build fails, engineers should see the build inputs, image identity, and rollout output together. That small delay are worth it during an incident.

Verify the digest inside Kubernetes

rollout status proves that the controller completed its rollout. It does not by itself prove that every ready pod is running the digest you expected. I add a second check that reads the pod image IDs and compares them with the receipt.

kubectl -n checkout get pods \
  -l app=checkout-api \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].imageID}{"\n"}{end}'

The imageID normally contains the registry digest. Fail the job if a ready pod reports a different value, or if the selector finds no ready pods. This catches a surprising number of bad manifests, stale selectors, and namespace mistakes.

For a safe rollback, retain the previous receipt for the same deployment. A rollback should be a choice between two known identities, not a guess at which old tag was still in ECR. If the pod is healthly but the application is wrong, the receipt gives the operator a clean boundary for comparing the two versions.

Where temporary email fits in release testing

Email verification is often part of a release smoke test, especially for signup, password reset, and invitation flows. It should be treated as a separate test dependency with its own run ID, retention rule, and cleanup. A preflight check before shipping can create the inbox, trigger the message, and record only the evidence needed for the test.

For isolated non-production checks, a team may use tempmailso or a disposable email generator as the mailbox fixture. The important part is to keep that address out of production data and to attach the message ID or assertion result to the same CI receipt. I have seen a ticket labelled tamp mail com during a rushed triage; clear fixture naming makes those handoffs less confusing.

Do not make mailbox availability the only signal for a successful deploy. If the image digest is wrong, a passing email assertion does not rescue the release. The infrastructure receipt and the application smoke-test result should remain separate fields that are evaluated together.

A short incident checklist

When a release looks suspicious, I check these in order:

1. Compare the receipt commit with the change under review.
2. Compare the recorded ECR digest with the deployment manifest.
3. Read pod imageID values from the target namespace.
4. Check whether the rollout used the intended cluster context.
5. Run the application smoke test and match its run ID to the receipt.
6. Roll back using the previous digest if the evidence disagrees.

This is not a replacement for observability. It is the small contract between Docker, AWS, CI/CD, and Kubernetes that observability often assumes but does not enforce. Once the receipt is automatic, the release process feels a bit calmer: fewer screenshots, fewer guesses, and less time spent asking who pushed what.