Kubernetes Deploys Need Failure Receipts

#aws #cicd #devops #kubernetes

A failed Kubernetes deployment is not a diagnosis. It is only a status change.

That distinction became obvious while operating EKS releases through CI/CD. A pipeline would turn red, the rollout command would time out, and the team would start asking the same questions: which pod failed, what image was running, did the readiness probe ever pass, and did the cluster reject the manifest? The answer was often spread across several terminals and, sometimes, already gone.

I now treat every deployment as producing a small failure receipt. It is a durable bundle of facts from the run: commit, image, namespace, rollout state, recent events, and selected logs. It makes the next investigation much less guessy. It also works better than searching old test labels such as fake e mail com when a shared QA inbox is part of the release flow.

The failure signal is not enough

kubectl rollout status is useful, but it answers a narrow question: did the rollout reach the requested state before the timeout? It does not explain why the state was not reached.

Common causes include:

a container image that cannot be pulled
a readiness probe pointed at the wrong path
a missing Secret or ConfigMap key
insufficient CPU or memory in the node group
an application that starts but never becomes ready

Those causes need different fixes. A single red check mark hides that difference, so the first task is to collect evidence while the objects still exist in the same state as the incident.

I keep the receipt close to the workflow run. For longer investigations, the same approach pairs well with restore context for operational emails and trace-friendly CI fixtures: each artifact should say which scenario and revision produced it.

What a deployment receipt should contain

The useful minimum is small enough to collect on every run:

Git SHA and workflow run ID.
Image digest, not just the mutable image tag.
Cluster, namespace, and deployment name.
Rollout status and the deployment YAML summary.
Pod descriptions, recent warning events, and container termination reasons.
A bounded sample of logs from pods created by this revision.

I avoid dumping the entire cluster. That creates noisy artifacts and can expose unrelated data. A receipt should be scoped to the workload and trimmed to a known size. The command below is intentionally boring:

set -o pipefail

kubectl -n "$NAMESPACE" describe deployment "$DEPLOYMENT" > receipt/deployment.txt
kubectl -n "$NAMESPACE" get pods -l app="$APP" -o wide > receipt/pods.txt
kubectl -n "$NAMESPACE" get events --sort-by=.lastTimestamp \
  --field-selector involvedObject.name="$DEPLOYMENT" > receipt/events.txt
kubectl -n "$NAMESPACE" logs -l app="$APP" --all-containers \
  --since=10m --tail=300 > receipt/logs.txt || true

The || true on logs is deliberate. A pod may be gone, or the container may never have started. That should not erase the more important deployment and event evidence. The collection step can report that logs were unavailable inside the receipt itself.

Capture evidence in GitHub Actions

A deployment job should always upload the receipt, including on failure. In GitHub Actions, the key is putting collection in an if: always() step after the rollout attempt:

- name: Roll out
  id: rollout
  run: kubectl -n "$NAMESPACE" rollout status deployment/checkout --timeout=180s

- name: Collect deployment receipt
  if: always()
  run: ./ci/collect-deployment-receipt.sh

- name: Upload deployment receipt
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: deployment-receipt-${{ github.run_id }}
    path: receipt/
    retention-days: 14

The rollout can still fail the job. always() only guarantees that the evidence step gets a chance to run. Keep credentials out of the artifact: redact environment dumps, avoid kubectl get secret -o yaml, and check logs for tokens before uploading them.

For a successful rollout, the receipt is still valuable. It records what was deployed and gives a baseline for later comparisons. For a failed rollout, it turns a vague timeout into a short list of testable hypotheses. That is a much better handoff between the pipeline and the person on call.

Use Kubernetes events carefully

Events are excellent clues but poor long-term storage. They can be evicted, coalesced, or replaced as the cluster changes. I collect them during the run and never assume they will still be available when someone opens the ticket tomorrow.

I also capture the image digest from the running pod, because checkout:latest is not an audit trail. If the digest is missing, that itself is worth recording as a release control failure. In AWS environments, I keep the ECR image URI and digest beside the Git SHA in the workflow summary, then compare those values during rollback.

A small operating checklist

Before calling the workflow complete, I check:

Does every rollout attempt upload an artifact?
Can the receipt identify one cluster, namespace, workload, and revision?
Are events and logs bounded in size?
Are secrets and authorization headers excluded?
Can another engineer reproduce the first diagnostic commands?
Is the artifact retained long enough for the team’s review cycle?

This is not a replacement for metrics, tracing, or proper alerting. It is the cheap evidence layer between a CI/CD failure and a useful investigation. A green deploy should leave a baseline; a failed deploy should leave clues. With that contract in place, Kubernetes debugging feels less like archaeology and more like operations.

DEV Community