Helm Rollback Strategy: Safe Values Promotion in Production

#kubernetes #devops

Originally published on kuryzhev.cloud

One bad helm upgrade and your production values file is history — literally — unless you have a helm rollback strategy that survives concurrent deploys, stateful side effects, and plain human error. I've watched a team roll back to what they *thought* was the last good revision, only to land on a "superseded" release with a broken feature flag still baked in. Here's what we changed after that incident, and what we now enforce before every promotion.

Pin exact chart versions and snapshot values before every promotion

The biggest source of rollback confusion is not knowing what was actually deployed. values.yaml in your chart repo is not the source of truth — the live cluster state can drift from it after hotfixes, manual --set overrides, or an emergency patch someone forgot to commit. Before every helm upgrade, run helm get values <release> -o yaml and store it alongside your CI artifacts, tagged with the git commit SHA and the release revision number. If you can't answer "what values were live five minutes ago" in under 30 seconds, your rollback plan is theoretical, not real.

Roll back by revision number, never by "previous" assumptions

Always run helm history <release> first and confirm the exact revision status before touching anything. helm rollback <release> 0 means "go to previous revision" — and that's dangerous the moment two deploys happen close together, because "previous" might be a failed or superseded revision, not the last known-good one. I stopped trusting revision 0 after a rollback landed us on a revision that had already been marked superseded due to a race between two pipeline runs.

# Wrong: assumes "previous" is safe
helm rollback payments-api 0 -n prod

# Right: explicit revision confirmed via helm history first
helm history payments-api -n prod --max 5
helm rollback payments-api 41 -n prod --wait --timeout 5m

Watch out: without --wait --timeout 5m, Helm — and your CI/CD pipeline — will report success the instant the rollback command is accepted, not when pods are actually healthy. That's how you get a "successful" deploy alert five minutes before the real outage.

Increase --history-max deliberately, but know the storage tradeoff

Helm's default --history-max=10 is fine for low-churn services, but if you've had a bad deploy streak, ten revisions can disappear fast. Each revision is stored as a Kubernetes Secret in the release namespace (sh.helm.release.v1.<name>.v<rev>), so bumping history-max isn't free — on a release doing 50+ deploys a day, that's real etcd bloat and slower helm list/helm history calls. We settled on 15–20 for critical services and purge history entirely on decommissioned releases with helm uninstall --keep-history=false.

Diff before you rollback — don't trust memory

Install the helm-diff plugin (v3.9+) and run helm diff rollback <release> <revision> before you commit to anything. Combine it with helm rollback --dry-run --debug to surface hook ordering or immutable field issues before they hit the cluster. Skipping this because "it's just a rollback" is the mistake I see most often — values promoted since that revision (secrets, replica counts, resource limits) can silently regress and nobody notices until the pods start OOMKilling.

Rollback doesn't undo everything — watch stateful side effects

This is the misconception that bites teams hardest: Helm rollback only reverts the Kubernetes objects Helm tracks. It does not rerun pre-upgrade hooks, does not undo a database migration, and does not restore PVC data. If your Job has helm.sh/hook: pre-upgrade with hook-delete-policy: before-hook-creation, a rollback can leave that Job orphaned in the namespace — audit it manually, don't assume Helm cleaned up after itself. Treating a Helm rollback as equivalent to a full git revert for the entire system is how migrations end up permanently mismatched with the application code they were supposed to support.

Promote values through environments, not through manual edits

Use layered values files — values-base.yaml, values-staging.yaml, values-prod.yaml — merged in a fixed -f order, and never inline --set flags in production pipelines. Promotion should mean copying a validated, already-tested values file from staging to prod through a CI step with a visible diff, not someone re-typing numbers from memory on a Friday afternoon.

# values-base.yaml — shared defaults, no secrets
replicaCount: 2
image:
  repository: registry.internal/payments-api
  tag: "1.8.2"          # pinned explicitly, promoted via CI, never "latest"

resources:
  requests:
    cpu: 250m
    memory: 256Mi

---
# values-prod.yaml — env-specific overrides
replicaCount: 6
resources:
  requests:
    cpu: 500m
    memory: 512Mi
secretsRef:
  # Real values live in a SOPS-encrypted file, referenced here, never inlined
  name: payments-api-secrets-sops

Keep secrets out of these files entirely. Use SOPS or the External Secrets Operator pattern so a rollback never resurrects an old plaintext credential sitting in Secret history from three revisions ago — that's a real security consideration, not a hypothetical one.

Test the rollback path itself, not just the deploy path

Run rollback drills in a staging namespace on a schedule — monthly is enough — to confirm helm rollback actually restores service without someone SSH-ing in to patch things manually. Lock CI/CD concurrency per release with a mutex or a concurrency: group in GitHub Actions so a rollback and a fresh deploy can never race against each other. The worst time to discover your rollback path fails on an immutable field error (looking at you, Deployment selector changes) is during a live incident at 2am.

#!/usr/bin/env bash
# rollback-with-snapshot.sh
# Safe Helm rollback workflow: snapshot current values, diff, confirm, rollback.

set -euo pipefail

RELEASE="payments-api"
NAMESPACE="prod"
SNAPSHOT_DIR="./values-snapshots"

mkdir -p "$SNAPSHOT_DIR"

echo "==> Fetching release history"
helm history "$RELEASE" -n "$NAMESPACE" -o json > "${SNAPSHOT_DIR}/history-$(date +%s).json"

# List the last 5 revisions for operator review
helm history "$RELEASE" -n "$NAMESPACE" --max 5

read -rp "Enter target revision number to rollback to: " TARGET_REV

echo "==> Snapshotting CURRENT live values before touching anything"
helm get values "$RELEASE" -n "$NAMESPACE" -o yaml \
  > "${SNAPSHOT_DIR}/current-before-rollback-$(date +%Y%m%d%H%M).yaml"

echo "==> Snapshotting TARGET revision values for comparison"
helm get values "$RELEASE" -n "$NAMESPACE" --revision "$TARGET_REV" -o yaml \
  > "${SNAPSHOT_DIR}/target-rev-${TARGET_REV}.yaml"

echo "==> Diffing current vs target (requires helm-diff plugin)"
helm diff rollback "$RELEASE" "$TARGET_REV" -n "$NAMESPACE" || true

read -rp "Proceed with rollback to revision ${TARGET_REV}? (yes/no) " CONFIRM
if [[ "$CONFIRM" != "yes" ]]; then
  echo "Aborted."
  exit 1
fi

echo "==> Executing rollback"
helm rollback "$RELEASE" "$TARGET_REV" \
  -n "$NAMESPACE" \
  --wait \
  --timeout 5m \
  --history-max 15

echo "==> Post-rollback verification"
helm status "$RELEASE" -n "$NAMESPACE"
kubectl get pods -n "$NAMESPACE" -l app.kubernetes.io/instance="$RELEASE"

echo "Done. Snapshots saved in ${SNAPSHOT_DIR}/"

If you're running ArgoCD or Flux, this whole helm rollback strategy shifts: prefer git revert plus a sync over a manual helm rollback, so you don't end up with git and cluster state disagreeing about what "current" means. We cover more of that gitops-vs-manual tradeoff in our DevOps_DayS archive if you want the longer version. Whatever pattern you pick, the goal is the same: a rollback should be boring, tested, and never the first time you've actually run the command in anger.