Originally published on kuryzhev.cloud
A solid Helm rollback strategy is usually missing the night a chart upgrade reaches production and the pods start crash-looping. Picture a typical case. The chart was upgraded in staging, and the team copied a few overrides across. The production command then ran with --reuse-values because it "keeps what is already there". The new chart version expects a key that production's stored values never had. Rendering either fails outright or, worse, succeeds with an empty string where a database mode should be.
Failure scenario: a promotion that rolls back to the wrong thing
Consider a service chart moving from 1.8 to 1.9. Version 1.9 adds database.sslMode with a sensible default and renames ingress.class to ingress.className. Staging was upgraded from a values file in Git and works. Production was upgraded by an engineer running something like this:
# Risky: renders the new chart with the LAST release's values, not the chart's new defaults
helm upgrade payments myrepo/payments --version 1.9.0 \
--namespace prod --reuse-values --set image.tag=2.14.0
With --reuse-values, Helm takes the previous release's values and uses them in place of the new chart's values.yaml defaults. Keys that exist only in the new chart therefore get no default. The new template reads .Values.database.sslMode, gets nothing, and the app starts with an invalid connection string. The deployment stalls and alerts fire.
The instinctive fix is helm rollback payments. That helps only partly. It restores the previous revision's manifests and values. It does not undo a database migration that already ran in a hook, a CRD that was applied, or a Secret rotated outside Helm. The release is back to "deployed", yet nobody is sure what state the system is really in.
Why it happens
Several documented behaviors combine here, and none is a bug.
Watch out for --reuse-values. It reuses the stored values of the last release, merged with any new --set or -f overrides. The new chart's default values are not applied, so newly added or renamed keys come through empty. Helm 3.14 added --reset-then-reuse-values, which resets to the new chart's defaults and then reapplies your previous overrides. Verify the exact semantics in the helm upgrade reference for your version.
Second, a Helm rollback is a new revision, not time travel. helm rollback payments 7 creates revision N+1 using the manifest and values stored in revision 7. Only objects Helm manages are touched. Files under a chart's crds/ directory are installed once and, per the documented behavior, are not upgraded or deleted by Helm. Hooks such as migration Jobs are not reversed.
Third, history can disappear. Helm keeps a limited number of revisions per release (the --history-max default is 10). A noisy pipeline that retries upgrades can push the last good revision out of the window.
Another gotcha: if the latest revision is in failed state, the "previous" revision may itself be a failed attempt. Always read helm history before choosing a number.
The fix: a Helm rollback strategy built on pinned inputs
The goal has three parts:
- Every production release is reproducible from Git.
- Every release is validated before it applies.
- A release that never becomes ready is reverted automatically.
Promote a values file and chart version together, not a live release's state.
Layer the values so the environment file contains only real differences:
# values/base.yaml - shared, promoted unchanged from staging to prod
replicaCount: 2
database:
sslMode: require
ingress:
className: nginx
# values/prod.yaml - only genuine environment differences
replicaCount: 6
ingress:
host: payments.example.com
Add a schema to the chart so a missing or renamed key fails at render time instead of at runtime. Helm validates values.schema.json on install, upgrade, lint, and template:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"required": ["database", "ingress"],
"properties": {
"database": {
"type": "object",
"required": ["sslMode"],
"properties": { "sslMode": { "enum": ["disable", "require", "verify-full"] } }
},
"ingress": {
"type": "object",
"required": ["className"]
}
}
}
Then run the same sequence in every environment. The script pulls the exact chart version once, so lint, dry-run, diff, and upgrade all operate on the same pinned artifact. The flags below are the Helm 3 spellings. Helm 4 renames some of them (for example, --atomic becomes --rollback-on-failure), so check helm upgrade --help for your installed version.
set -euo pipefail
CHART_REF=oci://registry.example.com/charts/payments
CHART_VERSION=1.9.0
IMAGE_TAG=2.14.0
helm pull "$CHART_REF" --version "$CHART_VERSION" --untar --untardir ./build
CHART=./build/payments
VALUES=(-f values/base.yaml -f values/prod.yaml --set image.tag="$IMAGE_TAG")
# schema + template checks
helm lint "$CHART" "${VALUES[@]}" --strict
# API server validation
helm template payments "$CHART" "${VALUES[@]}" -n prod \
| kubectl apply --dry-run=server -n prod -f -
# helm-diff plugin: review the real change
helm diff upgrade payments "$CHART" "${VALUES[@]}" -n prod
# auto-rollback if not Ready within the timeout
helm upgrade --install payments "$CHART" "${VALUES[@]}" \
-n prod --atomic --wait --timeout 5m --history-max 20
When a manual rollback is still needed, inspect first and roll back to a known-good revision explicitly. helm get values shows the user-supplied values for that revision. Add --all to see the fully computed set.
helm history payments -n prod # find last revision with status "deployed"
helm get values payments -n prod --revision 7 # confirm what that revision actually contained
helm rollback payments 7 -n prod --wait --timeout 5m --cleanup-on-fail
See the helm rollback reference for the full flag list.
Handling what Helm does not roll back
Treat schema changes as a separate, forward-compatible track. Use the expand-then-contract pattern:
- Release N adds the new column and still works with the old code.
- A later release removes the old column.
A Helm rollback of the application then never meets a database it cannot read.
For CRDs, manage them in a separate release or pipeline step, and version them independently of the application chart. Do not assume that rolling back a chart rolls back its CRDs.
Pods that are stuck can also be reverted at the workload level with kubectl rollout undo, as described in the Kubernetes Deployment documentation. That bypasses Helm's history, though, so reconcile Git afterward.
Prevention checklist
Use this as a pipeline gate or a pull request template for any chart promotion. More operational write-ups live on the DevOps_DayS home page.
[ ] Chart version pinned (helm pull --version) and recorded in Git with the values files
[ ] Promotion copies base.yaml unchanged; prod.yaml holds only real differences
[ ] No --reuse-values in CI; if overrides must persist, use --reset-then-reuse-values
[ ] values.schema.json required keys cover every key the templates read
[ ] helm lint --strict and server-side dry-run pass before upgrade
[ ] helm diff output reviewed with the same values and --set flags as the upgrade
[ ] --atomic (or --rollback-on-failure in Helm 4) with --wait and an explicit --timeout
[ ] --history-max high enough to keep several known-good revisions
[ ] CRDs and DB migrations shipped in separate, backward-compatible steps
[ ] Readiness probes reflect real dependencies, so --wait means something
[ ] Rollback rehearsed in staging: helm history, helm get values, helm rollback
[ ] After any manual rollback, Git is updated to match the cluster state
The last item matters most. Suppose a rollback leaves Git saying 1.9 while the cluster runs 1.8. That mismatch is a second incident waiting for the next pipeline run. Make the revert a commit, let the pipeline apply it, and keep the manual command for emergencies only.
Finally, rehearse. A rollback you have only read about is likely to fail on its first real use. Common causes are a missing revision, a hook side effect, or a flag that changed between Helm versions. Run the drill against staging whenever you upgrade Helm itself.
Top comments (0)