How to recover a Helm 3 release stuck in pending-upgrade

A Helm release stuck in pending-upgrade blocks helm upgrade with another operation (install/upgrade/rollback) is in progress, and the useful surprise is that it does not block helm rollback. The pending check lives in prepareUpgrade and guards concurrent upgrades; pkg/action/rollback.go has no equivalent. So try helm rollback to the last good revision FIRST. Deleting a release-history Secret is a real mutation of Helm's storage and it is almost never the answer. A stuck pending-install on revision 1 has no earlier revision to roll back to, but helm uninstall has no pending check either and clears the abandoned objects with it, so that case is an uninstall and a reinstall. On a payments platform we work with, that was revision 47 sitting pending for 22 minutes after the CI runner was killed mid-upgrade, with the retry set to resolve the pipeline's chart version range to a newer patch, one that had added a values.schema.json the production values file did not satisfy. This guide is the order we run it in, with the two moves that will cost you a resource if you get them backwards.

Problem signals:

• helm upgrade exits immediately with Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
• helm history -o json shows the newest revision as pending-upgrade with no deployed revision above it
• The release does not appear in helm list -n <ns> at all, only in helm list -a -n <ns>
• A Secret the new chart version re-templated exists but its key decodes to 0 bytes, and the pods CrashLoop on auth
• helm upgrade with the existing production values file returns values don't meet the specifications of the schema(s) in the following chart(s)

Why does helm upgrade say another operation is in progress?

Read the history before you type rollback

The lock is a stored revision record, not a process holding a mutex.

This guide assumes Helm 3 driving the chart directly, from CI or from a laptop, with the default secret storage driver, against EKS 1.29. If Argo CD renders your chart and applies the manifests itself, none of what follows applies, because in that mode there is no Helm release history to inspect and the recovery is a git and sync-status problem instead.

Helm does not hold a lock in memory. It writes a Secret per revision, named sh.helm.release.v1.<release>.v<n>, and every operation reads the newest one first. If that newest record says pending-upgrade, helm upgrade concludes an operation is still running somewhere and refuses to start another. The CI job that wrote it may have been killed 22 minutes ago. Helm has no way to know that, so the release stays wedged until you change the record.

The first surprise is that the release can look like it does not exist. helm list -n settlement filters to deployed and failed releases, so a pending one is simply absent from the output, which sends people looking for a deleted release that is sitting right there. Run helm list -a -n settlement instead and it appears.

# The two cheap reads. Do both before you touch anything.
helm history payments-worker -n settlement -o json \
  | jq -r '.[] | "\(.revision)\t\(.status)\t\(.description)"'

# Same answer straight from the storage layer, no Helm binary involved:
kubectl get secret -n settlement \
  -l owner=helm,name=payments-worker \
  -L version,status --sort-by=.metadata.creationTimestamp

helm history renders an aligned table by default, which is useless in a ticket. The -o json form pipes cleanly, and the kubectl form works when the Helm binary is arguing with you.

In our case the JSON came back with revision 46 still marked deployed and revision 47 as pending-upgrade, description Preparing upgrade. That is the shape you are looking for: a head revision in a pending state, with the last good revision still deployed beneath it. Helm demotes the previous revision to superseded only on the success path, so a run killed mid-apply leaves 46 exactly where it was. If instead the head revision says failed, you are in a different and much easier situation, because Helm will happily accept a helm rollback against a failed release without any of the surgery below.

The rollback is the first branch. A stuck pending-install on revision 1 has no earlier revision to roll back to, so it goes to helm uninstall, not to a hand-edit of Helm storage.

Try the rollback first, then clear the record only if you must

Back up every release Secret first

Start with the rollback, because the pending guard does not apply to it. helm rollback payments-worker 46 -n settlement --wait --timeout 5m writes a new revision rather than destroying one. If it returns cleanly, the release is deployable again; if the head it rolled back from was itself a rollback, read the note on what 47 leaves behind in the next section.

We did not start there. On the night this happened we went straight to clearing the record by hand, and the rollback we ran afterwards is the one that fixed it. Reading pkg/action/rollback.go later made the point uncomfortable: there is no pending check in the rollback path, so that rollback would have worked on its own and, unless it stopped partway, worked better, because the Secret delete changed what it diffed against. It is worth being plain about that, because the procedure below circulates as folklore and most of the people running it do not need to.

You need it in one situation, and it is narrower than the folklore suggests. When the head revision is pending-install on revision 1 there is no earlier revision to roll back to, but that does not make a hand-edit of storage the answer. helm uninstall carries no pending check either, uninstall.go tests only for an already-uninstalled release, and it clears both the release record and the objects the half-finished install left behind. Uninstall, then install again. Deleting the Secret by hand is defensible only when those objects have to stay in place. A reinstall adopts objects that already carry this release's ownership metadata, which a Helm install stamps on before creating anything; objects without it need --take-ownership, which landed in Helm 3.17. A failed rollback is NOT that situation. Rollback.Run calls Releases.Create with pending-rollback before it touches any resource, and on failure records that same revision as failed, so after a rollback that errors your head is usually a failed rollback revision (a failed hook or a killed client can leave it pending-rollback), and the fix is at the resource level, not in storage. Re-read helm history before you conclude otherwise.

Before deleting a revision record, take the whole history to a file. This is the one step that turns an irreversible mistake into an inconvenient one, and it costs six seconds.

kubectl get secret -n settlement \
  -l owner=helm,name=payments-worker -o json \
  | jq 'del(.items[].metadata.resourceVersion, .items[].metadata.uid,
         .items[].metadata.creationTimestamp, .items[].metadata.managedFields)' \
  > /tmp/payments-worker-helm-history.json

# Confirm the file has every revision that still exists, not just the ones you remember.
# With the default --history-max 10 this lists about ten, not one per revision.
jq -r '.items[].metadata.name' /tmp/payments-worker-helm-history.json

The jq filter is the point. A plain get -o yaml carries resourceVersion, which the API server refuses outright on a create, plus uid and creationTimestamp that it would silently overwrite, so the untouched dump restores nothing at the moment you need it.

Now confirm which revision is actually stuck, and confirm it from the payload rather than the label. The release body inside that Secret is not JSON and you cannot patch it as JSON. Helm gzips the release JSON, base64 encodes the result itself, and then Kubernetes base64 encodes the whole thing again into data.release. Open the Secret in an editor and you get an opaque blob. Anyone who tells you to flip the status field from pending-upgrade to failed with a kubectl patch has not tried it.

kubectl get secret sh.helm.release.v1.payments-worker.v47 \
  -n settlement -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gunzip | jq -r '.info.status, .info.description'

Two base64 decodes, then gunzip. One decode gives you more base64 and people assume the data is corrupt.

That printed pending-upgrade and Preparing upgrade, which matched the label. Read what follows as the record of what we ran that night, not as the step to copy. We deleted that single Secret, which made revision 46 the head of the history again and let Helm treat the release as one it could operate on. With 46 sitting there deployed, the rollback on its own would have recovered the release and, unless its first attempt stopped before finishing its resource update (next section), done it better: after the delete, the rollback diffed 46 against 46, so anything revision 47 had added, objects and fields both, was left in place. It is written out because these are the commands that circulate as folklore, and the two judgment calls buried in them are worth having before you are somewhere you need them. If your own head revision is pending and an earlier revision is beneath it, stop at the rollback.

kubectl delete secret sh.helm.release.v1.payments-worker.v47 -n settlement

helm rollback payments-worker 46 -n settlement --wait --timeout 5m

Delete the record, not the release: helm uninstall here would take the workload down with it, which is why the uninstall route belongs to a stuck pending-install on revision 1 and not to this. This pair is what we ran, not what this case needs. A pending head with an earlier revision under it stops at the rollback.

Two judgment calls in that pair of commands. Delete exactly one Secret, the pending one, and name it in full. A label selector delete against owner=helm,name=payments-worker removes the entire history including the revision you are about to roll back to, and then your only route home is the backup file you just wrote. Second, name the target revision explicitly. helm rollback payments-worker -n settlement with no revision argument rolls back to the revision before the head. That is the trap after a delete: with 47 gone the head IS 46, so a bare rollback targets 45 and quietly skips the revision you were aiming for. Type the number.

We use --wait --timeout 5m rather than a bare rollback because a rollback that returns success while pods are still terminating tells you nothing. With --wait, Helm returns only after the Deployment reports enough replicas ready (desired minus its maxUnavailable), so a non-zero exit is real information. The cost is honest: for as long as the rollback actually runs, up to five minutes for the wait plus up to five for each step of each rollback hook, no helm upgrade against that release will start. A second rollback or an uninstall still would, since neither carries the pending guard.

What 'no ConfigMap with the name X found' means during a rollback

When rollback fails on a live resource Helm has no record of

The second failure people hit is a rollback that gets past the lock and then dies on a specific resource. The message reads like Helm is confused about what exists. It is not.

Error: no ConfigMap with the name "payments-worker-broker" found

Helm 3 through 3.21 emits this from its update path when the object exists in the cluster but has no entry in the release record it is diffing against. A rollback prints it bare, as here, and records its own Rollback "payments-worker" failed: ... as the revision description, not on stderr. 3.22.0 does not stop on this condition: it notes it only under --debug and patches the object against its live state, which can drop fields another controller set on it, such as an HPA-managed replicas.

Get the direction right, because getting it backwards sends you looking in the wrong place for an hour. Update in Helm 3's pkg/kube/client.go takes the previous release's manifest and the target manifest, and walks the target. It looks each resource up live first, and a miss there is harmless: Helm creates the object and carries on. The error comes one step later, when it looks the same resource up in the previous release's manifest and finds nothing. So the object is in the cluster, and in the manifest being applied, and absent from the record Helm is comparing against. That is what a rollback to revision 46 runs into when revision 47's record, the one being rolled back from, does not list an object that 46 does. One way to get here is a kill during the pre-upgrade hooks or the apply itself, before Helm's delete pass, which leaves one of 46's own objects that 47's chart dropped; an object something outside the release created is the other.

The recovery is to run the rollback again with the revision typed, helm rollback payments-worker 46 -n settlement --wait --timeout 5m, and to delete nothing before it. The failed attempt has already written its own revision, a failed rollback whose manifest is revision 46's, and the retry diffs against that record, so this time the object is found in it and patched. That failed attempt is now the head, so a bare helm rollback would target 47, the revision that wedged. Read what revision 46 expects with helm get manifest payments-worker --revision 46 -n settlement, and 47's with --revision 47. Check the meta.helm.sh/release-name annotation first, on every object 46's manifest has and 47's does not, not only the one in the error, so you know they belong to this release: the first attempt stopped at the first such object, and the retry adopts all of them. If one names another release, or none, find out what put it there before you roll back again. Any rollback that diffs 46 against 46 can leave part of 47 in place: one retried after a first attempt that stopped before finishing its resource update, as a rollback that stops on this error always does on 3.21 and earlier, one run after a deleted record, and one from a head that was itself a rollback to 46. After a deleted record, --revision 47 no longer holds 47: the rollback took that number for its own record, a copy of 46's, so decode 47's manifest from the backup file as the FAQ on deleting the Secret shows. None of these rollbacks prunes objects only 47 created, and each can leave fields and list entries 47 added, and on custom resources the values 47 changed, because Helm patches those from the record alone. So once the release reports deployed, compare 47's manifest with 46's and with what is live, and put back by hand what still differs, with the care of any manual change to production: leave replicas on a workload an HPA targets, which would reset to 1, leave any object annotated helm.sh/resource-policy: keep, and never delete a CustomResourceDefinition without checking what else uses it. The retry can itself fail where keeping both versions of a field is invalid, an env var whose value 47 swapped for valueFrom for instance; the error names the field, so put it back to 46's form on the live object in one change and run the retry again. Do not reach for --force on the rollback, and the reason is sharper than "it is risky": on 3.21 and earlier it does nothing at all on this path. force is only consulted inside updateResource, and this error never gets that far: the visitor checks the cluster with helper.Get first, then looks the resource up in the previous release's manifest, and returns no %s with the name %q found from that second lookup. The flag sits downstream of the point where it fails. Where --force does apply it sends a full replace (helper.Replace, a PUT) instead of patching, which discards fields another controller owns, an HPA-managed replicas being the usual casualty, and fails outright on immutable resources such as a Job.

There is a related trap on the workload itself. A chart bump can change the template of a Secret the release already had, renaming the values key it reads for instance, so the key renders empty because the production values never set the new key: the Secret exists, the Deployment mounts it, and the key inside is an empty string. Kubernetes is perfectly happy with that. The pods are not, and you get a CrashLoopBackOff whose logs blame the broker rather than the chart. Check the length, not the presence.

# Presence proves nothing. Length does.
kubectl get secret payments-worker-broker-auth -n settlement \
  -o jsonpath='{.data.password}' | base64 -d | wc -c
# 0

# After a rollback to the last good revision, the same command returns the real byte count.

A Secret that decodes to 0 bytes passes every existence check and fails every connection.

Verification is four reads and they should all agree. helm status payments-worker -n settlement -o json | jq -r '.info.status' returns deployed. helm history payments-worker -n settlement -o json | jq -r '.[-1].status' returns deployed for the head revision. The old pending entry is still listed if you got here by rolling back, because a rollback appends a revision rather than removing one; it is gone only if you deleted its Secret. kubectl get pods -n settlement -l app.kubernetes.io/name=payments-worker shows every pod Running with a restart count that stops climbing over the next few minutes. And helm get manifest against the head revision matches what is live, which is the check that catches a rollback that succeeded on paper while something else was quietly reconciling the cluster back.

How do you fix a chart schema error without deleting the schema?

Getting the values file past values.schema.json

The release is deployable again, and now you have the next problem: the retry resolved the pipeline's chart version range to a newer patch, and that patch ships a values.schema.json your production values file does not satisfy. Helm validates values against that schema in prepareUpgrade, before it writes a revision or writes anything to the API server. That ordering matters more than it looks: a run that fails schema validation creates no release record at all, so a schema error can never be what left you in pending-upgrade. Revision 47's chart version carried no schema. The newer patch had been published since 47's run. Revision 47 wedged because the runner was killed; the newer patch's schema was waiting to break the retry, which is exactly what it did.

Error: UPGRADE FAILED: values don't meet the specifications of the schema(s) in the following chart(s):
payments-worker:
- at '/broker/pool/maxIdle': got string, want integer

The chart did not change what the field means. It started enforcing a type that was previously accepted as free text. This is the wording from Helm 3.18.5 on; earlier 3.x releases print - broker.pool.maxIdle: Invalid type. Expected: integer, given: string.

That is the whole class of failure. A field that was unstructured for two years held "25" in quotes because someone templated it out of a CI variable, and the new schema declares it an integer. Nothing about the running workload was wrong. The schema simply started looking.

Fix it locally before you go near the cluster, against the chart version the retry resolved: helm repo update, then helm pull <repo>/payments-worker --version '<the pipeline's version range>' --untar --untardir ./retry-chart (a fresh directory, because --untar refuses to write over an existing payments-worker directory inside it, which a vendored ./charts already holds; the range resolves to the patch the retry got unless a newer one has been published since), then helm lint ./retry-chart/payments-worker -f values/production.yaml runs the same schema validation without a Kubernetes connection, which turns a ten minute deploy-and-fail cycle into a two second one. Once lint is clean, helm upgrade payments-worker ./retry-chart/payments-worker -n settlement -f values/production.yaml --dry-run=server renders with lookup functions resolved against the cluster. (helm upgrade reads the cluster's API versions for .Capabilities in either mode, so a template reaching for an API version the cluster has dropped fails both.) It does not submit anything for admission review, so a Gatekeeper or Kyverno rejection is still waiting for you on the real upgrade; to cover that, run helm template payments-worker ./retry-chart/payments-worker -n settlement -f values/production.yaml | kubectl apply -n settlement --server-side --force-conflicts --dry-run=server -f -. helm template renders without the cluster, so anything that depends on lookup or .Capabilities can render differently from what the upgrade would submit. The server-side form of --dry-run arrived in Helm 3.13, so on older clients you get the client-side render only.

We do not delete or blank out the values.schema.json in a vendored copy of the chart to make the error go away. We have inherited two clusters where someone did exactly that, and in both the next chart bump reintroduced the schema and the same incident happened again with a different on-call engineer and no memory of the first one. Fix the values file. If the schema itself is genuinely wrong for your use, pin the chart version, open the issue upstream, and write the pin's reason in the values file where the next person will read it.

For prevention, helm upgrade --atomic helps, but not with the failure in this story. It implies --wait and rolls the release back if the upgrade does not converge, and that rollback is issued by the Helm client process itself. Kill that process and nothing is left to issue it, so a hard-killed CI job still leaves the release pending. What guards against this specific case is the runner's own limit. --timeout applies to each step separately, the hooks and the wait, and with --atomic the rollback after a failure gets full timeouts of its own, so the runner's job limit has to exceed that whole sequence, not the flag. Set it shorter and it kills Helm partway through, which is precisely how the release ends up pending. Helm 3.7 and later trap SIGTERM: without --atomic they record the release as failed and exit, and with --atomic they then run the rollback, waiting up to --timeout per step. Where the runner lets you set its SIGTERM-to-SIGKILL grace, size it to that rollback, or the kill lands mid-rollback and leaves the release pending-rollback. On Helm 4 the flag is --rollback-on-failure, and --atomic survives as a deprecated alias (on helm install only from 4.1.2). The tradeoff is real and we tell clients about it up front: --atomic holds the release for the entire timeout window, so on a rollout that takes four minutes with a ten minute timeout, a failed deploy can block the next one for the full ten minutes plus however long the rollback takes, up to another ten and more with hooks, rather than failing fast. That is usually the right trade for a payments path and usually the wrong one for a batch worker that deploys thirty times a day. We walk through where that line sits per service in our Kubernetes and CI/CD stabilization work.

Common questions about clearing a stuck Helm release

What people ask after the first rollback lands

• Will helm rollback work while the release is still in pending-upgrade? Yes, and it is the first thing to try. The guard is upgrade-only: prepareUpgrade returns errPending when lastRelease.Info.Status.IsPending(), and its own comment scopes it to concurrent upgrades acting as a pessimistic lock. pkg/action/rollback.go carries no such check, so a rollback to a known-good revision runs and leaves a new deployed revision as the head. The pending row keeps its pending-upgrade status in the history, because a successful rollback supersedes only revisions that were already deployed, and it blocks nothing once the guard reads a deployed head. (A rollback that fails while applying resources marks it superseded; one that fails its --wait or a hook leaves it pending-upgrade.) When there is no prior revision at all, a pending-install on revision 1, the answer is still not storage surgery: helm uninstall has no pending check either and takes the abandoned objects with it, so uninstall and install again. A rollback that ERRORS is not that case: it has usually already written its own revision (failed, or pending-rollback after a hook failure or a kill), so re-read the history before touching storage.
• Is deleting a sh.helm.release.v1 Secret safe? Deleting one revision record removes Helm's memory of that revision and touches no running resource at that moment. It is not free, though: the rollback that follows then diffs 46 against 46, so objects revision 47 added stay live, and so can fields it added, where a rollback alone diffs 47 against 46 and removes them. That holds when the first rollback attempt gets through its resource update; if it failed before that and you ran it again, the retry diffs 46 against 46 too, on any version. If you do delete one, delete by full name, never by label selector, and keep the backup file until you have put back what 47 left, not just until helm status reports deployed: from the moment of the delete it is the only copy of 47's manifest, and a rollback then reuses number 47 for its own record. Delete the file once you are done, since it holds every rendered Secret of the release. If you already deleted the record, decode 47's manifest from the backup (not from helm get manifest --revision 47, nor from a dump taken after the rollback: once you have rolled back after the delete, the v47 record is the rollback's own) with jq -r '.items[] | select(.metadata.name=="sh.helm.release.v1.payments-worker.v47") | .data.release' /tmp/payments-worker-helm-history.json | base64 -d | base64 -d | gunzip | jq -r .manifest and compare it with helm get manifest payments-worker --revision 46 -n settlement and with what is live, object by object, to find what to put back, with the same three exceptions as the rollback-error section: replicas on a workload an HPA targets, objects annotated helm.sh/resource-policy: keep, and any CustomResourceDefinition until you have checked what else uses it.
• Can I just patch the status field inside the release Secret? The payload is gzipped JSON, base64 encoded by Helm and then base64 encoded again by Kubernetes, so there is no JSON there to patch. If you want to inspect it, use the decode round trip above. To change the release state, use the supported route: roll back to the last good revision.
• Does any of this apply if Argo CD manages the app? No. When Argo CD renders the chart and applies the manifests itself, there is no Helm release history in the cluster to repair. The failure looks similar and the fix is entirely different.
• My rollback succeeded but the pods still CrashLoop. Check the Secret and ConfigMap byte lengths, not their existence. A key that decodes to 0 bytes satisfies every reference check the Deployment does and fails at connection time, which is why the logs blame a downstream service.

When the stuck release is on the payments path and nobody wants to type delete

If the release has been pending for an hour

The hard part of this procedure is not the commands. It is that the move that circulates is kubectl delete secret against a revision record on a production release, typed with an engineering lead watching at the point in an incident where confidence is lowest, when a rollback alone would do. The failure modes are unforgiving in a specific way: a label-selector delete that takes the whole history, a rollback to an implicit revision that lands one further back than you meant, a --force that replaces a StatefulSet and drops fields you needed left alone. Every one of those is recoverable if the backup exists and unrecoverable if it does not.

We do this work with teams who deploy through Helm from CI and have never had to open a release Secret before. Our part is being the second pair of eyes on the rollback, reading the history with you before anything is typed, and then leaving behind the --atomic and timeout settings that stop the same job from wedging the release next quarter. Most of these calls run under an hour once we can see helm history output.

If a release is pending right now and you would rather not be the one deciding what to run, book an infrastructure review and we will get on a call and work through the history with you the same day. If it is already recovered and you want the CI path hardened so it does not recur, that is the same conversation with less adrenaline, and it is the shape of work described in our Kubernetes release failure playbook.

---

Originally published at https://infraforge.agency/insights/helm-rollback-failed-release-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.