How to recover a Helm 3 release stuck in pending-upgrade

A Helm release stuck in pending-upgrade blocks helm upgrade with another operation (install/upgrade/rollback) is in progress, and the useful surprise is that it does not block helm rollback. The pending check lives in prepareUpgrade and guards concurrent upgrades; pkg/action/rollback.go has no equivalent. So try helm rollback to the last good revision FIRST. Deleting a release-history Secret is a real mutation of Helm's storage and it is almost never the answer. A stuck pending-install on revision 1 has no earlier revision to roll back to, but helm uninstall has no pending check either and clears the abandoned objects with it, so that case is an uninstall and a reinstall. On a payments platform we work with, that was revision 47 sitting pending for 22 minutes after the CI runner was killed mid-upgrade, with a chart bump that had added a values.schema.json the production values file no longer satisfied waiting to break the retry. This guide is the order we run it in, with the two moves that will cost you a resource if you get them backwards.

Problem signals:

• helm upgrade exits immediately with Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
• helm history -o json shows the newest revision as pending-upgrade with no deployed revision above it
• The release does not appear in helm list -n <ns> at all, only in helm list -a -n <ns>
• A Secret created by the half-finished upgrade exists but its key decodes to 0 bytes, and the pods CrashLoop on auth
• helm upgrade with the existing production values file returns values don't meet the specifications of the schema(s) in the following chart(s)

Why does helm upgrade say another operation is in progress?

Read the history before you type rollback

The lock is a stored revision record, not a process holding a mutex.

This guide assumes Helm 3 driving the chart directly, from CI or from a laptop, with the default secret storage driver, against EKS 1.29. If Argo CD renders your chart and applies the manifests itself, none of what follows applies, because in that mode there is no Helm release history to inspect and the recovery is a git and sync-status problem instead.

Helm does not hold a lock in memory. It writes a Secret per revision, named sh.helm.release.v1.<release>.v<n>, and every operation reads the newest one first. If that newest record says pending-upgrade, helm upgrade concludes an operation is still running somewhere and refuses to start another. The CI job that wrote it may have been killed 22 minutes ago. Helm has no way to know that, so the release stays wedged until you change the record.

The first surprise is that the release can look like it does not exist. helm list -n settlement filters to deployed and failed releases, so a pending one is simply absent from the output, which sends people looking for a deleted release that is sitting right there. Run helm list -a -n settlement instead and it appears.

# The two cheap reads. Do both before you touch anything.
helm history payments-worker -n settlement -o json \
  | jq -r '.[] | "\(.revision)\t\(.status)\t\(.description)"'

# Same answer straight from the storage layer, no Helm binary involved:
kubectl get secret -n settlement \
  -l owner=helm,name=payments-worker \
  -L version,status --sort-by=.metadata.creationTimestamp

helm history renders an aligned table by default, which is useless in a ticket. The -o json form pipes cleanly, and the kubectl form works when the Helm binary is arguing with you.

In our case the JSON came back with revision 46 still marked deployed and revision 47 as pending-upgrade, description Preparing upgrade. That is the shape you are looking for: a head revision in a pending state, with the last good revision still deployed beneath it. Helm demotes the previous revision to superseded only on the success path, so a run killed mid-apply leaves 46 exactly where it was. If instead the head revision says failed, you are in a different and much easier situation, because Helm will happily accept a helm rollback against a failed release without any of the surgery below.

The rollback is the first branch. A stuck pending-install on revision 1 has no earlier revision to roll back to, so it goes to helm uninstall, not to a hand-edit of Helm storage.

Try the rollback first, then clear the record only if you must

Back up every release Secret first

Start with the rollback, because the pending guard does not apply to it. helm rollback payments-worker 46 -n settlement --wait --timeout 5m writes a new revision rather than destroying one. If it returns cleanly, you are done.

We did not start there. On the night this happened we went straight to clearing the record by hand, and the rollback we ran afterwards is the one that fixed it. Reading pkg/action/rollback.go later made the point uncomfortable: there is no pending check in the rollback path, so that rollback would have worked on its own and the Secret delete bought us nothing. It is worth being plain about that, because the procedure below circulates as folklore and most of the people running it do not need to.

You need it in one situation, and it is narrower than the folklore suggests. When the head revision is pending-install on revision 1 there is no earlier revision to roll back to, but that does not make a hand-edit of storage the answer. helm uninstall carries no pending check either, uninstall.go tests only for an already-uninstalled release, and it clears both the release record and the objects the half-finished install left behind. Uninstall, then install again. Deleting the Secret by hand is defensible only when those objects have to stay in place, and then you still owe Helm an adoption step: --take-ownership on the next install or upgrade, which landed in Helm 3.17. A failed rollback is NOT that situation. Rollback.Run calls Releases.Create with pending-rollback before it touches any resource, and on failure records that same revision as failed, so after a rollback that errors your head is a failed rollback revision and the fix is at the resource level, not in storage. Re-read helm history before you conclude otherwise.

Before deleting a revision record, take the whole history to a file. This is the one step that turns an irreversible mistake into an inconvenient one, and it costs six seconds.

kubectl get secret -n settlement \
  -l owner=helm,name=payments-worker -o json \
  | jq 'del(.items[].metadata.resourceVersion, .items[].metadata.uid,
         .items[].metadata.creationTimestamp, .items[].metadata.managedFields)' \
  > /tmp/payments-worker-helm-history.json

# Confirm the file has every revision that still exists, not just the ones you remember.
# With the default --history-max 10 this lists about ten, not one per revision.
jq -r '.items[].metadata.name' /tmp/payments-worker-helm-history.json

The jq filter is the point. A plain get -o yaml carries resourceVersion, which the API server refuses outright on a create, plus uid and creationTimestamp that it would silently overwrite, so the untouched dump restores nothing at the moment you need it.

Now confirm which revision is actually stuck, and confirm it from the payload rather than the label. The release body inside that Secret is not JSON and you cannot patch it as JSON. Helm gzips the release JSON, base64 encodes the result itself, and then Kubernetes base64 encodes the whole thing again into data.release. Open the Secret in an editor and you get an opaque blob. Anyone who tells you to flip the status field from pending-upgrade to failed with a kubectl patch has not tried it.

kubectl get secret sh.helm.release.v1.payments-worker.v47 \
  -n settlement -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gunzip | jq -r '.info.status, .info.description'

Two base64 decodes, then gunzip. One decode gives you more base64 and people assume the data is corrupt.

That printed pending-upgrade and Preparing upgrade, which matched the label. Read what follows as the record of what we ran that night, not as the step to copy. We deleted that single Secret, which made revision 46 the head of the history again and let Helm treat the release as one it could operate on. With 46 sitting there deployed, the rollback on its own would have done the same work, so the delete bought us nothing. It is written out because these are the commands that circulate as folklore, and the two judgment calls buried in them are worth having before you are somewhere you need them. If your own head revision is pending and an earlier revision is beneath it, stop at the rollback.

kubectl delete secret sh.helm.release.v1.payments-worker.v47 -n settlement

helm rollback payments-worker 46 -n settlement --wait --timeout 5m

Delete the record, not the release: helm uninstall here would take the workload down with it, which is why the uninstall route belongs to a stuck pending-install on revision 1 and not to this. This pair is what we ran, not what this case needs. A pending head with an earlier revision under it stops at the rollback.

Two judgment calls in that pair of commands. Delete exactly one Secret, the pending one, and name it in full. A label selector delete against owner=helm,name=payments-worker removes the entire history including the revision you are about to roll back to, and then your only route home is the backup file you just wrote. Second, name the target revision explicitly. helm rollback payments-worker -n settlement with no revision argument rolls back to the revision before the head. That is the trap after a delete: with 47 gone the head IS 46, so a bare rollback targets 45 and quietly skips the revision you were aiming for. Type the number.

We use --wait --timeout 5m rather than a bare rollback because a rollback that returns success while pods are still terminating tells you nothing. With --wait, Helm returns only after the Deployment reports its expected replicas ready, so a non-zero exit is real information. The cost is honest: for as long as the rollback actually runs, up to that five minute ceiling, no helm upgrade against that release will start. A second rollback or an uninstall still would, since neither carries the pending guard.

What 'no ConfigMap with the name X found' means during a rollback

When rollback fails on a live resource Helm has no record of

The second failure people hit is a rollback that gets past the lock and then dies on a specific resource. The message reads like Helm is confused about what exists. It is not.

Error: no ConfigMap with the name "payments-worker-broker" found

Helm 3 emits this from its update path when the object exists in the cluster but has no entry in the release record it is diffing against. A rollback prints it bare, as here, while the same condition reached through helm upgrade arrives prefixed with UPGRADE FAILED:. Match on the no <Kind> with the name "<name>" found body, which is stable across both, rather than on the prefix. Rollback records its own Rollback "payments-worker" failed: ... as the revision description, not on stderr.

Get the direction right, because getting it backwards sends you looking in the wrong place for an hour. Update in Helm 3's pkg/kube/client.go takes the previous release's manifest and the target manifest, and walks the target. It looks each resource up live first, and a miss there is harmless: Helm creates the object and carries on. The error comes one step later, when it looks the same resource up in the previous release's manifest and finds nothing. So the object is in the cluster, and in the manifest being applied, and absent from the record Helm is comparing against. That is what a rollback to revision 46 runs into when something outside the release created the object, or an earlier half-finished upgrade left it behind without recording it.

The recovery is to delete the live object and run the rollback again. Deleting it makes Helm's live lookup miss, and a miss is the harmless path: Helm creates the resource from revision 46's manifest and records it properly this time. Read what revision 46 expects first with helm get manifest payments-worker --revision 46 -n settlement, so you know what is about to be recreated and can confirm nothing else depends on the object's current contents. Do not reach for --force on the rollback, and the reason is sharper than "it is risky": on this path it does nothing at all. force is only consulted inside updateResource, and this error never gets that far: the visitor checks the cluster with helper.Get first, then looks the resource up in the previous release's manifest, and returns no %s with the name %q found from that second lookup. The flag sits downstream of the point where it fails. Where --force does apply it sends a full replace (helper.Replace, a PUT) instead of patching, which discards fields another controller owns, an HPA-managed replicas being the usual casualty, and fails outright on immutable resources such as a Job or a Service clusterIP.

There is a related trap on the workload itself. A half-finished upgrade frequently creates a Secret from a template whose input value never rendered, so the Secret exists, the Deployment mounts it, and the key inside is an empty string. Kubernetes is perfectly happy with that. The pods are not, and you get a CrashLoopBackOff whose logs blame the broker rather than the chart. Check the length, not the presence.

# Presence proves nothing. Length does.
kubectl get secret payments-worker-broker-auth -n settlement \
  -o jsonpath='{.data.password}' | base64 -d | wc -c
# 0

# After a clean rollback or upgrade, the same command returns the real byte count.

A Secret that decodes to 0 bytes passes every existence check and fails every connection.

Verification is four reads and they should all agree. helm status payments-worker -n settlement -o json | jq -r '.info.status' returns deployed. helm history payments-worker -n settlement -o json | jq -r '.[-1].status' returns deployed for the head revision. The old pending entry is still listed if you got here by rolling back, because a rollback appends a revision rather than removing one; it is gone only if you deleted its Secret. kubectl get pods -n settlement -l app.kubernetes.io/name=payments-worker shows every pod Running with a restart count that stops climbing over the next few minutes. And helm get manifest against the head revision matches what is live, which is the check that catches a rollback that succeeded on paper while something else was quietly reconciling the cluster back.

How do you fix a chart schema error without deleting the schema?

Getting the values file past values.schema.json

The release is deployable again, and now you still have the original problem: the chart version you were upgrading to ships a values.schema.json that your production values file does not satisfy. Helm validates values against that schema in prepareUpgrade, before it writes a revision or touches the API server. That ordering matters more than it looks: a run that fails schema validation creates no release record at all, so a schema error can never be what left you in pending-upgrade. Revision 47 wedged because the runner was killed; the schema was waiting to break the retry, which is exactly what it did.

Error: UPGRADE FAILED: values don't meet the specifications of the schema(s) in the following chart(s):
payments-worker:
- broker.pool.maxIdle: Invalid type. Expected: integer, given: string

The chart did not change what the field means. It started enforcing a type that was previously accepted as free text.

That is the whole class of failure. A field that was unstructured for two years held "25" in quotes because someone templated it out of a CI variable, and the new schema declares it an integer. Nothing about the running workload was wrong. The schema simply started looking.

Fix it locally before you go near the cluster. helm lint ./charts/payments-worker -f values/production.yaml runs the same schema validation without a Kubernetes connection, which turns a ten minute deploy-and-fail cycle into a two second one. Once lint is clean, helm upgrade payments-worker ./charts/payments-worker -n settlement -f values/production.yaml --dry-run=server renders with a cluster connection, so lookup functions resolve and .Capabilities.APIVersions reflects what the cluster actually serves instead of Helm's built-in defaults. That catches a template reaching for an API version the cluster has dropped. It does not submit anything for admission review, so a Gatekeeper, Kyverno or pod-security rejection is still waiting for you on the real upgrade; to cover that, run helm template ... | kubectl apply --server-side --dry-run=server -f -. The server-side form of --dry-run arrived in Helm 3.13, so on older clients you get the client-side render only.

We do not delete or blank out the values.schema.json in a vendored copy of the chart to make the error go away. We have inherited two clusters where someone did exactly that, and in both the next chart bump reintroduced the schema and the same incident happened again with a different on-call engineer and no memory of the first one. Fix the values file. If the schema itself is genuinely wrong for your use, pin the chart version, open the issue upstream, and write the pin's reason in the values file where the next person will read it.

For prevention, helm upgrade --atomic helps, but not with the failure in this story. It implies --wait and rolls the release back if the upgrade does not converge, and that rollback is issued by the Helm client process itself. Kill that process and nothing is left to issue it, so a hard-killed CI job still leaves the release pending. What guards against this specific case is the opposite ordering to the one people reach for: Helm's --timeout must expire BEFORE the CI runner's own job timeout. Then Helm gives up on its own terms, unwinds, and exits. Set the runner's limit shorter and it kills Helm partway through, which is precisely how the release ends up pending. On Helm 4 the flag is --rollback-on-failure; --atomic is deprecated on helm upgrade and is an unknown flag on helm install. The tradeoff is real and we tell clients about it up front: --atomic holds the release for the entire timeout window, so on a rollout that takes four minutes with a ten minute timeout, a failed deploy blocks the next one for ten minutes rather than failing fast. That is usually the right trade for a payments path and usually the wrong one for a batch worker that deploys thirty times a day. We walk through where that line sits per service in our Kubernetes and CI/CD stabilization work.

Common questions about clearing a stuck Helm release

What people ask after the first rollback lands

• Will helm rollback work while the release is still in pending-upgrade? Yes, and it is the first thing to try. The guard is upgrade-only: prepareUpgrade returns errPending when lastRelease.Info.Status.IsPending(), and its own comment scopes it to concurrent upgrades acting as a pessimistic lock. pkg/action/rollback.go carries no such check, so a rollback to a known-good revision runs and leaves a new deployed revision as the head. The pending row keeps its pending-upgrade status in the history, because performRollback supersedes only revisions that were already deployed, and it blocks nothing once the guard reads a deployed head. When there is no prior revision at all, a pending-install on revision 1, the answer is still not storage surgery: helm uninstall has no pending check either and takes the abandoned objects with it, so uninstall and install again. A rollback that ERRORS is not that case: it has already written its own failed revision, so re-read the history before touching storage.
• Is deleting a sh.helm.release.v1 Secret safe? Deleting one revision record removes Helm's memory of that revision. It does not touch a single running resource. The danger is scope, not the act: delete by full name, never by label selector, and keep the backup file until helm status reports deployed.
• Can I just patch the status field inside the release Secret? The payload is gzipped JSON, base64 encoded by Helm and then base64 encoded again by Kubernetes, so there is no JSON there to patch. If you want to inspect it, use the decode round trip above. To change the release state, use the supported route: clear the pending revision, then roll back.
• Does any of this apply if Argo CD manages the app? No. When Argo CD renders the chart and applies the manifests itself, there is no Helm release history in the cluster to repair. The failure looks similar and the fix is entirely different.
• My rollback succeeded but the pods still CrashLoop. Check the Secret and ConfigMap byte lengths, not their existence. A key that decodes to 0 bytes satisfies every reference check the Deployment does and fails at connection time, which is why the logs blame a downstream service.

When the stuck release is on the payments path and nobody wants to type delete

If the release has been pending for an hour

The hard part of this procedure is not the commands. It is that the correct move is kubectl delete secret against a revision record on a production release, with an engineering lead watching, at the point in an incident where confidence is lowest. The failure modes are unforgiving in a specific way: a label-selector delete that takes the whole history, a rollback to an implicit revision that lands one further back than you meant, a --force that replaces a StatefulSet and drops fields you needed left alone. Every one of those is recoverable if the backup exists and unrecoverable if it does not.

We do this work with teams who deploy through Helm from CI and have never had to open a release Secret before. Our part is being the second pair of eyes on the delete, reading the history with you before anything is typed, and then leaving behind the --atomic and timeout settings that stop the same job from wedging the release next quarter. Most of these calls run under an hour once we can see helm history output.

If a release is pending right now and you would rather not be the one running the delete, book an infrastructure review and we will get on a call and work through the history with you the same day. If it is already recovered and you want the CI path hardened so it does not recur, that is the same conversation with less adrenaline, and it is the shape of work described in our Kubernetes release failure playbook.

---

Originally published at https://infraforge.agency/insights/helm-rollback-failed-release-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.