At ten to eleven on a Tuesday night in July we had a fix for a broken refund screen, reviewed and built, and our deploy job could not install it. Every attempt failed within a second with UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress. Nothing was in progress, and nothing had been for six hours.
Helm 3 keeps the history of each release as Secrets in the release's namespace, one per revision, and each revision has a status. An upgrade writes a new revision as pending-upgrade, applies the changes, waits if asked to, and then marks it deployed or failed. If the Helm process dies in the middle, nothing is left to write the final status. The revision stays pending-upgrade, and every later upgrade sees a pending operation and refuses to start, on the assumption that someone else is still running it.
Our deploy jobs run on self-hosted CI runners in Kubernetes, and that afternoon the runners had moved to spot instances to save money. At five o'clock a node was reclaimed while a deploy of the payments admin service was waiting for its rollout. Two minutes later the node was gone, and Helm with it. We use the atomic flag, which rolls back a failed upgrade, but that rollback is carried out by the same Helm process, and it was gone. The job showed as cancelled, which read like a hiccup.
At eleven we ran helm rollback to the last deployed revision, which cleared the pending one, and deployed the fix at twenty past. Customers had been without the refund screen for an extra half hour.
Deploy jobs now run on a separate pool of on demand runners, with a pod disruption budget and a grace period longer than our longest rollout. Before every upgrade the deploy job checks the release status, and if it finds a pending revision older than the job's own timeout, it rolls back to the last deployed revision and says so in the deploy channel. A scheduled check reports any release that has been pending for more than thirty minutes. And a cancelled deploy job now pages the person who started it, because cancelled and finished are not the same thing.
A release tool's lock is a note left by a process that expects to come back. When the process cannot come back, somebody has to read the note and throw it away.
– Sergey Shinder
Top comments (0)