In September I renamed an Argo CD application and took our payments API down for six minutes.
We deploy with an app of apps. A parent Application watches a directory of child Application manifests, one per service, with automated sync and pruning turned on. Each child carries the resources finalizer, copied from Argo's own examples. That finalizer means: when this Application is deleted, delete everything it manages first.
We were preparing a second region, so I renamed the payments application to payments-eu1. One line of the manifest and one file name, approved by two people. To Kubernetes there is no such thing as a rename. There is a new object called payments-eu1 and a missing object called payments.
The parent synced a minute after the merge. It created the new Application, and because pruning was on and the old manifest had gone from git, it deleted the old one. The finalizer did what it was written to do. Argo deleted every resource the old application managed: the Deployment, the Services, the ConfigMaps, the autoscaler. The pods terminated. The new application synced, found nothing there and created all of it again from the same manifests, which took six minutes of image pulls, readiness checks and an autoscaler starting again from its minimum. The only things that survived were the database volumes, because claims created from StatefulSet templates were never in git and Argo had never managed them.
The diff had told the truth. It just did not say delete anywhere, because the deletion was written in a finalizer on another object, not in the change.
Child Application manifests now carry the Prune=false sync option, so the parent never deletes an application because its file disappeared. It marks it out of sync and waits for a person. Front door and stateful resources carry Delete=false, so even a cascade leaves them where they are. Renaming has a written procedure: create the new application, remove the finalizer from the old one, let the new one adopt the resources, and only then remove the old manifest. And a check comments on any pull request that removes or renames an Application file, and requires an approval from the platform team.
In a GitOps repository a file name is not a label. Removing a file is an instruction, and the system will carry it out with exactly the thoroughness you gave it.
– Sergey Shinder
Top comments (0)