In September we moved invoice reminders from an old CronJob in our billing namespace to the new billing service, which sends them from its own scheduler. The release was two pull requests: one turned the new reminders on, and one deleted the CronJob's manifest from the billing repository. Both merged on the same afternoon. Argo CD synced, the new reminders started, and we closed the ticket.
Six weeks later a customer forwarded us two reminders for the same invoice, sent on the same morning eleven minutes apart and worded slightly differently. One came from the new service. The other came from a CronJob we had deleted.
The billing Application had automated sync turned on, which we had read as Argo CD making the cluster match Git. It does, for everything that is in Git. Automated sync does not delete a resource that has disappeared from Git unless prune is also turned on, and prune is off by default, as a safety measure, so that a mistake in a repository cannot quietly delete production resources. The CronJob was still in the cluster, still on its schedule, sending reminders every weekday at eight.
Argo CD had told us. The Application had been OutOfSync since the afternoon of the release, with one resource marked as requiring pruning. The badge had been yellow for so long that people had stopped seeing it, and several of our other Applications were yellow too, for harmless differences nobody had cleaned up.
We deleted the CronJob by hand that morning and wrote to every customer who had received both reminders. Then we went through every Application and found seven more resources waiting to be pruned, including the Deployment and Ingress of a promotions page we had retired in the spring, still serving it.
Prune is now on for every Application, with the Prune=false sync option set on the few resources, such as persistent volume claims, that we never want removed automatically. Any Application that stays OutOfSync for more than an hour alerts its owning team, and we fixed the harmless differences so that a yellow badge means something again. Retiring a component now has its own checklist step: confirm in the cluster, not in Git, that it is gone.
We had treated deleting a file as the release. With prune off, it was a request waiting for someone to confirm it.
– Sergey Shinder
Top comments (1)
The line that would have told you sooner is in the application controller log. When the only drift is resources waiting to be pruned and automated prune is off, Argo CD logs "Skipping auto-sync: need to prune extra resources only but automated prune is disabled" and stops there. It never becomes an event or a condition, so nobody sees it.
The Application status does carry it. Every entry in status.resources that is waiting gets requiresPruning: true, so one read-only command lists the leftovers across all Applications:
It's worth running before you turn prune on anywhere, because that list is what the first sync will delete.