Argo CD Rollback Checklist for Safe GitOps Sync

Originally published on kuryzhev.cloud

---

Why this checklist

Last month a teammate ran argocd app sync straight from the CLI during an incident, skipped the diff review, and 90 seconds later self-heal quietly reverted his emergency fix because Git still pointed at the broken revision. That's the failure mode this Argo CD rollback checklist exists to prevent: ad-hoc syncs under pressure that skip validation and leave drift invisible until the next reconciliation loop fires.

Here's the real cost of getting this wrong. A bad manifest auto-syncs to production before anyone notices — maybe a resource limit got dropped, maybe a ConfigMap reference broke. With selfHeal: true and a default polling interval around three minutes, Argo CD will happily keep re-applying that bad state every time someone tries to patch around it manually with kubectl. What should've been a five-minute rollback turns into an hour of chasing your own tail because nobody reverted Git first.

This checklist is written for teams running Argo CD 2.9+ with automated sync policies across multiple clusters or environments — the point where manual tribal knowledge stops scaling and you need something repeatable pinned to a runbook. If you're still doing single-cluster manual syncs, some of this will feel like overkill, but the rollback discipline items (10-14 below) matter even then.

The checklist (numbered)

Pre-sync checks (1-5)

1. Check app health status first. argocd app get <app> before touching anything — don't sync into a Degraded app assuming a sync will fix it.
2. Review the diff, always. Run argocd app diff <app> --hard-refresh to catch drift from manual kubectl edit sessions that Argo CD hasn't flagged yet.
3. Confirm sync windows are configured if you have blackout periods — a sync during a locked window on a payment service is how you get paged twice.
4. Validate PreSync/PostSync hooks haven't changed between the current and target revision, especially anything with hook-delete-policy: HookSucceeded.
5. Pin the target revision to a tag or SHA in prod. Tracking main or HEAD in production is convenient until someone merges a hotfix directly and it deploys instantly.

Sync execution (6-9)

6. Know what your sync options actually do. Prune=true deletes resources removed from Git — that's not always safe, especially mid-rollback when you might be temporarily out of sync with intent.
7. Respect sync waves. The argocd.argoproj.io/sync-wave annotation matters for rollback too — reverting CRDs after the controllers that depend on them breaks things in a very confusing way.
8. Consider disabling self-heal temporarily during manual incident recovery. Combined with a tight sync interval, self-heal plus a manual fix creates a race condition where your patch gets stomped before you finish typing the next command.
9. Dry-run before you commit. Use --dry-run to preview what a sync will actually change, particularly on large multi-thousand-resource Applications where a full sync spikes kube-apiserver and repo-server CPU.

Rollback readiness (10-14)

10. Set revisionHistoryLimit explicitly. The default is 10 — bump it to 20-30 for critical apps if you need a longer rollback window for audits.
11. Test argocd app rollback in staging before you need it in prod. Rollback only works if the history entry still exists — check with argocd app history <app> first.
12. Treat Git revert as the source of truth, not the UI rollback button. More on why in the next section.
13. Wire up notification hooks on rollback so the team sees it happen in Slack, not just in the audit log three days later.
14. Verify rollback restores both manifests and Helm values/params — not just the chart version reference.

Commonly missed items

The biggest and most common mistake: confusing a UI or CLI rollback with an actual Git revert. When you click "rollback" in the Argo CD UI, it deploys an older revision to the cluster — but it does not touch Git. The next automated sync, especially with selfHeal: true, sees Git still pointing at the broken commit and re-applies it. I've watched this happen twice in the same incident because nobody flagged it the first time. The fix is boring but non-negotiable: revert in Git, push, then let Argo CD sync from the reverted state.

Second gotcha: teams forget the default revisionHistoryLimit of 10. On an app that deploys a dozen times a day, that history rolls off fast. Try to roll back to something from two days ago and you get:

FATA[0000] rpc error: code = NotFound desc = application checkout-service
  does not have deployment id 6 in history

By then it's too late to fix — you needed the higher limit set before the incident, not during it.

Third: self-heal will silently revert out-of-band cluster changes, which sounds great until it masks a real problem. If someone scales a deployment manually to fight a load spike and self-heal reverts it back down five minutes later, that looks like a mystery outage instead of what it actually is.

Fourth, and the one that bites Helm users specifically: rollback restores the chart version and values file reference tracked in Git, but if your values are templated from an external ConfigMap or a Vault secret, those aren't versioned by Argo CD at all. Your manifests roll back cleanly; your actual runtime config doesn't. I stopped trusting "rollback complete" messages after this cost us a config drift bug that took two days to trace.

Also worth flagging on the security side: rollback and sync-override commands should be gated through argocd-rbac-cm. Without scoped policies, anyone with sync permission can force a rollback to an arbitrary revision, bypassing whatever PR review process you think is protecting production. See the Argo CD RBAC docs if you haven't locked this down yet.

Automation ideas

None of this should live in someone's head. The checklist above is a starting point for automation, not a replacement for it.

Start with a CI hook that runs argocd app diff --hard-refresh against the target branch before merge, so drift and unintended changes surface in the PR instead of after deploy. Pair that with an ApplicationSet using the PR generator to spin up preview environments automatically — it removes a lot of the manual sync risk that happens when people test feature branches against shared clusters.

Notifications matter more than most teams give them credit for. The Notifications controller (GA since 2.9) supports triggers like on-sync-failed and on-health-degraded, but on-sync-status-unknown is commonly missed — apps stuck in an Unknown state don't fire on-sync-failed, so they fail silently unless you add explicit alerting for that state.

The most useful thing we built was a rollback wrapper script that enforces Git-first rollback instead of ad-hoc CLI usage. Here's a rollback-safe Application manifest and the wrapper script we use during incidents:

# argocd-app-rollback-safe.yaml
# Example Application manifest with rollback-safe settings applied

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-service
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/checkout-service.git
    targetRevision: main           # pin to a tag/SHA in prod instead of "main" for stricter control
    path: manifests/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: checkout
  syncPolicy:
    automated:
      prune: true                  # WARNING: deletes resources removed from Git — verify before enabling
      selfHeal: true                # disable manually during incident rollback to avoid race conditions
    syncOptions:
      - ApplyOutOfSyncOnly=true     # reduces apply time on large diffs
      - CreateNamespace=false
  revisionHistoryLimit: 20          # default is 10 — bump for longer rollback window on critical apps

#!/usr/bin/env bash
# Rollback runbook wrapper — enforces Git-first rollback instead of UI-only rollback
set -euo pipefail

APP_NAME="checkout-service"
BAD_REVISION=$(git rev-parse HEAD)
PREV_REVISION=$(git rev-parse HEAD~1)

echo "Reverting Git to previous known-good revision: $PREV_REVISION"
git revert --no-edit "$BAD_REVISION"
git push origin main

echo "Forcing Argo CD to pick up the revert immediately"
argocd app sync "$APP_NAME" --revision "$PREV_REVISION" --prune

echo "Verifying health and sync status post-rollback"
argocd app get "$APP_NAME" --refresh

Before running any rollback, cross-check the history against actual Git SHAs so you're not guessing which ID corresponds to which commit:

$ argocd app history checkout-service
ID  DATE                           REVISION
5   2024-05-01 14:22:10 +0000 UTC  a1b2c3d (v1.4.2)
6   2024-05-02 09:11:03 +0000 UTC  d4e5f6a (v1.4.3)
7   2024-05-03 16:45:59 +0000 UTC  9f8e7d6 (v1.5.0)   <-- current, suspected bad deploy

# Rollback to last known-good (ID 6), NOT via UI click — use Git revert first, then:
$ argocd app rollback checkout-service 6

If your history has already rolled off past revisionHistoryLimit, this command fails with the NotFound error shown earlier — another reason to set that limit deliberately rather than trust the default. For the full sync options reference, the official Argo CD sync options docs are worth bookmarking alongside this checklist.

This Argo CD rollback checklist won't prevent every bad deploy — nothing does. But it turns rollback from a stressful, memory-dependent scramble into a scripted, auditable process, which is the whole point of GitOps in the first place. If you're building out broader CI/CD guardrails around this, our CI/CD checklist and quality gates post covers the pipeline side that feeds into this rollback flow.

Related

• More GitOps workflow patterns and Argo CD deep-dives
• Kubernetes reliability and cluster operations guides
• CI/CD pipeline design, gates, and rollback strategies