Oleksandr Kuryzhev

Posted on Oct 18 Originally published at kuryzhev.cloud

Argo CD rollback: git revert vs argocd app rollback

#devops

Originally published on kuryzhev.cloud

A bad image tag merges at 16:50, auto-sync applies it within minutes, and the checkout service starts returning 500s. Someone has to choose an Argo CD rollback method right now: revert the commit in Git, or run argocd app rollback against the cluster. Both work, but they behave very differently around auto-sync, audit history, and what the next sync does to your fix.

This comparison covers the two approaches as classes, not as step-by-step runbooks. Check the details against the official Argo CD documentation for your installed version, since sync and history behavior has changed across releases.

When your Argo CD rollback choice matters

On a quiet day, either method looks fine. The choice matters when time pressure, auto-sync, and team process collide.

Argo CD treats Git as the desired state. A rollback that does not touch Git creates drift between what the cluster runs and what the repository says. Automated sync with selfHeal enabled closes that gap by re-applying the broken commit. Without selfHeal, the drift persists until the next new commit or a manual sync, and either one re-applies whatever Git says. That is the central tension of this topic.

The decision also depends on a few conditions:

Sync policy: automated with selfHeal, automated without it, or manual.
Branch protection: whether a revert needs a reviewed pull request and how long that takes.
Blast radius: one application, or an app-of-apps or ApplicationSet with many children.
Stateful changes: database migrations, CRD changes, or PreSync hooks that do not undo themselves.

Here is a typical automated policy, the setup where the choice bites hardest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://git.example.com/platform/deploy.git
    targetRevision: main        # tracks a branch, so every merge is a deploy
    path: apps/checkout
  destination:
    server: https://kubernetes.default.svc
    namespace: checkout
  syncPolicy:
    automated:
      prune: true               # deletes resources removed from Git
      selfHeal: true            # reverts manual cluster changes

Option A: Git revert, let Argo CD sync it

The first approach is to revert the offending commit on the tracked branch and let the controller converge. Git stays the single source of truth throughout.

# revert a regular (non-merge) commit
git revert <commit-sha>

# or, for a merge commit, -m 1 picks the mainline parent
git revert -m 1 <merge-commit-sha>

git push origin main

# optional: skip the polling interval instead of waiting for it
argocd app get checkout --refresh

Pros:

Git and cluster stay aligned, so auto-sync and selfHeal keep working in your favor.
The revert is a reviewable, attributable commit, which suits audit and change-management requirements.
It scales to many apps: a single revert in a shared config repo reaches every ApplicationSet-generated app that renders from it. Per-app repos or overlays need one revert each.
It does not depend on Argo CD's stored deployment history, which is capped by revisionHistoryLimit (verify the default for your version).

Cons:

It is slower. Branch protection, required reviews, and CI checks all sit between you and the fix.
Unless a webhook or a manual refresh triggers it, the controller waits for its next poll of the repository. The default interval is roughly three minutes; verify it in your config.
Watch out for reverting a merge without -m, or reverting a commit that later commits depend on. Conflicts at 2 a.m. are a real cost.

Option B: argocd app rollback to a prior deployment

The second approach uses Argo CD's recorded sync history. Each history entry records the source and Git revision that was deployed. You pick an earlier deployment ID, and Argo CD regenerates the manifests for that revision and applies them, regardless of what the tracked branch currently says.

# list recorded deployments with their IDs and Git revisions
argocd app history checkout

# the CLI refuses to roll back while automated sync is enabled, so disable it first
# (this removes the automated block, including prune and selfHeal)
argocd app set checkout --sync-policy none

# re-deploy the revision recorded as history ID 42
argocd app rollback checkout 42

# only after the Git revert has merged, restore the original policy
argocd app set checkout --sync-policy automated --auto-prune --self-heal

Pros:

It is fast and needs no pull request, so it fits an incident where minutes matter.
It targets a revision Argo CD previously synced successfully, not one you have to reconstruct by hand.
It acts on one application and is easy to explain to an on-call engineer.

Cons:

Watch out for auto-sync. The documented behavior is that rollback is not allowed on an app with automated sync enabled. If you re-enable it before fixing Git, the controller re-applies the broken commit.
If the Application is generated by an ApplicationSet, or managed by a parent app with selfHeal, the owning controller may revert your sync-policy change. Check how your setup preserves manual changes before relying on this path.
The cluster now runs something Git does not describe. That drift is intentional but easy to forget, and the app shows as out of sync.
History is limited and per-application. Behavior with multi-source applications has varied by version, so verify before relying on it.
It still needs the old revision to be reachable. Argo CD's repo-server fetches it from Git, or serves it from its cache if the cache is still warm.
Rollback in Argo CD is not a data rollback. A migration run by a hook stays applied, and a deleted PVC stays deleted.

Decision matrix

Match your situation to the row that fits best. The columns reflect the documented behavior of each approach, not measured timings.

Condition	Git revert	argocd app rollback
Auto-sync + selfHeal on, many apps	Fits naturally	Needs sync policy changes first, per app
Active incident, PR approval slow	Slower	Faster
Strict audit trail required	Strong (commit history)	Weaker unless you record the action
Git outage or repo unreachable	Not possible	Not guaranteed; depends on repo-server cache
Old target beyond history limit	Works	May not be available
Risk of leaving cluster drifted	Low	High if not followed by a Git fix

As a quick checklist before you choose:

1. Is automated sync enabled on this app?          yes -> Git revert is the default
2. Is the incident costing money every minute?      yes -> consider app rollback, then fix Git
3. Can a revert PR merge within your incident SLO?  yes -> Git revert
4. Did the bad release run a migration or hook?     yes -> neither undoes it; plan a forward fix
5. After app rollback, who owns the Git revert?     name a person before you start

Evidence-based recommendation

For most teams running automated sync, make Git revert the standard Argo CD rollback path. It keeps the declared state and the live state aligned, leaves a reviewable record, and avoids the trap of the controller re-deploying the bad version. Pair it with webhooks from your Git host and a pre-approved fast path for revert PRs, such as a documented emergency merge rule, so the speed gap narrows.

Keep argocd app rollback as a break-glass tool, not a routine one. Use it when an incident is actively burning budget and the review path cannot move fast enough. Treat it as a three-step operation:

Run the rollback itself.
Merge the Git revert immediately.
Only then, re-enable automated sync with the same prune and selfHeal settings you had before.

Skipping the Git revert is a typical failure mode: sync is turned back on and the broken release returns.

Two conditions change this advice. First, if you run manual sync on production, app rollback is less risky because nothing will silently overwrite it. Second, if your riskiest changes involve stateful components, neither method is the real safety net. In that case, look at progressive delivery with Argo Rollouts, which can abort and roll back a release under Argo CD's management. Be careful with the native Kubernetes Deployment rollback. On an Argo CD-managed app, kubectl rollout undo is just another form of drift that selfHeal will revert. Also design migrations to be backward compatible so a rollback is safe in the first place.

Whichever you pick, rehearse it in a non-production cluster and write the exact commands into your runbook. For related platform topics, browse the other guides on kuryzhev.cloud.

DEV Community