The Drift Problem That Breaks GitOps at Scale

The hard part of GitOps at scale is not the sync engine. It is drift: everything that changes your cluster after Argo CD says it is in sync. If your first post on why LinkedIn outgrew Argo CD covered when the engine breaks, this covers the quieter failure -- the one that makes your Git repo lie to you.

By the end you will be able to identify the three main sources of drift, write a diff job that catches them, and decide when self-heal does more harm than good.

The three sources of drift

Most teams blame engineers running kubectl edit. That is real, but it is the smallest share in a large cluster.

• Controllers that add defaults, inject sidecars, or mutate resources on every reconcile loop.
• Deploy pipelines that patch live manifests for canaries, rollbacks, or emergency fixes.
• Operators that manage subresources and rewrite status or spec fields continuously.

Each of these produces a live state that differs from Git, and each needs a different response.

A quick triage table

Source	What it looks like	Correct response
Mutating webhook	Sidecar appears, env vars added	IgnoreKnownFields or owner-reference the injected bits
Deploy pipeline patch	Image tag, replica count, or command differs	Move the patch into Git or into a kustomize overlay
Operator-owned fields	Fields flip back seconds after sync	Remove those fields from your manifests entirely

The third row is the one teams miss. If an operator owns a field, Argo CD will fight it forever, report OutOfSync, and your alerting learns to ignore that app. Once alert fatigue sets in, real drift gets through.

Why self-heal makes it worse

The default advice is to set selfHeal: true and let Argo CD revert everything. That works until a deploy pipeline needs to roll back quickly. If your rollback tool patches the live Deployment and self-heal reverts it within seconds, your rollback silently never happened.

A policy that works in practice:

• Self-heal enabled for app-of-apps and platform namespaces.
• Self-heal disabled for namespaces where deploy tooling legitimately patches live state.
• A diff job that reports drift in self-heal-disabled apps without reverting it.

Write the drift job

You do not need new infrastructure to catch drift. Argo CD already computes a diff; you just need to read it on a schedule and alert on the right things.

#!/usr/bin/env bash
set -euo pipefail

APP="$1"

argocd app get "$APP" --refresh --output json \
  | jq -r '
    .status.conditions[]?
    | select(.type == "OrphanedResource" or .type == "Progressing")
    | .message'

# Compare live state to target state via the API
argocd app diff "$APP" --server-side-generate=false \
  || echo "DRIFT detected in $APP"

Run this from CronJob or CI every few minutes. The point of argocd app diff is that it respects your ignore-differences config, so you only alert on drift that matters. Pipe the output to Slack with an app label, not just a cluster name -- responders need to know who owns it.

For the fields operators legitimately own, declare that in the Application manifest rather than muting alerts ad hoc:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas        # owned by the autoscaler
        - /spec/template/spec/containers/0/resources  # mutated by VPA

Treat every entry in ignoreDifferences as a debt item with an owner. Each one is a place where Git is no longer the source of truth.

The deploy pipeline is the real fix

Most pipeline drift disappears when you stop patching live state. Instead of kubectl set image, have the pipeline commit to an environment overlay and let sync do the work:

# .github/workflows/deploy.yaml (fragment)
- name: Update image tag
  run: |
    cd envs/${{ inputs.env }}
    kustomize edit set image \
      myapp=${{ inputs.registry }}/${{ inputs.image }}:${{ inputs.tag }}
    git commit -am "deploy ${{ inputs.tag }} to ${{ inputs.env }}"
    git push

Yes, this is slower than a direct patch. It buys you something more valuable: the rollback is git revert, and your audit trail is the Git log. Every incident review starts from a state everyone can read.

If sync latency is unacceptable for genuine emergencies, build one audited break-glass path that pauses the Application first (argocd app set "$APP" --sync-policy none), does the patch, and files a ticket to reconcile Git afterward. One sanctioned escape hatch beats a dozen undocumented ones.

What I would do differently

I would set up the drift job on day one, before the first webhook or operator lands. Drift is cheap to detect and expensive to untangle. I would also cap ignoreDifferences entries per app and review them quarterly. And I would document, per namespace, who is allowed to touch live state -- because GitOps without that contract is just a sync engine with extra steps.

Key takeaways

• Most drift at scale comes from controllers and deploy pipelines, not engineers editing the cluster by hand.
• Enabling self-heal everywhere silently breaks any rollback tool that patches live state.
• Schedule argocd app diff and alert on it; every ignoreDifferences entry needs an owner and a review date.
• Move deploy-time patches into Git overlays so rollback becomes git revert.
• Provide one audited break-glass path that pauses sync, rather than tolerating ad-hoc patches.

Source

Why LinkedIn Outgrew Argo CD and What It Means for Your GitOps -- this follow-up adds the drift-detection and deploy-pipeline practices the original did not cover.

Support this work

These write-ups are researched and published with no paywall, sponsor, or tracking. If one saved you an afternoon, a small tip keeps them coming.

USDT, USDC or USDD · TRC-20 (Tron)

TFTNsfyomKrnUutRjBTGVULp19ByW29KbY