This guide was written by Michael Zion, CEO at MeteorOps, and first published on the MeteorOps blog.
Bottom line
GitOps is a way of running deployments where the desired state of your systems lives in Git, and an agent inside your environment continuously pulls that state and makes reality match it. The real benefits are concrete: every production change is a reviewed commit, rollback is a git revert, manual drift gets corrected automatically, your CI system no longer needs admin credentials to your clusters, and a lost cluster can be rebuilt from a repository.
The real costs are just as concrete. You have to solve secrets, environment promotion and repository layout yourself; tools like the Horizontal Pod Autoscaler (HPA) will fight your sync controller until you configure around them; and debugging a failed sync adds a layer between a developer and their running pod. GitOps pays off for teams running several services on Kubernetes across more than one environment. It is usually not worth it for a single app on a managed platform, or for a team without anyone who will own the setup.
This guide covers what GitOps actually is (using the OpenGitOps principles, not vendor definitions), each benefit and what it takes to get it, the challenges teams hit in practice, a decision table for when to use it and when not to, Argo CD vs Flux, a worked example, an adoption path, and how to measure whether it worked.
What GitOps is: the four OpenGitOps principles
"GitOps" has been stretched by marketing to mean almost anything involving Git. The useful definition comes from OpenGitOps, a CNCF working group that published four principles (version 1.0.0):
Declarative. "A system managed by GitOps must have its desired state expressed declaratively." You describe what should exist (Kubernetes manifests, Helm values, Kustomize overlays), not the commands to create it.
Versioned and Immutable. "Desired state is stored in a way that enforces immutability, versioning and retains a complete version history." In practice this is Git, though OCI registries also qualify.
Pulled Automatically. "Software agents automatically pull the desired state declarations from the source." The agent runs next to the target system and fetches changes; nothing outside pushes into it.
Continuously Reconciled. "Software agents continuously observe actual system state and attempt to apply the desired state." This is the part most "GitOps" pipelines skip.
The last two principles are what separate GitOps from "we keep our YAML in Git and CI runs kubectl apply". A pipeline that pushes manifests on merge gives you versioning, but it applies changes once and then stops looking. If someone edits a Deployment by hand an hour later, nothing notices. A GitOps agent (Argo CD or Flux, in almost every real setup) notices and either reports the drift or reverts it.

Pushing manifests on merge applies them once; only continuous reconciliation catches drift afterwards.
GitOps vs DevOps vs CI/CD
DevOps is a culture and set of practices for shared ownership of delivery and operations. CI/CD (continuous integration and continuous delivery) is the automation that builds, tests and ships code. GitOps is one specific way to implement the deployment half of CD: CI still builds and tests your code and pushes an image; GitOps takes over from "the new image exists" to "the cluster runs it". It does not replace your CI system and it is not an alternative to DevOps.
The real benefits of GitOps (and what each one requires)
Most benefit lists are written as if you get everything for free once you install Argo CD. You do not. Each benefit below comes with the condition that makes it real.
1. An audit trail you get for free
Every change to production is a commit with an author, a timestamp, a pull request, reviewers and a diff. When someone asks "what changed before the incident at 14:05?", the answer is git log on the environment path, not a search through CI logs and Slack. For teams going through SOC 2 or ISO 27001, this gives auditors change management evidence (who approved what, when) without a separate ticketing ritual.
What it requires: branch protection with required reviews on the environment repository or path, and no other way to change production. If engineers still have write access with kubectl, your audit trail has a hole exactly where incidents happen.
2. Rollback is git revert
Because the cluster follows Git, reverting the commit that bumped an image tag or changed a config value rolls the environment back through the same reviewed, logged path as any deployment. There is no special rollback runbook to remember during an incident.
What it requires: deployments must be expressed as a single, self-contained change (one image tag bump, one values change). If a release is spread across several commits in different repositories, a revert is no longer one action. Also note that Argo CD's own UI rollback is disabled for Applications with automated sync; the Argo CD docs state that "Rollback cannot be performed against an application with automated sync enabled." Git revert is the rollback. Database migrations are not rolled back by reverting a manifest, so design migrations to be backward compatible.
3. Drift detection and correction
Drift is the gap between what you think is running and what is running: a hotfixed ConfigMap, a replica count someone scaled by hand, a resource deleted by mistake. A reconciling agent detects this continuously. With Argo CD's selfHeal or Flux's default behaviour of re-applying on every interval, the agent reverts it. This is the single biggest operational difference between GitOps and push-based CD.
What it requires: deciding what the agent should own and what it should ignore (see the HPA challenge below). Drift correction that fights a legitimate controller is worse than no drift correction.
4. A better security model: pull-based delivery
In push-based CD, your CI system (GitHub Actions, GitLab CI, Jenkins) holds credentials that can modify production clusters. CI runners execute arbitrary code from pull requests and third-party actions, so they are a high-value target. In pull-based GitOps, the agent runs inside the cluster with in-cluster permissions, and CI only needs permission to push an image and, in some setups, to open a commit or pull request against the config repository. The cluster API server does not need to be reachable from your CI provider at all.

With pull-based delivery, the credentials that can change production never leave the cluster.
What it requires: actually removing the old CI credentials and human admin access. Teams often add GitOps alongside their existing kubectl pipeline "temporarily" and end up with two attack surfaces instead of one. Also remember that write access to the config repository now effectively equals deploy access, so protect it like production.
5. Self-service for developers, with guardrails
Once the platform team has defined how apps are laid out, a developer ships a new service or changes a resource limit by opening a pull request against a known path. Review rules, policy checks in CI (schema validation, Open Policy Agent or Kyverno policies) and CODEOWNERS files decide what needs platform approval. This is how a small platform team supports many product engineers without becoming a ticket queue.
What it requires: a repository structure and templates that developers can understand. If adding a service means touching six files across three directories, self-service will not happen.
6. Disaster recovery and cluster rebuilds
If everything that runs in a cluster is declared in Git, rebuilding a cluster is: create the cluster, install the GitOps agent, point it at the repository, wait. The same mechanism lets you spin up a new region or a full preview environment, which is the foundation of the one-click environment idea.
What it requires: really everything in Git, including cluster add-ons (ingress controller, cert-manager, external-dns, monitoring). Persistent data (databases, volumes) is not restored by GitOps; you still need backups. Test the rebuild before you need it, because the first attempt usually exposes a resource someone created by hand two years ago.
7. Faster, more frequent, less scary deployments
When deploying is a small pull request and rollback is a revert, teams deploy smaller changes more often. That is the mechanism by which GitOps moves DORA metrics, discussed at the end of this guide. GitOps does not make your tests better or your services more reliable; it removes friction and fear from the deployment step.
GitOps challenges: what teams actually struggle with
These are the problems that show up in the first three to six months of running GitOps in production. None of them is a reason to avoid GitOps, but each needs a deliberate decision.
Secrets management
You cannot commit plaintext secrets to Git, yet GitOps says the desired state lives in Git. There are three common answers:
Reference, do not store: External Secrets Operator or the Secrets Store CSI Driver. Git holds a pointer ("this secret comes from AWS Secrets Manager key X"); an in-cluster controller fetches the value. This is usually the best default.
Encrypt in Git: Sealed Secrets, or SOPS with age or a cloud KMS key. Flux decrypts SOPS natively through the Kustomization
spec.decryptionfield.Inject at render time with a plugin such as argocd-vault-plugin. The Argo CD secret management docs "strongly caution against this style", partly because "Argo CD stores generated manifests in plaintext in its Redis cache", and recommend managing secrets on the destination cluster instead.
Environment promotion
GitOps tells you how to make an environment match Git. It does not tell you how a change moves from dev to staging to production. Common patterns are a directory per environment with Kustomize overlays (promotion is a pull request that copies a tag from one overlay to the next), a values file per environment for Helm, or automation that opens the promotion pull request after staging checks pass. Avoid a long-lived Git branch per environment: branches drift apart, merges carry unrelated changes, and nobody can easily answer "what is different between staging and prod?". Directories make that question a diff.
Repository structure
Monorepo or one config repo per team? App code and manifests together or separate? There is no universal answer, but two rules hold. First, keep deployable configuration separate from application source when CI writes back to it, otherwise image-bump commits trigger new CI builds and pollute history. Second, decide early how cluster add-ons, shared platform config and app config are separated, because moving hundreds of resources between paths later means carefully orchestrated pruning.
Drift vs controllers that legitimately change state (HPA and friends)
The HPA changes spec.replicas on your Deployment. If Git also sets replicas: 3 and self-heal is on, the GitOps agent and the HPA fight, and you get replica counts snapping back on every sync. The fix is to stop declaring replicas for autoscaled workloads, and tell the agent to ignore the field. In Argo CD that is ignoreDifferences on /spec/replicas, plus the RespectIgnoreDifferences=true sync option so the ignored field is also left alone during sync, not only in the diff (both documented in the Argo CD sync options). The same applies to fields mutated by admission webhooks, operators and service meshes. We cover the autoscaling side in how to configure Kubernetes HPA without causing scaling flaps.

Stop declaring replicas for autoscaled workloads and tell the agent to ignore the field.
Debugging sync failures
Before GitOps, a failed deploy showed up red in the CI job the developer was watching. After GitOps, CI is green (it pushed an image and a commit), and the failure happens asynchronously in the cluster: a schema error, a missing CRD, a failing health check, an immutable field change. Developers need to know where to look. Wire the agent's notifications (Argo CD Notifications or the Flux notification-controller) into the team's channel and into commit statuses, and give developers read access to the Argo CD UI or Flux status, otherwise every failed sync becomes a platform team ticket.
Separating CI from CD
The clean split is: CI builds, tests, scans and pushes an image, then records the new tag in the config repository (directly or through a pull request). The GitOps agent deploys. Teams get into trouble when CI keeps doing deployment-shaped work (running helm upgrade "just for this one service", running migrations with cluster credentials), because that reintroduces push access and splits the source of truth.

CI ends at a commit to the config repo; everything after that belongs to the agent.
Image update automation
Someone has to write the new image tag into Git. Options, from simplest to most automated:
CI commits the tag (for example with
kustomize edit set imageoryq) to the dev overlay after a successful build. Simple, explicit, and works with any agent.Flux image automation:
ImageRepository,ImagePolicyandImageUpdateAutomation(APIimage.toolkit.fluxcd.io/v1) scan a registry and commit tag updates to lines marked with a comment like# {"$imagepolicy": "flux-system:podinfo"}.Argo CD Image Updater, a separate project. Its docs describe it as under active development and invite users to test it on non-critical environments, so treat it accordingly for production.
Multi-cluster and fleet management
One cluster is easy. Ten clusters across regions and environments raise new questions: one central Argo CD managing all clusters (a single pane of glass, but a single, highly privileged control plane) or an agent per cluster (more isolation, more to operate)? How do you template the same add-on across clusters with small differences? Argo CD answers the templating question with ApplicationSets; Flux with Kustomize overlays per cluster path and, increasingly, the Flux Operator's ResourceSets. Pick one model deliberately before cluster number three.
Ordering, CRDs and bootstrap
A CRD must exist before resources that use it; a namespace before what lives in it; cert-manager before Certificates. Flux expresses this with dependsOn between Kustomizations; Argo CD with sync waves and sync phases. Expect to spend real time on bootstrap ordering the first time you rebuild a cluster from scratch.
When to use GitOps and when not to
GitOps is most valuable when there is a lot of state to keep consistent and more than a handful of people changing it. Use this table as a starting point.
| Situation | Recommendation | Why |
|---|---|---|
| 1 to 3 engineers, one app on a PaaS (Vercel, Heroku, Cloud Run, App Runner) | Skip GitOps | The platform already gives you declarative deploys and rollback. A Kubernetes GitOps stack would be overhead without a cluster to reconcile. |
| Small team, one Kubernetes cluster, a few services | Adopt, but keep it minimal | One agent, one config repo, plain manifests or Kustomize. You get audit, rollback and DR cheaply. Avoid ApplicationSets and fancy promotion until you need them. |
| 5 to 50 engineers, multiple services, dev/staging/prod on Kubernetes | Strong fit | This is where drift, "who changed what" and slow manual deploys start to hurt. Self-service and promotion via pull requests pay back quickly. |
| Multiple clusters, regions or tenants; platform team serving product teams | Strong fit, invest in fleet tooling | Templated add-ons, cluster bootstrap from Git and centralized visibility become essential. |
| Regulated environment (SOC 2, ISO 27001, PCI) | Strong fit | Reviewed commits are change-management evidence, and removing CI cluster credentials shrinks the attack surface. |
| Mostly serverless or managed services, little or no Kubernetes | Use Infrastructure as Code with PR-based plans instead | Terraform or OpenTofu with plan-on-PR and apply-on-merge gives most of the audit benefit. Continuous reconciliation is possible (for example via Crossplane) but is rarely the first problem to solve. |
| No one available to own the setup | Wait | A half-configured GitOps setup (secrets unsolved, drift fights, nobody reading sync errors) is worse than a simple, well-understood pipeline. |
A useful smell test: if your team can't answer "what exactly is running in production right now, and who approved it?" in under a minute, GitOps will help. If you can, and deploys are not painful, it may not be your highest-leverage project. Our guide to calculating your company's required DevOps capacity helps decide whether you have the people to own it.
Can GitOps manage cloud infrastructure, not just Kubernetes apps?
Yes, if the infrastructure is expressed as Kubernetes resources that a controller reconciles: Crossplane, or cloud-specific controllers such as AWS Controllers for Kubernetes and Google Config Connector. See deploying AWS resources using Crossplane on Kubernetes. For most startups, Terraform with pull-request plans is the pragmatic choice for cloud resources, with GitOps handling what runs inside the clusters.
Argo CD vs Flux: which GitOps tool to choose
Both are CNCF graduated projects (Flux graduated in November 2022, Argo in December 2022), both are production grade, and both implement the four principles. The difference is mostly in operating model and user experience.
| Argo CD | Flux | |
|---|---|---|
| Core model | An Application resource points at a source (Git, Helm repo or OCI) and a destination cluster and namespace. |
A set of controllers: sources (GitRepository, OCIRepository, HelmRepository) plus appliers (Kustomization, HelmRelease). |
| UI | Rich built-in web UI showing resource trees, diffs, sync status and logs. A big reason teams pick it. | No UI in core Flux. The Flux Operator ships a Flux Web UI (status dashboard with OIDC and RBAC); many teams use the CLI and notifications. |
| Multi-cluster | Commonly one central instance managing many clusters; ApplicationSets generate Applications per cluster, per directory or per pull request. | Commonly one Flux per cluster, each bootstrapped from its own path in Git. Can also target remote clusters via kubeconfig. |
| Multi-tenancy | AppProjects restrict which repos, clusters, namespaces and resource kinds a team can deploy, plus SSO and Argo CD RBAC. | Relies on Kubernetes RBAC: each Kustomization can impersonate a service account scoped to a tenant's namespaces. |
| Helm | Renders charts with helm template and applies the output; Helm release history is not used. |
HelmRelease performs real Helm installs and upgrades, with Helm hooks, tests and rollback remediation. |
| Secrets | Bring your own (External Secrets, Sealed Secrets, CSI driver). | Native SOPS decryption in Kustomizations, or bring your own. |
| Image automation | Argo CD Image Updater (separate project, still maturing). | Built-in image automation controllers. |
| Change detection | Polls repos every 3 minutes by default (120s plus up to 60s jitter), plus webhooks. | Per-object interval, plus webhook receivers via notification-controller. |
| Progressive delivery | Argo Rollouts (canary, blue-green). | Flagger. |
Our default advice: if developers will interact with deployments and you want them to see what is happening without learning kubectl, pick Argo CD. If you want a lightweight, Kubernetes-native agent per cluster, heavy Helm usage with real Helm lifecycle, native SOPS and built-in image automation, pick Flux. Both are fine choices; the bigger risk is spending weeks choosing. For canary and blue-green on top of Argo CD, see zero-downtime Kubernetes releases with Helm and Argo Rollouts.
A worked GitOps example: repo layout and manifests
Here is a minimal, realistic setup for one service with dev and production environments, using Kustomize overlays. The same layout works with either tool.
platform-config/
apps/
checkout-api/
base/
deployment.yaml
service.yaml
hpa.yaml
kustomization.yaml
overlays/
dev/
kustomization.yaml # image tag for dev, smaller resources
production/
kustomization.yaml # image tag for prod, prod resources
clusters/
dev/
checkout-api.yaml # Argo CD Application or Flux Kustomization
production/
checkout-api.yaml
The production overlay pins the image tag. Promotion to production is a pull request that changes newTag here, and rollback is reverting that pull request:
# apps/checkout-api/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
images:
- name: ghcr.io/acme/checkout-api
newTag: "1.42.0"
Option A: Argo CD Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: checkout-api-production
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/acme/platform-config.git
targetRevision: main
path: apps/checkout-api/overlays/production
destination:
server: https://kubernetes.default.svc
namespace: checkout
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicas # owned by the HPA, not Git
syncPolicy:
automated:
prune: true # delete resources removed from Git
selfHeal: true # revert manual changes in the cluster
syncOptions:
- CreateNamespace=true
- RespectIgnoreDifferences=true
Notes: prune and selfHeal are both off by default in Argo CD, so drift correction only happens if you turn them on. In real setups, replace project: default with a dedicated AppProject that limits which repos and namespaces this team can deploy to.
Option B: Flux GitRepository and Kustomization
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: platform-config
namespace: flux-system
spec:
interval: 1m
url: https://github.com/acme/platform-config.git
ref:
branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: checkout-api
namespace: flux-system
spec:
interval: 10m
path: ./apps/checkout-api/overlays/production
prune: true
sourceRef:
kind: GitRepository
name: platform-config
targetNamespace: checkout
wait: true
timeout: 3m
Notes: Flux re-applies the desired state on every interval, which corrects drift. For a private repository, add a secretRef on the GitRepository. For the HPA case, the simplest Flux-side fix is to leave replicas out of the Deployment manifest entirely, so the HPA owns it. Add dependsOn if this app needs, for example, cert-manager to be ready first.
Whichever tool you use, add health checks that mean something. A sync that "succeeds" with pods in a restart loop is a false green; see how to configure Kubernetes liveness probes without causing restart loops.
A practical GitOps adoption path: from first app to fleet
Pick one non-critical service and one environment. Install Argo CD or Flux, point it at a config repo, and deploy that service with automated sync. Goal: the team sees a merge turn into a deployment in a few minutes.
Solve secrets before the second service. Choose External Secrets Operator, Sealed Secrets or SOPS and document the pattern. This is the decision teams most regret postponing.
Move cluster add-ons into Git. Ingress controller, cert-manager, external-dns, metrics and logging agents. Now the cluster itself is reproducible.
Add environments and promotion. Overlays per environment, CI writes the dev tag, promotion to staging and production by pull request. Add notifications and commit statuses so developers see sync results.
Turn on self-heal and remove manual access. Once teams trust the flow, enable self-heal, remove CI cluster credentials and reduce human write access to break-glass roles with audit logging.
Prove the rebuild. Create a fresh cluster from Git and measure how long it takes to serve traffic. Fix whatever was missing.
Scale to the fleet. ApplicationSets or per-cluster Flux paths, policy checks in CI, preview environments per pull request if useful, progressive delivery for critical services.

Each step builds the trust and coverage the next one depends on.
Common GitOps mistakes and how to fix them
Keeping kubectl write access "for emergencies" for everyone. Fix: a break-glass role that is audited and rarely used; emergencies go through a fast-tracked pull request.
A branch per environment. Fix: directories or overlays per environment on one branch.
Using the
latesttag. Git does not change when the image does, so there is no audit trail and no revert. Fix: immutable tags or digests.Declaring replicas for autoscaled workloads. Fix: remove them and ignore the field in the agent.
Turning on prune on day one without a review. A path mistake can delete a namespace's worth of resources. Fix: enable prune after the layout is stable, and protect critical resources (Argo CD supports a
Prune=falsesync option per resource).Nobody watching sync status. Fix: notifications to the owning team's channel and failing commit statuses.
Secrets via render-time plugins. Fix: move to destination-cluster secret management as Argo CD recommends.
How to measure GitOps value with DORA metrics
GitOps should move delivery metrics, and you should measure before and after. DORA currently defines five software delivery metrics:
Change lead time: time from commit to running in production. Watch whether deployment queues and manual steps disappear.
Deployment frequency: how often you deploy. Smaller, routine pull request deploys should increase it.
Failed deployment recovery time: how long it takes to restore service after a failed deployment. Revert-based rollback should cut this sharply.
Change fail rate: share of deployments that need urgent fixing (rollback or hotfix).
Deployment rework rate: share of deployments that were unplanned and caused by a production incident.
The commit history in your config repository plus the agent's sync events give you most of the raw data: commit time, sync time, reverts and their timestamps. Add two GitOps-specific health signals: how often drift is detected (it should trend toward zero as manual access disappears) and how many syncs fail (it should be low and owned by the team that made the change, not the platform team).
GitOps implementation checklist
Desired state for apps and cluster add-ons is in Git, in a structure developers understand.
Branch protection and required reviews on environment paths; CODEOWNERS for platform-owned paths.
One secrets approach, documented, with no plaintext secrets in Git.
Immutable image tags; CI writes tags, the agent deploys.
Promotion between environments is a pull request.
Fields owned by other controllers (HPA replicas, webhook-injected fields) are ignored by the agent.
Sync notifications reach the owning team; developers can see sync status and logs.
Self-heal and prune enabled deliberately; critical resources protected.
CI has no cluster admin credentials; human write access is break-glass only.
A cluster rebuild from Git has been tested and timed.
DORA baseline captured before rollout.
When outside GitOps help makes sense
Most teams can install Argo CD or Flux in an afternoon. What takes months is the design around it: repository structure, secrets, promotion, multi-cluster, and migrating existing services without downtime. Bringing in experienced engineers makes sense when you are migrating a large number of existing services, moving to multiple clusters or regions, preparing for a compliance audit, or when the people who would own GitOps are already fully booked on product work. The goal of any good engagement should be a setup your own team can run and extend afterwards, not a dependency.
GitOps FAQ
What are the main benefits of GitOps?
A complete audit trail of production changes, rollback by reverting a commit, automatic detection and correction of drift, a safer pull-based security model where CI holds no cluster credentials, developer self-service through pull requests, and the ability to rebuild clusters from Git. Each benefit depends on removing other ways to change production.
What is the biggest disadvantage of GitOps?
The added complexity around the edges: secrets, environment promotion, repository structure and debugging asynchronous sync failures all need deliberate design. For a small team with one app on a managed platform, that overhead can outweigh the benefits.
What are the four principles of GitOps?
According to OpenGitOps: the desired state is declarative; it is versioned and immutable; it is pulled automatically by software agents; and it is continuously reconciled against the actual state.
Is GitOps only for Kubernetes?
The principles are not Kubernetes specific, but the mature tooling (Argo CD, Flux) is built for Kubernetes. Cloud resources can be managed the GitOps way through Kubernetes-based controllers such as Crossplane, while many teams use Terraform with pull-request plans for infrastructure instead.
Is GitOps the same as CI/CD?
No. CI builds and tests code and produces artifacts. GitOps is a way to do the deployment part of CD, where an agent pulls the desired state from Git and reconciles the environment. You still need CI.
Should I use Argo CD or Flux?
Choose Argo CD if you want a strong UI and a central control plane for many clusters and teams. Choose Flux if you prefer a lightweight per-cluster agent, native Helm lifecycle, built-in SOPS decryption and image automation. Both are CNCF graduated and production ready.
How do you roll back with GitOps?
Revert the commit (or pull request) that introduced the change, and the agent syncs the environment back. In Argo CD, UI rollback is disabled for Applications with automated sync, so Git revert is the intended path. Database schema changes need to be backward compatible because reverting manifests does not undo migrations.
How do you handle secrets in GitOps?
Either store a reference in Git and let External Secrets Operator or the Secrets Store CSI Driver fetch the value from a secret manager, or store encrypted secrets with Sealed Secrets or SOPS. Avoid render-time injection plugins, which Argo CD's own docs caution against.
If you are weighing GitOps for your stack and want a second opinion on structure, secrets and tool choice, you can book a free consultation with a senior DevOps engineer.
Top comments (0)