devtocash

Posted on Jul 28 • Originally published at devtocash.com

ArgoCD GitOps: Best Practices for Production Deployments

#kubernetes #cicd #argocd #gitops

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

Introduction

GitOps isn't just a buzzword. It's a deployment model where Git is the single source of truth, and an operator continuously reconciles your cluster state with what's in the repository. ArgoCD is the most popular GitOps tool in the CNCF ecosystem—and for good reason.

But deploying ArgoCD in production requires more than kubectl apply -f install.yaml. You need a strategy for organizing applications, managing secrets, handling multi-cluster deployments, and implementing progressive delivery.

This guide covers production-proven ArgoCD patterns with real configuration examples.

Installation & Initial Setup

Install ArgoCD in your cluster:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Access the UI:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Get the initial admin password:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

Production hardening: Configure an Ingress with TLS instead of port-forwarding. Use an external identity provider (OIDC) instead of local admin accounts:

# argocd-cm ConfigMap
data:
  url: https://argocd.yourcompany.com
  oidc.config: |
    name: Okta
    issuer: https://your-org.okta.com
    clientID: argocd
    clientSecret: $oidc.okta.clientSecret
    requestedScopes: ["openid", "profile", "email", "groups"]

Disable the admin account once OIDC is working:

kubectl patch configmap argocd-cm -n argocd \
  -p '{"data":{"admin.enabled":"false"}}'

The App of Apps Pattern

Don't create each ArgoCD Application manually. Use the App of Apps pattern—a parent Application that manages child Applications:

# root-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/infra-repo.git
    targetRevision: main
    path: apps/
    directory:
      recurse: true
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Your repo structure:

infra-repo/
  apps/
    monitoring/
      prometheus.yaml
      grafana.yaml
    backend/
      api-server.yaml
      worker.yaml
    frontend/
      web-app.yaml

Each child Application references its own Helm chart or Kustomize directory. Adding a new service is a PR to apps/, not a manual ArgoCD operation.

Sync Policies

Choose your sync strategy carefully:

Policy	Behavior	Best For
Manual	Human clicks Sync	Production critical, needs approval
Automated + prune	Auto-sync + delete removed resources	Staging, dev environments
Automated + selfHeal	Auto-sync + revert manual changes	Enforcing Git as source of truth

Production recommendation:

syncPolicy:
  automated:
    prune: false        # Never auto-delete in production
    selfHeal: true      # Revert manual kubectl changes
  syncOptions:
  - CreateNamespace=true
  - PruneLast=true      # Delete old resources after new ones are healthy

Health Checks

Custom health checks catch issues ArgoCD's default probes miss:

# In your Application spec
spec:
  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas  # Ignore HPA changes
  - group: autoscaling
    kind: HorizontalPodAutoscaler
    jsonPointers:
    - /spec/metrics/0/resource/target/averageUtilization

Secrets Management

Never store plaintext secrets in Git. ArgoCD integrates with external secret managers:

Sealed Secrets

Encrypt secrets for Git storage:

kubeseal --controller-name=sealed-secrets \
  --controller-namespace=kube-system \
  < secret.yaml > sealed-secret.yaml

The sealed secret is safe to commit. Only the cluster's Sealed Secrets controller can decrypt it.

External Secrets Operator

Sync secrets from AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-credentials
  data:
  - secretKey: DATABASE_URL
    remoteRef:
      key: prod/database-url

Vault Plugin (argocd-vault-plugin)

Inject secrets at render time using placeholders:

# In your manifest
apiVersion: v1
kind: Secret
metadata:
  name: api-secret
  annotations:
    avp.kubernetes.io/path: "secret/data/myapp"
type: Opaque
stringData:
  API_KEY: <api-key>
  DB_PASS: <db-password>

Progressive Delivery with Argo Rollouts

ArgoCD + Argo Rollouts enable blue-green and canary deployments:

kubectl apply -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Replace standard Deployments with Rollouts:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-server
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 5m}
      - setWeight: 40
      - pause: {duration: 5m}
      - setWeight: 100
  selector:
    matchLabels:
      app: api-server
  template:
    # ... pod spec

This sends 20% of traffic to the new version, pauses for 5 minutes of monitoring, then ramps to 40%, then 100%. If ArgoCD detects the Rollout, it tracks the progressive deployment status instead of a simple healthy/unhealthy.

Multi-Cluster Deployments

ArgoCD can manage applications across multiple clusters from a single control plane:

# Register an external cluster
argocd cluster add my-production-cluster

Define which cluster each Application targets:

spec:
  destination:
    server: https://prod-cluster-api.example.com
    namespace: production

For air-gapped environments, deploy a separate ArgoCD instance per cluster and use Git as the synchronization mechanism between them.

Disaster Recovery

Your Git repository is your disaster recovery plan. To restore a cluster:

# 1. Provision fresh cluster
# 2. Install ArgoCD
# 3. Apply the root App of Apps

kubectl apply -f root-app.yaml

ArgoCD will reconcile everything—namespaces, deployments, configmaps, secrets (via External Secrets)—back to the state defined in Git. Recovery time: minutes, not hours.

Critical backup: Regularly back up ArgoCD's own state:

argocd admin export -n argocd > argocd-backup.yaml

This captures Application definitions, projects, and cluster credentials. Without it, you'd need to manually recreate all Application resources.

Multi-Cluster ArgoCD Patterns

Managing one cluster is straightforward. Managing ten requires a different architecture.

Hub-and-Spoke Pattern

One ArgoCD instance (the hub) manages multiple clusters:

apiVersion: v1
kind: Secret
metadata:
  name: prod-cluster
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
stringData:
  name: prod-us-east
  server: https://prod-us-east.example.com:6443
  config: |
    {
      "bearerToken": "...",
      "tlsClientConfig": {"insecure": false}
    }

argocd cluster add prod-us-east --name prod-us-east
argocd cluster add prod-eu-west --name prod-eu-west
argocd cluster add staging-us-east --name staging-us-east

Now deploy an application to multiple clusters:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: api-server
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            environment: production
  template:
    spec:
      source:
        repoURL: https://github.com/company/deploy.git
        path: charts/api-server
        targetRevision: main
      destination:
        server: '{{ server }}'
        namespace: api
      syncPolicy:
        automated:
          prune: true

This single ApplicationSet deploys to every production cluster. Add a new cluster and it deploys automatically.

Per-Cluster ArgoCD (Isolated)

For compliance-heavy environments (finance, healthcare), run ArgoCD in each cluster:

Cluster A (PCI):  ArgoCD instance A  -->  Cluster A only
Cluster B (Prod): ArgoCD instance B  -->  Cluster B only
Cluster C (Dev):  ArgoCD instance C  -->  Cluster C only

Each ArgoCD instance has its own RBAC, audit logs, and sync policies. Cross-cluster changes require separate PRs.

Sync Strategies for Zero-Downtime Deployments

Automated Sync with Prune

The most common strategy: ArgoCD syncs automatically and removes resources not in Git:

argocd app set api-server --sync-policy automated --auto-prune --self-heal

Risk: A malformed git push can delete production resources. Mitigate with:

argocd app set api-server --sync-policy automated \
    --auto-prune --self-heal --sync-option PrunePropagationPolicy=foreground

Blue-Green and Canary Deployments

ArgoCD Rollouts extend ArgoCD with progressive delivery:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-server
spec:
  strategy:
    blueGreen:
      activeService: api-server-active
      previewService: api-server-preview
      autoPromotionEnabled: false
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api-server
          image: myapp:{{ .Values.image.tag }}

On update, ArgoCD Rollouts creates a preview environment, runs smoke tests, and waits for manual approval before switching traffic.

Sync Waves for Dependency Ordering

Control the order resources are applied:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "5"

Wave	Resources	Reason
-5	CRDs	Must exist before everything
0	Namespaces, NetworkPolicies	Foundation
5	Databases, ConfigMaps	Services depend on these
10	Deployments, Services	Main workloads
15	Ingress, ServiceMonitors	External access and monitoring

ArgoCD applies wave -5 first, waits until all resources are healthy, then moves to wave 0, and so on.

RBAC and Multi-Team Access

For organizations with multiple teams sharing ArgoCD:

# argocd-rbac-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.csv: |
    # Platform team: full admin access
    g, platform-team, role:admin

    # Backend team: manage their own apps only
    p, role:backend-team, applications, get, backend/*, allow
    p, role:backend-team, applications, sync, backend/*, allow
    p, role:backend-team, applications, create, backend/*, allow
    g, backend-team, role:backend-team

    # Read-only for QA
    p, role:qa, applications, get, *, allow
    p, role:qa, clusters, get, *, allow
    g, qa-team, role:qa

SSO integration with OIDC:

# argocd-cm.yaml
data:
  oidc.config: |
    name: Okta
    issuer: https://company.okta.com
    clientID: $ARGOCD_OIDC_CLIENT_ID
    clientSecret: $ARGOCD_OIDC_CLIENT_SECRET
    requestedScopes: ['openid', 'profile', 'email', 'groups']
    requestedIDTokenClaims: {'groups': {'essential': true}}

Now developers authenticate with their SSO and see only the applications they own.

ArgoCD Config Management Plugins (CMP)

For non-Helm deployments, use CMPs to support any templating tool:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  configManagementPlugins: |
    - name: kustomize-with-helm
      generate:
        command: ["sh", "-c", "kustomize build --enable-helm ."]

Now ArgoCD can render Kustomize overlays that use Helm charts inside, combining the best of both tools.

Conclusion

ArgoCD transforms Kubernetes deployments from imperative kubectl apply to declarative Git-based reconciliation. The key production patterns: App of Apps for scale, external secrets for security, selfHeal for enforcement, and Argo Rollouts for progressive delivery.

Start simple: one Application, manual sync. Then graduate to App of Apps with automated sync in staging. Add secrets management before production. Finally, layer progressive delivery when zero-downtime deployments matter.

The goal isn't just GitOps—it's making deployments boring, predictable, and reversible.

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.

DEV Community