ArgoCD on OpenShift: managing 30 clusters with ApplicationSet and the app-of-apps pattern

Running one OpenShift cluster with Argo CD is easy. Running thirty — three environments, four regions, two tenancy models — is a different problem. At that scale, the bottleneck is never the sync engine: it is the number of Application objects a human has to author, review and keep consistent. Thirty clusters times fifteen platform components is 450 Applications. Nobody maintains that by hand.

This article describes a working topology: a single root Application (app-of-apps) bootstrapping a set of ApplicationSet resources, cluster secrets carrying environment metadata as labels, and Kustomize overlays resolving the per-environment delta. Everything is declarative, everything is in Git, and adding a cluster is a one-line label change.

Context: why one Application per cluster breaks down

The naive pattern is a directory of Applications:

apps/
  prd-eu-01-logging.yaml
  prd-eu-01-monitoring.yaml
  prd-eu-02-logging.yaml
  ...

This fails for three reasons.

Drift by copy-paste. Each file duplicates the same 25 lines. A fix to syncPolicy or to ignoreDifferences lands in 12 files and misses 18. Reviewers cannot diff 450 near-identical manifests meaningfully.

No lifecycle for clusters. Decommissioning a cluster means finding and deleting every Application referencing it. Forgotten Applications stay Unknown forever and pollute health dashboards and alerting.

No rollout control. You want dev first, then stg, then prd in batches of two. Static Applications with automated.selfHeal sync everything simultaneously the moment a commit lands on main.

ApplicationSet solves all three: it is a controller that generates and reconciles Applications from a generator (clusters, git, matrix, merge), owns their full lifecycle, and supports progressive rollout strategies.

Foundation: OpenShift GitOps and cluster registration

Start from the operator-managed instance. Verify what you are running — generator and strategy support varies significantly across versions:

oc get csv -n openshift-gitops -o custom-columns=NAME:.metadata.name,VERSION:.spec.version
oc -n openshift-gitops exec deploy/openshift-gitops-server -- argocd version --short

Cluster secrets are your inventory

Argo CD stores remote clusters as Secret objects labeled argocd.argoproj.io/secret-type: cluster. The labels on that secret are the metadata the cluster generator will expose to templates. Treat them as your source of truth.

ARGOCD_HOST=$(oc get route openshift-gitops-server -n openshift-gitops -o jsonpath='{.spec.host}')
argocd login "$ARGOCD_HOST" --sso --grpc-web

# kubeconfig context must already exist locally
argocd cluster add prd-eu-01-admin \
  --name prd-eu-01 \
  --label env=prd --label region=eu --label tier=platinum --label wave=1 \
  --service-account argocd-manager \
  --system-namespace openshift-gitops-agent \
  --grpc-web --yes

For reproducibility, prefer declaring the secret in Git with the token injected by External Secrets Operator or Sealed Secrets:

apiVersion: v1
kind: Secret
metadata:
  name: cluster-prd-eu-01
  namespace: openshift-gitops
  labels:
    argocd.argoproj.io/secret-type: cluster
    env: prd
    region: eu
    tier: platinum
type: Opaque
stringData:
  name: prd-eu-01
  server: https://api.prd-eu-01.example.com:6443
  config: |
    {
      "bearerToken": "<injected-by-eso>",
      "tlsClientConfig": {
        "insecure": false,
        "caData": "<base64-cluster-ca>"
      }
    }

Verify the inventory is complete before building anything on top of it:

oc get secrets -n openshift-gitops -l argocd.argoproj.io/secret-type=cluster \
  -o custom-columns=CLUSTER:.metadata.name,ENV:.metadata.labels.env,REGION:.metadata.labels.region
argocd cluster list

Repository layout

Two repositories: one for the GitOps control plane, one (or several) for workloads. The control-plane repo:

gitops-platform/
├── bootstrap/
│   ├── root-application.yaml        # applied once, by hand or by the operator
│   └── appsets/
│       ├── platform-components.yaml
│       └── tenant-namespaces.yaml
├── projects/
│   ├── platform.yaml
│   └── tenants.yaml
└── components/
    ├── cluster-logging/
    │   ├── base/
    │   │   ├── kustomization.yaml
    │   │   ├── namespace.yaml
    │   │   ├── operatorgroup.yaml
    │   │   ├── subscription.yaml
    │   │   └── clusterlogging.yaml
    │   └── overlays/
    │       ├── dev/kustomization.yaml
    │       ├── stg/kustomization.yaml
    │       └── prd/kustomization.yaml
    ├── compliance-operator/
    └── network-policies/

The contract is strict: every component exposes overlays/<env> for every env label used on cluster secrets. A missing overlay must fail loudly, not silently fall back to base.

The root Application (app-of-apps)

One object bootstraps everything. It is the only manifest you ever apply imperatively.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: openshift-gitops
  annotations:
    argocd.argoproj.io/sync-wave: "-10"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/acme/gitops-platform.git
    targetRevision: main
    path: bootstrap
    directory:
      recurse: true
      include: '{appsets/*.yaml,../projects/*.yaml}'
  destination:
    server: https://kubernetes.default.svc
    namespace: openshift-gitops
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ApplyOutOfSyncOnly=true
      - ServerSideApply=true

oc apply -f bootstrap/root-application.yaml
argocd app get root --grpc-web

From here, adding an ApplicationSet is a pull request against bootstrap/appsets/.

ApplicationSet with a matrix generator

The cluster generator alone gives you one Application per cluster. Combined with a git directory generator in a matrix, you get one Application per (cluster, component) pair — the 450 objects, generated from 40 lines.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-components
  namespace: openshift-gitops
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - clusters:
              selector:
                matchExpressions:
                  - key: env
                    operator: In
                    values: [dev, stg, prd]
          - git:
              repoURL: https://github.com/acme/gitops-platform.git
              revision: main
              directories:
                - path: components/*
                - path: components/_experimental
                  exclude: true
  template:
    metadata:
      name: '{{.nameNormalized}}-{{.path.basename}}'
      namespace: openshift-gitops
      labels:
        env: '{{index .metadata.labels "env"}}'
        region: '{{index .metadata.labels "region"}}'
        component: '{{.path.basename}}'
      finalizers:
        - resources-finalizer.argocd.argoproj.io
    spec:
      project: platform
      source:
        repoURL: https://github.com/acme/gitops-platform.git
        targetRevision: main
        path: '{{.path.path}}/overlays/{{index .metadata.labels "env"}}'
      destination:
        server: '{{.server}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
          - RespectIgnoreDifferences=true
        retry:
          limit: 5
          backoff:
            duration: 20s
            factor: 2
            maxDuration: 5m

goTemplateOptions: ["missingkey=error"] is not cosmetic: without it, a cluster secret missing the env label silently renders components/cluster-logging/overlays/ and syncs the wrong tree. With it, the generator errors and the ApplicationSet reports the failure.

Dry-run the generation before merging:

argocd appset generate bootstrap/appsets/platform-components.yaml -o yaml | \
  yq '.[].metadata.name'

Then inspect what the controller actually produced:

argocd appset get platform-components
argocd app list -l component=cluster-logging -o wide
oc get applications.argoproj.io -n openshift-gitops -l env=prd --no-headers | wc -l

Kustomize overlays carry the environment delta

Keep base cluster-agnostic. The overlay encodes retention, replicas, storage class, and nothing else.

# components/cluster-logging/overlays/prd/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: openshift-logging
resources:
  - ../../base
patches:
  - target:
      group: logging.openshift.io
      version: v1
      kind: ClusterLogging
      name: instance
    patch: |-
      - op: replace
        path: /spec/logStore/elasticsearch/nodeCount
        value: 5
      - op: replace
        path: /spec/logStore/retentionPolicy/application/maxAge
        value: 30d
      - op: add
        path: /spec/logStore/elasticsearch/storage/storageClassName
        value: gp3-csi
labels:
  - pairs:
      app.kubernetes.io/managed-by: openshift-gitops
      env: prd
    includeSelectors: false

Validate locally — this is the cheapest CI gate you can add:

for env in dev stg prd; do
  for c in components/*/; do
    kustomize build "${c}overlays/${env}" >/dev/null \
      || echo "BROKEN: ${c}overlays/${env}"
  done
done

Ordering: sync waves inside an Application

Within one Application, argocd.argoproj.io/sync-wave sequences resources. The canonical operator pattern is namespace → OperatorGroup → Subscription → custom resource, with a guard so Argo CD does not dry-run a CR whose CRD does not exist yet.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-logging
  namespace: openshift-logging
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  channel: stable-6.1
  name: cluster-logging
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic
---
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
  annotations:
    argocd.argoproj.io/sync-wave: "10"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  managementState: Managed

Waves are per Application, not global. To order across Applications, put the sync-wave annotation on the generated Application itself (the root app treats them as children) or rely on RollingSync.

Progressive rollout across 30 clusters

Enable progressive syncs on the operator-managed instance, then declare steps matching the labels you set in the ApplicationSet template.

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: openshift-gitops
  namespace: openshift-gitops
spec:
  applicationSet:
    extraCommandArgs:
      - --enable-progressive-syncs
  controller:
    sharding:
      enabled: true
      replicas: 3
    processors:
      operation: 25
      status: 50
    resources:
      requests: {cpu: "1", memory: 2Gi}
      limits: {memory: 6Gi}
  repo:
    replicas: 3

# inside spec of the ApplicationSet
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: env
              operator: In
              values: [dev]
        - matchExpressions:
            - key: env
              operator: In
              values: [stg]
          maxUpdate: 50%
        - matchExpressions:
            - key: env
              operator: In
              values: [prd]
          maxUpdate: 2

Each step waits for the previous batch to reach Healthy. A failing dev cluster stops the promotion before it reaches production. Drive and observe it from the CLI:

argocd app wait -l env=stg --health --timeout 900 --grpc-web
argocd app list -l env=prd --output wide | awk '$3!="Synced" || $4!="Healthy"'
argocd app diff prd-eu-01-cluster-logging --local components/cluster-logging/overlays/prd

Three application-controller shards comfortably handle 30 clusters; the default sharding algorithm distributes clusters by hash. If one cluster is much larger than the others, switch to round-robin via spec.controller.env with ARGOCD_CONTROLLER_SHARDING_ALGORITHM.

Security best practices

Never grant cluster-admin to the Argo CD service account on remote clusters. argocd cluster add --service-account lets you bind a scoped ClusterRole. Build it from the component inventory: operator subscriptions, namespaces, network policies, RBAC — not */*. Audit periodically with oc auth can-i --list --as=system:serviceaccount:openshift-gitops-agent:argocd-manager.

Make AppProject the real boundary. ApplicationSet can generate any Application; the project is what constrains it. Deny cluster-scoped resources you never intend to manage:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: platform
  namespace: openshift-gitops
spec:
  sourceRepos:
    - https://github.com/acme/gitops-platform.git
  destinations:
    - server: '*'
      namespace: 'openshift-*'
    - server: '*'
      namespace: 'platform-*'
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace
    - group: operators.coreos.com
      kind: '*'
    - group: rbac.authorization.k8s.io
      kind: ClusterRole
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota
    - group: security.openshift.io
      kind: SecurityContextConstraints
  signatureKeys:
    - keyID: 4AEE18F83AFDEB23
  orphanedResources:
    warn: true

Require signed commits. signatureKeys on the project plus GPG keys loaded into Argo CD means an unsigned commit on main cannot sync. This is the single most effective control against a compromised CI token pushing to the GitOps repo.

No plaintext secrets in Git, ever. Use External Secrets Operator or Sealed Secrets, and add spec.extraConfig resource exclusions so Argo CD does not cache sensitive objects it has no business tracking. Cluster bearer tokens in openshift-gitops are the crown jewels: restrict get secrets in that namespace to the operator's own service accounts.

Lock down the ApplicationSet controller. It can create and delete Applications cluster-wide. Run it with --policy create-update during onboarding so a bad generator cannot mass-delete Applications, and disable SCM providers (--enable-scm-providers=false) if you only use cluster and git generators — SCM generators can be abused to reach arbitrary endpoints with the controller's token.

Use OpenShift OAuth, disable the local admin. Set spec.sso.provider: dex with the OpenShift connector (or your OIDC IdP), map groups to Argo CD RBAC roles, and set spec.disableAdmin: true once SSO works. Give tenants sync on their project only, never applications, delete, */*.

Pin everything. targetRevision: main is convenient for the control plane but dangerous for workloads: pin tenant Applications to tags or commit SHAs, and use installPlanApproval: Manual for operators on production clusters so an operator upgrade is a deliberate Git change.

Conclusion

The pattern that scales to 30 clusters has four moving parts, and none of them is a per-cluster manifest:

• Cluster secrets with labels are the inventory. env, region, tier drive every downstream decision. Onboarding a cluster is one labeled secret; offboarding is deleting it, and the ApplicationSet garbage-collects the Applications.
• A single root Application bootstraps the ApplicationSets and AppProjects. Exactly one oc apply in the entire lifecycle of the platform.
• Matrix generators (clusters × git directories) turn 40 lines of YAML into hundreds of Applications, with goTemplateOptions: ["missingkey=error"] to fail fast on incomplete metadata.
• Kustomize overlays plus sync waves plus RollingSync handle, respectively, the environment delta, intra-application ordering, and inter-cluster promotion.

Add kustomize build validation and argocd appset generate to CI, scope your remote service accounts, enforce signed commits, and the thirty-cluster fleet becomes less operational work than the three-cluster one you started with.