DEV Community

Cover image for Zero-Downtime Argo CD Migrations: The Ultimate Guide to ApplicationSet Refactoring
Yoni Gofman
Yoni Gofman

Posted on

Zero-Downtime Argo CD Migrations: The Ultimate Guide to ApplicationSet Refactoring

#argocd #sre #devops #kubernetes

In a GitOps world, your ApplicationSet is the source of truth for your fleet. But what happens when you need to rename a set, refactor your generators, or migrate workloads to a completely new Kubernetes cluster?

The default behavior of Argo CD is Cascading Deletion: delete the parent (AppSet), and you delete the children (Applications) and the grandchildren (K8s Resources). In production, this is a nightmare.

This guide covers how to bypass the "Delete-everything" trap using the Orphan Strategy for three real-world scenarios.

1. The Foundation: The "Orphan" Safety Net

Before performing any migration, you must ensure that deleting the ApplicationSet doesn't trigger a cleanup of your production environment.

Add this to your ApplicationSet spec:

spec:
  syncPolicy:
    preserveResourcesOnDeletion: true
Enter fullscreen mode Exit fullscreen mode

What this does: It tells the controller to remove the ApplicationSet and the Application CRDs but leave the actual Kubernetes resources (Deployments, Services, etc.) running in the cluster.

2. The Pruning Pitfall: Don't Let Self-Heal Kill Your Migration

This is the most critical part of the migration. If your Root App or ApplicationSet has Prune: true and SelfHeal: true enabled, Argo CD will try to delete anything that doesn't match the Git state the second you make a change.

Senior Strategy:

  1. Disable Pruning Temporarily: Before starting the migration, set prune: false in your SyncPolicy.
  2. Verify via Dry-Run: Use the CLI to see what would be pruned: argocd app sync <app-name> --dry-run --prune
  3. Use PrunePropagationPolicy=orphan: If you are deleting via kubectl, you can specify the propagation policy to ensure resources aren't reaped.

3. Scenario A: In-Place Refactoring (Renaming/Repo Change)

Use this when you want to rename an AppSet or move its definition to a different Git repository without causing downtime.

Step-by-Step:

  1. Patch the Existing Set: Apply the preserveResourcesOnDeletion: true patch.
  2. The "Detachment": Delete the old ApplicationSet.
    • Result: Your Applications will now appear in the Argo CD UI as "Orphans".
  3. Deploy the New Set: Create the new ApplicationSet. Ensure the template generates Applications with the exact same name as the orphaned ones.

Pro-Tip: Use Server-Side Apply (SSA)
To avoid metadata conflicts when the new manager takes over, enable SSA in the template:

spec:
  template:
    spec:
      syncPolicy:
        syncOptions:
          - ServerSideApply=true
Enter fullscreen mode Exit fullscreen mode

4. Scenario B: Target Cluster Migration

Use this when you are moving workloads from an old cluster to a brand new one.

The Strategy: Side-by-Side Provisioning

  1. Dual-Targeting: Update your generator to include both the old and the new cluster.
  2. Avoid Collisions: Ensure your application names include the cluster name (e.g., name: '{{cluster}}-{{app}}') to keep them unique.
  3. The Cut-over: * Verify the new cluster is Healthy.
    • Switch your DNS/Traffic to the new cluster.
    • Remove the old cluster from the generator. The preserveResourcesOnDeletion flag will keep the old apps as orphans until you're ready to manually delete them.

5. Scenario C: The "App of Apps" Hierarchy

When a Root App manages your ApplicationSet, you have a three-layer deletion chain.

Breaking the Chain

  1. Orphan the AppSet: Delete the Root App using the --cascade=false flag via the CLI:

    argocd app delete root-app --cascade=false

  2. Disable Root-Level Pruning: Ensure the new Root App is deployed with prune: false for the first sync to prevent it from cleaning up the existing AppSet if there's a minor naming mismatch.

  3. Re-parenting: Once the new Root App "adopts" the AppSet, you can safely re-enable pruning.

Senior Level Checklist

  • Finalizers: Check for resources-finalizer.argocd.argoproj.io. If it's there and you don't use the orphan flags, resources will be deleted.
  • Ignore Scaling Drift: Use ignoreDifferences for replicas to prevent the migration from resetting your production scale.
  • Sync Strategy: Always start with Manual Sync during a migration. Only switch back to Automated once you've verified the adoption.

Summary Runbook

  1. Disable Pruning on the Root/Parent level.
  2. Patch preserveResources on the AppSet.
  3. Decouple (use cascade=false).
  4. Deploy the new definition & Verify via Dry-run.

Top comments (0)