Solved: Beginner-friendly ArgoCD challenge. Practice GitOps with zero setup

#devops #programming #tutorial #cloud

🚀 Executive Summary

TL;DR: ArgoCD often ignores custom health checks for Kustomize applications, particularly for resources like CronJobs, leading to applications stuck in a ‘Progressing’ state. This occurs because ArgoCD loads its configuration from argocd-cm before processing application-specific Kustomize overlays. The solution involves managing ArgoCD’s configuration, including custom health checks, as separate infrastructure-as-code, either by defining the full argocd-cm in a Kustomize base or injecting them via Helm values.

🎯 Key Takeaways

ArgoCD loads its configuration from argocd-cm *before* processing application Kustomize overlays, causing custom health checks defined within application manifests to be ignored.
Attempting to use Kustomize patchesStrategicMerge on argocd-cm from within an application’s manifests will not work for custom health checks due to the timing of configuration loading.
The correct GitOps approach for custom health checks is to manage ArgoCD’s configuration as part of your infrastructure-as-code, either by providing a complete argocd-cm.yaml in your ArgoCD Kustomize base or by injecting resource.customizations via the official ArgoCD Helm chart’s values.yaml.

Struggling with ArgoCD’s custom health checks for Kustomize apps getting ignored? Here’s the real-world, GitOps-friendly fix for that frustrating ‘Progressing’ status that drives every DevOps engineer nuts.

ArgoCD, Kustomize, and the Missing Health Check: A GitOps War Story

I remember it like it was yesterday. It was 2 AM, during a “low-risk” maintenance window for our new billing service. Everything was green on the pre-flight checks. The deployment pipeline kicked off, and we watched the ArgoCD UI light up. Pods were coming up, services were getting IPs… and then it just stopped. The whole application was stuck in a ‘Progressing’ state, with a little yellow spinning icon that mocked my lack of sleep. The culprit? A single, simple CronJob. ArgoCD had no idea what a “healthy” CronJob looked like, so it just sat there, waiting for a signal that would never come. My junior engineer on the call was starting to panic, and honestly, I was feeling the pressure. This little config issue was holding a critical release hostage.

So, Why Is This Happening? The Chicken and the Egg Problem

You’ve probably run into this. You’re a good GitOps practitioner. You want to add a custom health check for a resource, like a CronJob or some CRD, so you create a Kustomize patch to add the Lua script to the argocd-cm ConfigMap. You commit, you push, you sync… and nothing. The health check is ignored, and your app stays ‘Progressing’ forever.

Here’s the root cause, and it’s a subtle one: ArgoCD loads its own configuration from argocd-cm *before* it processes and applies your application’s Kustomize overlays.

Think about it. The ArgoCD Application Controller needs its config to know *how* to render manifests and check health. It can’t apply your patch to its own brain and then use that newly-patched brain to understand the patch. It’s a classic chicken-and-egg scenario. Your patch to argocd-cm is part of the application manifest sync, but the logic needed to understand your custom resource health is needed *before* the sync even begins.

Warning: Trying to use a Kustomize patchesStrategicMerge on the argocd-cm from within your application’s manifests will never work for this reason. The change comes too late in the process.

The Fixes: From Emergency Patch to Permanent Solution

Alright, let’s get you unstuck. There are a few ways to solve this, ranging from “I need this fixed 30 seconds ago” to “Let’s do this the right way so it never happens again.”

Solution 1: The “Break Glass In Case of Fire” Method

It’s 2 AM, the release is blocked, and management is watching. You don’t have time for GitOps purity. You need to stop the bleeding. In this case, you can bypass Git and edit the ConfigMap directly in the cluster.

Just run this command:

kubectl edit configmap argocd-cm -n argocd

Then, manually paste your resource.customizations data into the ConfigMap, save it, and restart the argocd-application-controller pod to force it to reload the config. This will get you unblocked immediately. But let me be clear: this is a hack. You’ve now introduced drift between your Git repo and your cluster’s state, which is the very thing GitOps is meant to prevent. Use this only in an emergency, and promise me you’ll follow up with Solution 2.

Solution 2: The GitOps-Native (And Correct) Kustomize Fix

The right way to solve this is to treat ArgoCD’s configuration as part of your infrastructure-as-code, managed in Git, separate from your application code. You don’t *patch* the ConfigMap; you define its entire desired state in your Git repository.

Step 1: Create a full argocd-cm.yaml manifest.

Instead of a small patch file, you’ll have a complete manifest for the argocd-cm ConfigMap. If you don’t have it, you can grab the default one from the official ArgoCD installation manifests. Now, add your customizations directly into the data section.

# In your infra-repo/argocd/base/argocd-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  # ... other argocd config settings ...

  # YOUR CUSTOM HEALTH CHECK GOES HERE
  resource.customizations: |
    batch/CronJob:
      health.lua: |
        hs = {}
        -- A CronJob is always considered "Healthy" in our book,
        -- as its health is determined by its last scheduled job's success,
        -- which ArgoCD can't easily know. This just stops it from blocking syncs.
        hs.status = "Healthy"
        hs.message = "CronJob does not have a real-time health status"
        return hs

Step 2: Point Kustomize to your new base file.

Your Kustomize setup for deploying/managing ArgoCD itself should now use this local, customized ConfigMap file as a resource. Your kustomization.yaml for ArgoCD might look something like this:

# In your infra-repo/argocd/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: argocd

resources:
  - install.yaml # The main ArgoCD installation manifest
  - argocd-cm.yaml # YOUR customized ConfigMap!

Now, when you apply your ArgoCD manifests, the argocd-cm is created with the health check logic from the very beginning. The controller starts up with the right configuration, and when it sees your application’s CronJob, it knows exactly what to do.

Solution 3: The ‘It Just Works’ Helm Method

If you’re managing your ArgoCD installation with the official Helm chart (which many of us do), this is even easier. The chart maintainers have already solved this problem for you. You can inject the custom health check directly through your values.yaml file.

Simply add this snippet to your values file:

# In your values.yaml for the ArgoCD Helm release
server:
  config:
    resource.customizations: |
      batch/CronJob:
        health.lua: |
          hs = {}
          hs.status = "Healthy"
          hs.message = "CronJob does not have a real-time health status"
          return hs

When you run helm upgrade, Helm will render the argocd-cm ConfigMap template with your customizations already baked in. This is, in my opinion, the cleanest approach if you’re already in the Helm ecosystem.

Pro Tip: Don’t fight your tools. If your team manages cluster add-ons with Helm, use the Helm solution. If you’re a pure Kustomize shop, use the Kustomize solution. The “best” solution is the one that fits your team’s existing workflow and is easiest to maintain.

At the end of the day, that 2 AM incident taught us a valuable lesson: your GitOps tool is just another application, and its configuration needs to be managed with the same discipline as everything else. Don’t let a simple config blind spot turn your next deployment into a late-night fire drill.