Your Argo CD Addons Are Only Tested in Production. Admit It.

Somebody on r/kubernetes asked a plain question this week: how do you test your Argo CD addons before merging?

The pain underneath it will sound familiar. Spin up a whole new cluster to test one addon change and you wait — the thread put it at roughly 45 minutes. Test on the shared dev cluster instead and you are not testing your change any more. You are testing your change plus three other people’s changes, all stepping on each other. Test on a local cluster and it lies politely, because your laptop has no VPC, no IAM, no real network policy — none of the things that actually break addons in production.

So most teams do the honest, terrible thing: they merge and watch production. Argo CD (a GitOps tool — it watches a Git repo and makes the cluster match it) becomes the test environment. That is not a testing strategy. That is hope with a dashboard.

Your addon does not need a bigger test cluster. It needs cheaper questions, asked earlier.

First, understand what an addon really is

An addon is the platform layer you install on a cluster so applications can live there: ingress controller (the traffic cop that routes outside requests to the right service), cert-manager, external-dns, monitoring agents, policy engines. Change one value in one of these and you can break every app on the cluster at once. That is why addon changes feel scary — the blast radius (how much breaks when it breaks) is the whole cluster, wearing a small YAML costume.

And here is the timing problem. This month alone, Microsoft made its managed Argo CD extension for AKS generally available (October 9), and Argo CD 3.5 landed with a pile of security fixes in October. More clusters are getting Argo, more addons are moving through Git — and the merge button is still the only real test most teams run.

Gate 1: render it. No cluster. Seconds.

The cheapest test is the one that never touches a cluster. In CI (continuous integration — the robot that runs checks on every pull request), render the addon exactly the way Argo CD will:

helm template my-addon ./addons/my-addon -f values.yaml > rendered.yaml
kustomize build addons/my-addon/overlays/dev > rendered.yaml

helm template asks Helm (a package manager for Kubernetes) to print the final YAML without installing anything. kustomize build does the same for Kustomize overlays (patches layered on base YAML). If this fails — a missing value, a broken template, a typo in a values file — you just saved a cluster spin-up. Run the render for every environment overlay, not just dev: the prod values file is exactly the one nobody renders until prod renders it for them.

Gate 2: schema-check the rendered YAML

Valid YAML is not valid Kubernetes. A file can parse perfectly and still name a field that does not exist in that API version. Tools like kubeconform check the rendered output against the real Kubernetes schemas in CI:

kubeconform -strict -summary rendered.yaml

This catches the quiet killers: an apiVersion the new cluster version dropped, a field renamed between versions, a string where the schema wants a number. It still proves nothing about behaviour — it proves the cluster will at least understand the sentence you are about to say to it.

Gate 3: diff it against the live cluster, before merge

This is the gate most teams skip, and it is the one that answers the only question a reviewer really has: what will actually change?

argocd app diff my-addon --local ./addons/my-addon

The --local flag is the whole trick: the Argo CD CLI compares your local, uncommitted files against what is live in the cluster, running your Helm or Kustomize rendering the same way the app defines it. Nothing is pushed to Git, nothing is synced. Post that diff on the pull request. A reviewer reading a values diff is guessing; a reviewer reading a rendered diff is reviewing.

A values diff shows what you typed. A rendered diff shows what the cluster gets. Review the second one.

Gate 4: if it must touch a cluster, give it its own

Some things only a real cluster can tell you: a webhook that rejects your object, a CRD (custom resource definition — your own extra Kubernetes object type) that must exist before the thing that uses it, a DaemonSet that will not schedule. Fair. But that cluster time should be a namespace, not a cluster, and it should belong to one pull request.

The pattern: CI creates a preview Application for the PR, pointed at a throwaway namespace on an existing non-prod cluster, lets Argo sync it, waits for Healthy, runs a smoke check, then deletes the namespace when the PR closes. Your VPC/IAM problem is real — a preview on a shared non-prod cluster inside the same cloud account gets you real IAM roles and real network policy, which a laptop cluster never will. What it does not get is other testers: the namespace is yours alone, so red actually means your change.

Gate 5: roll it out in stages, not in one merge

For addon fleets across many clusters, Argo CD’s ApplicationSet Progressive Syncs exist for exactly this: applications roll out stage by stage, and the next stage waits until the current stage’s apps report Healthy. It has been a beta feature since Argo CD v2.6.0 and needs the ApplicationSet controller flag (--enable-progressive-syncs) turned on. One cluster group proves the addon, then the next group moves. A bad addon dies in the canary group instead of touring the whole fleet.

The pre-merge checklist

• Render: helm template or kustomize build runs in CI for every overlay, including prod values.

• Schema: rendered output passes a strict schema check (for example kubeconform).

• Diff: argocd app diff --local output is posted on the PR and a human read it.

• Isolation: anything needing a cluster syncs to a per-PR namespace — never the shared dev free-for-all.

• Health: the preview app reaches Healthy and a smoke check passes before merge is allowed.

• Stages: fleet addons roll out with staged, health-gated promotion — not one sync to everywhere.

• Cleanup: closing the PR deletes its preview namespace automatically.

• Staging warning: test this whole flow in staging first before trusting it in production.

Notice what is missing from that list: a 45-minute cluster per pull request. You do not need more clusters. You need the questions in the right order — render, schema, diff, preview, stages — so the expensive cluster only ever answers the one question the cheap gates could not.

The fix, in one line: stop testing addons by merging them. Render first, diff second, preview in isolation, and let production be the last place your YAML surprises you — not the first.

Disclaimer

• This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.

• I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.

• Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.

• Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.

• Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

---

© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

Writing on monitoring, SRE, Terraform, Kubernetes and platform engineering — one hard-won lesson at a time.

Tell me in the replies: which gate does your team skip? That answer is usually where your next addon incident is already waiting.

Examples are illustrative — validate in your own environment before running in production.