Originally published on kuryzhev.cloud
The scenario
Argo CD said the deploy was "Healthy" — it just meant the manifests applied, not that checkout stopped returning 500s. We pushed a checkout service change, CI passed, Argo CD synced clean, dashboard all green. Ninety seconds later, our alerting channel lit up with 500s from real customers hitting the checkout flow.
The root cause wasn't subtle once we looked: Argo CD's health check for a plain kubectl.io/Deployment only confirms pods are running and ready. It has no opinion on whether the app is returning correct responses. We had auto-sync on, no canary step, no automated rollback — a bad image went from zero to 100% of pods in a single sync cycle. By the time a human noticed the error rate spike, every replica was already serving the broken build.
This is the setup we ended up with instead: swap the Deployment for an Argo Rollouts canary that ships to a small traffic slice first, gated by real Prometheus SLIs — error rate and p99 latency — before it's allowed to progress further. Argo CD's job goes back to what it's actually good at: keeping the cluster in sync with Git. Health judgment moves to Argo Rollouts, where it belongs.
Prerequisites
Before touching the Rollout spec, make sure these pieces are actually in place — skipping one of these is the fastest way to get stuck halfway through.
- Argo CD (a recent 2.x release) plus the Argo Rollouts controller and the
kubectl argo rolloutsplugin installed in-cluster. The Rollout CRD needs to be registered before you apply anything referencingkind: Rollout. - A metrics backend reachable from inside the cluster for AnalysisTemplates. We use Prometheus below, but Datadog and CloudWatch providers follow the same pattern — same fields, different provider block.
- A traffic-splitting layer that supports weighted routing: Istio VirtualService, NGINX ingress canary annotations, or Gateway API HTTPRoute. Pick one before you write the Rollout spec — the
trafficRoutingblock is provider-specific and you can't mix and match mid-tutorial. - An Argo CD Application already pointing at the app's manifests (Helm or Kustomize), currently deploying a plain Deployment. This gets swapped out for a Rollout, not layered on top of it — running both will just confuse the selector.
Step 1: Replace the Deployment with a Rollout
The pod spec doesn't change at all. Same containers, same labels, same selectors. The only structural change is kind: Deployment becoming kind: Rollout (from the argoproj.io/v1alpha1 API group), and spec.strategy turning into a canary block with explicit steps.
Start conservative. We use setWeight: 10 followed by a two-minute pause, then setWeight: 50 with another pause, then 100%. Tune the step count and weights to your blast radius tolerance, not to gut feeling — a service handling payment traffic should sit at 10% longer than an internal reporting tool.
# rollout.yaml — canary Rollout replacing a plain Deployment
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: checkout-service
spec:
replicas: 6
selector:
matchLabels:
app: checkout-service
template:
metadata:
labels:
app: checkout-service
spec:
containers:
- name: checkout-service
image: registry.example.com/checkout-service:__TAG__
ports:
- containerPort: 8080
strategy:
canary:
# traffic split target — must match your mesh/ingress resource
trafficRouting:
istio:
virtualService:
name: checkout-service-vs
routes:
- primary
steps:
- setWeight: 10
- pause: {duration: 2m}
- analysis:
templates:
- templateName: checkout-slo-check
- setWeight: 50
- pause: {duration: 3m}
- setWeight: 100
Watch out for this one: if your Argo CD Application has syncPolicy.automated.selfHeal: true, Argo CD will try to "fix" the Rollout mid-canary. A paused Rollout's scaled-down replica count looks like drift to Argo CD, and selfHeal will happily fight the canary controller for control of spec.replicas. We got bitten by this the first time — the canary kept snapping back to full replicas before the pause finished. Fix it by adding ignoreDifferences scoped to the Rollout kind, or disable selfHeal during release windows.
Step 2: Add health gates with an AnalysisTemplate
A pause step by itself just buys you time — someone still has to decide whether to promote. An AnalysisTemplate turns that pause into an automated go/no-go decision based on actual metrics instead of a person staring at a dashboard.
Define the template with a Prometheus query for error rate and one for p99 latency. Set successCondition and failureCondition as PromQL thresholds — not raw request counts, which don't normalize across traffic volume. Reference the template from the Rollout's step list so it runs automatically at the pause point.
# analysistemplate.yaml — health gate using real SLIs, not CPU/memory
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: checkout-slo-check
spec:
metrics:
- name: error-rate
interval: 30s
failureLimit: 2
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
sum(rate(http_requests_total{app="checkout-service",status=~"5.."}[2m]))
/
sum(rate(http_requests_total{app="checkout-service"}[2m]))
successCondition: result[0] < 0.02
failureCondition: result[0] >= 0.02
- name: p99-latency
interval: 30s
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket{app="checkout-service"}[2m]))
successCondition: result[0] < 0.8
failureCondition: result[0] >= 0.8
The most common mistake I see teams make here is gating promotion on CPU or memory instead of a business-facing SLI. A canary can look perfectly healthy on resource metrics while quietly returning wrong responses or timing out downstream calls. Gate on what users actually feel — error rate, latency, business events like completed checkouts — not infrastructure vitals that don't correlate with correctness.
Also tune interval deliberately. Set it too short and you'll get flapping on naturally noisy metrics, aborting good deploys for no reason. Set it too long and you delay rollback, which widens the blast radius exactly when you're trying to shrink it. There's no universal default — tune against your metric's actual variance.
Step 3: Wire in traffic splitting and Argo CD health awareness
None of this matters if the canary pods aren't receiving real traffic. Configure trafficRouting in the Rollout spec to match your mesh or ingress choice — Istio VirtualService/DestinationRule names, NGINX canary annotations, or a Gateway API HTTPRoute. This is what makes setWeight steps actually shift live requests, not just replica counts sitting idle.
Current Argo CD versions ship built-in resource health logic for argoproj.io/Rollout. Confirm it's active — it's what makes the Argo CD UI correctly show "Progressing" during a canary window instead of a misleading "Healthy" the moment pods come up. Without it, Argo CD reads Rollout status the same way it reads a Deployment, and you're back to the original problem.
Finally, add an Argo CD Notifications trigger on on-degraded and on-analysis-run-failed for the Rollout resource. A failed AnalysisRun should page someone, not sit silently paused waiting for a person to check the dashboard. We wired ours into the same Slack channel as our other alerts — no separate tooling needed.
One more thing worth budgeting for: canary windows mean stable and canary replica sets run simultaneously for the entire pause duration. If you're running multiple sequential steps, that overlap compute cost adds up — factor it into capacity planning, especially for services with tight resource quotas.
Verify and test
Don't trust this setup until you've watched it fail on purpose. Start by running a live deploy and watching the rollout in real time:
# live view of weight, revision, and AnalysisRun status together
kubectl argo rollouts get rollout checkout-service --watch
The test that actually matters isn't a clean deploy — it's a deliberately broken one. Ship an image with an injected 500-error handler or artificial latency and confirm the AnalysisRun trips failureCondition, aborts automatically, and scales traffic back to the stable version. If your gate can't catch a deploy you know is broken, it can't be trusted to catch one you don't know about.
Also exercise manual override at least once, so operators know the escape hatch works when automation gets it wrong:
kubectl argo rollouts promote checkout-service # force progression past a pause
kubectl argo rollouts abort checkout-service # roll back to stable immediately
Check that Argo CD reflects the abort as expected sync state, not as an error condition — if ignoreDifferences is scoped correctly from Step 1, this should be quiet. The kubectl argo rollouts dashboard command gives a local UI for rollout history and analysis runs if you want a visual trail after the fact.
Canary deployments with AnalysisTemplates add real operational overhead — longer deploy windows, extra compute running two replica sets side by side, and another CRD your team has to reason about during incidents. For a low-traffic internal tool, a plain rolling update with a decent post-deploy smoke test is often the more honest choice; you're not paying for machinery you don't need. But for something like checkout, where an incident costs real revenue, catching a bad release automatically at 10% traffic instead of manually at 100% is worth every bit of that overhead. We haven't had a full-blast bad deploy reach checkout since we made this switch, and that alone paid for the setup time. If you're evaluating this for your own stack, start with one high-traffic service, get the AnalysisTemplate gates right, and expand from there — don't roll it out cluster-wide on day one. For more on rollout strategies and rollback tradeoffs, see our DevOps_DayS archive.
Top comments (0)