DEV Community: Billy Walker

Progressive delivery on Kubernetes with Argo Rollouts

Billy Walker — Thu, 09 Jul 2026 19:05:06 +0000

Most teams meet the limits of a plain Kubernetes Deployment at the worst possible moment: mid-rollout, watching a new version march out to every pod while a graph slowly turns the wrong colour. A Deployment rolls pods out carefully enough — it respects readiness, it surges and drains in order — but it has one stubborn blind spot. Once the new ReplicaSet is Ready, it keeps going unless something stops it. There is no built-in way to say “send 10% of traffic here, wait, check an SLO, then continue”, or “only promote this preview stack if the metrics hold up”.

That missing control loop is exactly what Argo Rollouts adds. It keeps the Deployment mental model you already know and layers canary, blue-green, analysis, and automated rollback on top. For platform teams, that quietly turns rollout safety from a runbook someone has to remember into a controller that does it the same way every time. As of Argo Rollouts v1.9.0, that controller can drive percentage-based canaries, blue-green cutovers, and metric-gated promotion without asking you to rip out the rest of your Kubernetes delivery stack.

The gap in a normal Deployment

A Deployment is genuinely good at one thing: swapping one ReplicaSet for another while keeping enough healthy pods around to serve traffic. Readiness probes, maxUnavailable, and maxSurge all pull their weight here. What none of them give you is a first-class sense of whether the new version is actually any good.

That distinction matters most when “pods are Ready” and “this version is safe” quietly stop meaning the same thing. A service can start up cleanly and still come apart under real load. Error rate can sit flat through the first few requests and then climb once traffic arrives in earnest. Latency can drift at the 95th percentile while the average stays there looking reassuring. And a bad release often hurts just one path through the app — which is cold comfort when a straight rollout has already reached every user before anyone has the evidence to hit stop.

Argo Rollouts lives precisely in that gap. It is not a replacement for the Kubernetes scheduler, and it is not a GitOps controller — it does not want either job. It is a rollout controller that takes over the update strategy for a single workload and adds the parts a Deployment never had: promotion steps, pause points, analysis runs, and traffic-shaping integrations.

The mental model: Deployment behaviour, but with a promotion engine

The easiest way to hold Argo Rollouts in your head is “a Deployment with a state machine bolted on”. The Rollout resource still owns ReplicaSets and still reacts to pod template changes, but on top of that it keeps track of where a release currently sits in its promotion flow.

At a high level, the loop runs like this:

A new pod template lands in Git and is applied to the cluster.
The Rollout controller creates a new ReplicaSet.
The strategy decides what happens next: shift some traffic, create a preview stack, pause, or run analysis.
If the checks pass, the controller promotes the release.
If the checks fail, the controller aborts and keeps the stable version serving traffic.

That separation is really the whole idea. Kubernetes carries on running the pods, exactly as it did before. Argo Rollouts is the part that decides whether a new version has earned more traffic yet.

The pieces that matter

As of v1.9.0, the CRDs worth genuinely understanding are Rollout, AnalysisTemplate, ClusterAnalysisTemplate, AnalysisRun, and Experiment. That list looks longer than it needs to be, though — in day-to-day work most teams spend nearly all their time in the first three.

`Rollout`

The Rollout (apiVersion: argoproj.io/v1alpha1, kind: Rollout) sits at the centre of everything. It is deliberately built to be a drop-in replacement for a Deployment, which is why migrating an existing workload is mostly a matter of changing the apiVersion, the kind, and the strategy — the pod template underneath can stay much as it was.

A Rollout can either carry its own pod template or point at an existing Deployment through workloadRef. That second option is the gentle on-ramp: it lets you bring a live workload under Rollouts gradually, rather than rewriting the whole object in one nervous commit.

`AnalysisTemplate` and `AnalysisRun`

AnalysisTemplate describes the check; AnalysisRun is that check actually running during a rollout. The template might ask Prometheus for a success rate, a latency figure, or some service-specific business metric that only your team would think to measure. The run takes the answer and decides whether the rollout carries on, pauses, or aborts.

This is the part that makes Argo Rollouts something more than “rolling updates, but slower”. The controller is not simply counting down a timer between steps — it is waiting for evidence before it commits.

Canary vs blue-green

Argo Rollouts supports both, and it helps to remember that they are answers to different questions.

Canary is about gradual exposure. Raise the traffic a step at a time, watch what happens, then decide whether to keep going.
Blue-green is about a clean cutover. Stand the new version up beside the old one, satisfy yourself it works, then flip the active service across.

Neither is universally the right answer. Canary tends to be the better default for customer-facing services, where the whole point is for a bad version to touch as little traffic as possible before you catch it. Blue-green comes into its own when you want an obvious stable-versus-preview split and the reassurance of a fast switch back.

A concrete canary with analysis

The heart of a real Argo Rollouts setup is rarely the sheer volume of YAML — it is the promotion logic hiding inside it. The example below is deliberately small: a canary that shifts traffic in stages and refuses to promote unless a Prometheus-backed success-rate check agrees.

First, install the controller:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Then define the analysis template:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: checkout-success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 5m
      successCondition: result[0] >= 0.99
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            sum(
              rate(http_requests_total{service="{{args.service-name}}",status!~"5.."}[5m])
            ) /
            sum(
              rate(http_requests_total{service="{{args.service-name}}"}[5m])
            )

Then wire it into the rollout:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 8
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: ghcr.io/example/checkout:v2
          ports:
            - containerPort: 8080
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause:
            duration: 5m
        - analysis:
            templates:
              - templateName: checkout-success-rate
            args:
              - name: service-name
                value: checkout
        - setWeight: 30
        - pause:
            duration: 10m
        - setWeight: 100

The interesting part here is not the syntax — it is the sequence.

At 10% traffic, the rollout deliberately pauses long enough to gather some real request data. The AnalysisRun then measures the service against a threshold you wrote down in advance, and if that metric fails three times it aborts, rather than quietly waving the release on to 30% and then to everyone.

There is one subtlety worth internalising early. Without a traffic-routing integration, setWeight is an approximation drawn from ReplicaSet sizes rather than exact request-level shaping. At low replica counts that is often good enough, but it is not the same thing as a true 10% split. When you genuinely need that precision, wire Rollouts up to a traffic provider and give it stableService, canaryService, and trafficRouting. That is the line between “best effort, by pod count” and “deliberate, by request path”.

Where blue-green is simpler

Blue-green is the model most teams find easiest to explain out loud, mostly because it has fewer moving parts. There is an active stack, a preview stack, and a single moment of promotion between them.

In Rollouts, that shape becomes an activeService carrying live traffic and, optionally, a previewService pointed at the new ReplicaSet before it goes live. The part that makes it genuinely interesting is what you can hang either side of the cutover with prePromotionAnalysis and postPromotionAnalysis:

prePromotionAnalysis has to be satisfied that the preview version is good enough before the switch happens at all.
postPromotionAnalysis keeps watching once traffic has moved across, and can send it back to the old version if the new one starts to degrade.

The field I would commit to memory here is scaleDownDelaySeconds. It defaults to 30 seconds, giving the service selectors and traffic infrastructure time to converge before the old ReplicaSet is scaled away. It sounds like a trivial setting, but it is exactly the sort of thing that separates a rollout which “worked” as far as the controller was concerned from one that quietly dropped traffic down in the dataplane.

How this fits with GitOps

Argo Rollouts and Argo CD get along well, but it is worth being clear that they are doing two different jobs.

Argo CD reconciles the desired state in Git into the cluster. Argo Rollouts decides how a workload that has just changed should be promoted, once that change has landed. The boundary between them is clean, and that cleanness is the point.

In practice it means you do not have to smuggle rollout logic into your CI pipelines or bend your GitOps repo layout around it. Git still holds the desired pod template, Argo CD still applies it, and Rollouts takes over only for the runtime question of whether the new version deserves to keep advancing.

It also means rollback here is operational rather than declarative — a distinction that trips people up more often than you would expect. When a rollout aborts, Argo Rollouts does not go and rewrite Git on your behalf. It scales the release back to the stable version in the cluster, but the repository still holds the bad image tag until someone, or another controller, reverts it. That gap has teeth: if Argo CD is running with auto-sync and self-heal, it can read the aborted rollout as simply out of sync and try to re-apply the very version that just failed. Teams new to the model tend to walk straight past that boundary, and only notice it the first time the two controllers start pulling against each other.

Production reality: the sharp edges

Argo Rollouts demos beautifully and is surprisingly easy to misuse once real traffic is involved. The failure modes that actually bite are rarely exotic — they are the boring, avoidable ones.

Canary percentages are only exact if your traffic layer is exact

A setWeight: 10 step reads as though it means precisely ten percent, but without traffic routing it is really just a pod-count approximation. If your service runs seven replicas, there is no way for the controller to carve traffic into a clean tenth. When exact exposure genuinely matters, reach for a supported traffic manager rather than trusting the arithmetic.

Traffic-routed canaries can overprovision

The moment you add trafficRouting, the replica arithmetic shifts under you. maxSurge and maxUnavailable are no longer the whole story, particularly once setCanaryScale is in the mix, and it is easy to find yourself running more stable and canary pods at once than you had budgeted for during a promotion. That is usually the right trade for safety — just make sure it is a trade you planned for, rather than one you discover on the cluster.

Blue-green is simple, but not cheap

Running blue and green side by side doubles parts of your footprint by design — that is the whole mechanism, not a bug. If you lean on preview environments heavily, keep them short-lived. Rollouts is built for promotion windows measured in minutes, not for quietly maintaining a second long-lived environment that drifts further from the first with every passing day.

HPA needs to target the `Rollout`

If the workload is autoscaled, point the HPA at the Rollout itself — not the Deployment you migrated from, and certainly not an individual ReplicaSet. Get this wrong and you end up refereeing two controllers that hold different opinions about who owns the replica count, usually at the least convenient moment.

When Argo Rollouts is the right call

Reach for Argo Rollouts when the cost of a bad release is high enough that “the pods became Ready” simply is not a strong enough gate to stand behind. In practice that means customer-facing APIs, services with SLOs you actually measure, and teams who already have decent metrics but are still making the promotion call by hand, one deploy at a time.

What it is not is a decoration. Adopting it to make a deployment diagram look more sophisticated is a poor reason on its own. If a service is low-risk, its traffic layer is simple, and rollback is already fast, a plain Deployment may well remain the honest choice.

The real win was never that Rollouts hands Kubernetes yet another CRD. It is that it gives your release process a proper control loop: expose a little, measure what happens, then decide. That single step — pausing to look before committing — is the one a plain Deployment has never been able to take on its own.

EKS node autoscaling with Karpenter: just-in-time capacity without node groups

Billy Walker — Thu, 09 Jul 2026 19:01:51 +0000

Most people arrive at Karpenter after a slow build-up of small frustrations rather than one dramatic outage. Managed node groups start out simple, and then one day you notice you are spending real time deciding which group should be a little bigger, which instance family to add next, and how much headroom to leave "just in case". It starts to feel less like running a cluster and more like tending a spreadsheet.

Karpenter offers a gentler mental model. Instead of pre-sizing groups and hoping one of them fits, it watches the pods that cannot be scheduled, works out the capacity they actually need, and launches EC2 nodes to match. Two things shift as a result. Scaling up follows the workload rather than a template you wrote weeks ago, and scaling down becomes a policy you set once rather than a clean-up job you keep meaning to get to. On a busy cluster, that is often the difference between paying for a permanent buffer and only asking for capacity when there is real work waiting.

How Karpenter changes EKS node autoscaling

The simplest way to describe the change is this: Cluster Autoscaler works at the level of node groups, while Karpenter works at the level of individual pending pods. On paper that reads like a minor implementation detail. In day-to-day operation it turns out to be most of the story.

With Cluster Autoscaler, you make the important decisions up front — which instance families exist, how much spare room each group keeps, and which group grows when pods start to queue. Karpenter takes those same decisions and turns them into guardrails, then chooses the actual node shape at launch time, once it can see what is really being asked for.

That tends to suit clusters where the workloads do not all look alike:

web services that spike during deployments
queue workers whose demand follows traffic rather than CPU
short-lived CI or batch pods
teams mixing Spot and on-demand capacity in the same cluster

None of this means node groups are a mistake. They are simply static by design, and that is perfectly fine right up until your workload mix changes often enough that static sizing starts to feel expensive or brittle. That is usually the moment Karpenter starts to earn its place.

NodePool and EC2NodeClass split policy from AWS plumbing

On EKS, two Karpenter resources do most of the work: NodePool and EC2NodeClass.

NodePool describes scheduling and disruption policy — what kinds of nodes are allowed, how aggressively Karpenter can consolidate, and which workloads are welcome to land there.
EC2NodeClass describes the AWS side — subnets, security groups, AMI family, the IAM role or instance profile, storage, and the other EC2-specific details.

Keeping these separate is more useful than it first appears, because it holds your capacity intent apart from the cloud plumbing underneath it. A single EC2NodeClass can back several NodePools, and each NodePool can express a different set of rules about the workloads it serves.

A minimal setup looks like this:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ['c', 'm', 'r']
        - key: karpenter.sh/capacity-type
          operator: In
          values: ['spot', 'on-demand']
        - key: kubernetes.io/arch
          operator: In
          values: ['amd64']
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 10m

Two things are worth remembering here, and the docs are clear about both. Karpenter will not do anything at all until at least one NodePool exists — there is no implicit default waiting in the background. And a pod's own scheduling constraints are intersected with the NodePool requirements, so if a workload asks for something the pool does not allow — an instance family, a zone, an architecture — Karpenter simply will not launch a node for it. It sounds obvious written down, but it is one of the more common ways for a pending pod to look like a capacity shortage when it is really a policy mismatch.

Why just-in-time provisioning beats a larger buffer

The traditional habit on EKS is to keep enough spare capacity around that the next deployment or burst will probably fit. The trouble with "probably" is that it quietly shows up on the bill every month, whether or not you ever use it.

Karpenter lets you run a little closer to the line, because it provisions from real pending demand rather than a guess. If a rollout suddenly needs six more replicas, it can pick capacity that fits those pods instead of inflating a general-purpose group and hoping the shape is close. When the burst passes, the same controller can quietly remove or replace whatever it over-provisioned.

This helps most when your requests are uneven. A memory-heavy worker, an ARM-only build job, and a stateless frontend rollout do not really want to share a node. Node groups force you to approximate and accept some waste, whereas Karpenter can stay much closer to what each workload actually asked for.

The catch is that your constraints have to be explicit. Instance family, architecture, capacity type, disruption tolerance — all of it needs to be written down properly. Karpenter takes away the provisioning ceremony, but it does not take away the thinking.

Consolidation, drift, and interruption handling are the real day-two features

Fast provisioning is the headline, but the part you live with day to day is what happens after a node has landed.

Karpenter's disruption controller looks after two cases that quietly save a lot of effort:

Consolidation removes empty nodes, or swaps an underutilised one for a cheaper shape when the same pods can comfortably run elsewhere.
Drift replaces nodes that have fallen out of step with the current NodePool or EC2NodeClass — a new AMI selection, say, or changed launch requirements.

In a node-group-heavy setup, those two jobs — right-sizing and AMI rollovers — usually turn into separate pieces of scheduled maintenance. Karpenter keeps comparing what you asked for with what is actually running, so the fleet does not slowly drift out of shape between maintenance windows.

There is an honest caveat here, and the Karpenter docs do not hide from it: consolidation is only ever as good as the disruption rules you allow. Preferred anti-affinity and topology spread can quietly stop nodes from consolidating even when the scheduler could place the pods elsewhere, and PodDisruptionBudgets can do the same. So when someone says Karpenter is not saving them any money, the controller is rarely the first place to look. More often it is the workload policy wrapped around the nodes it is trying to drain.

Interruption handling matters too, especially if you lean on Spot. Karpenter can react to a Spot interruption warning, bring up replacement capacity ahead of time, and drain the old node in parallel — but all of that still depends on the workload tolerating disruption. Two minutes is not long when graceful shutdown, PDBs, and placement rules are all pulling in different directions.

The cost story here is the unglamorous one: a cluster that already had plenty of capacity in aggregate, just trapped in the wrong shape. Consolidation and drift are what turn that from a tidy architecture diagram into an actual lower bill.

Spot works well when the fallback path is real

Karpenter and Spot are often mentioned in the same breath, and for good reason — it can choose from a far wider range of instance types than a hand-maintained node-group layout, which is exactly what makes Spot workable. The part worth dwelling on is that Spot only pays off when the fallback is genuinely there.

A sensible EKS pattern usually looks something like:

prefer Spot for workloads that can tolerate interruption
allow on-demand in the same NodePool for when the service cannot wait for spare Spot capacity to appear
keep topology spread and PDBs sensible, so replacing one node does not set off a second incident
test your interruption handling before you actually need it

That last point carries more weight than it might seem. "We allow on-demand fallback" is a line of configuration; proving that a constrained workload can still schedule when a Spot pool dries up is a different thing entirely, and a good deal more reassuring when it happens at an awkward hour.

Where Karpenter is better than Cluster Autoscaler, and where it is not

Karpenter tends to win out over Cluster Autoscaler when an EKS cluster carries a genuine mix of workloads and their shapes change often. It is good at turning pending work into right-sized nodes without asking you to pre-bake every possible capacity combination into managed groups ahead of time.

It is a less obvious choice when the fleet is already settled — the node groups are well understood, the workload mix barely moves, and nothing about the current setup is causing pain. If what you have today is simple and cheap, Karpenter can end up being a sophisticated answer to a question nobody was really asking.

So the useful question is not whether Karpenter is newer or cleverer, because it is comfortably both. It is whether managing node groups has quietly become a drag on your time. If it has, the added complexity is usually a fair trade. If it has not, a well-kept Cluster Autoscaler setup may still be all you need.

Takeaway

Karpenter moves EKS node autoscaling away from managing node groups and towards provisioning that follows the workload. You describe the policy once through NodePool and EC2NodeClass, and from then on capacity is launched to match the pods actually in front of the scheduler, rather than the guess someone made last quarter.

The real payoff is not only quicker scale-up. It is the combination of right-sized launches, consolidation, drift replacement, and Spot fallback that holds together under pressure. Between them, consolidation and Spot can take a meaningful bite out of node spend without making scale-up feel fragile, though the exact number will always depend on your particular workload mix.

If you are already spending your week wrestling oversized node groups, slow scale-up, or a fleet that drifts out of shape between maintenance windows, Karpenter is well worth a serious look on EKS.