DEV Community

Florient DOGBE
Florient DOGBE

Posted on

Karpenter vs Cluster Autoscaler on EKS: migrate with zero downtime and halve the bill

#aws #kubernetes #karpenter #eks

Node autoscaling on EKS has two serious options: the venerable Cluster Autoscaler (CA) and Karpenter. They solve the same problem — "I have pending pods, give me capacity" — but with radically different models. CA reasons in terms of Auto Scaling Groups you pre-declared; Karpenter reasons in terms of pod requirements and picks EC2 instance types on the fly.

On real clusters, moving from CA to Karpenter typically cuts the EC2 bill by 40–60%, not because of magic, but because of three compounding effects: bin-packing on the right instance size, aggressive consolidation, and much easier Spot adoption. This article is a field comparison plus a step-by-step migration you can run on a production cluster without downtime.

Context: why Cluster Autoscaler hits a ceiling

Cluster Autoscaler works at the ASG level. For each node group it simulates "if I add one node of this shape, do the pending pods fit?". That design has structural consequences:

  • You own the shape catalog. Need m6i.large, c6i.2xlarge, r6i.4xlarge and an ARM variant? That's 4+ node groups, each with its own launch template, labels and taints. Multiply by AZ when you need zonal affinity for EBS.
  • Instances within a group must be homogeneous. CA assumes all nodes in an ASG have the same CPU/memory. Mixed instance policies work only if the shapes are equivalent, which kills Spot diversification (the single biggest factor in Spot interruption rate).
  • Scale-down is timid. Default --scale-down-unneeded-time is 10 minutes and the utilization threshold is 50%. CA removes empty-ish nodes; it never says "these three half-empty m5.2xlarge could be one m6i.2xlarge".
  • Fragmentation is permanent. After a week of deploys, you end up with 30% unused reserved capacity spread across nodes. Nobody notices because every node looks "half used".

Karpenter removes the node group abstraction. It watches unschedulable pods, computes the cheapest instance type (across hundreds of candidates) that satisfies their combined requirements, and launches it directly via the EC2 Fleet API. Then it continuously re-evaluates whether the current fleet can be replaced by something cheaper.

Cluster Autoscaler Karpenter
Unit of scaling ASG / managed node group NodeClaim → single EC2 instance
Instance selection Pre-defined per group Dynamic, price-aware, 100s of types
Typical node ready time 2–4 min (ASG + LT) 40–90 s
Scale-down Empty/underused nodes Consolidation (merge, replace, delete)
Spot Mixed instance policy, manual diversification Native, price-capacity-optimized, interruption queue
Upgrades New node group + drain Drift detection + automated rolling replacement
Config surface Flags + N ASGs 1 NodePool + 1 EC2NodeClass

Installing Karpenter next to Cluster Autoscaler

The two can coexist, and that is exactly what makes the migration safe. The rule: Karpenter-provisioned nodes must not be part of any ASG CA manages, and the Karpenter controller itself must run on nodes it does not own (otherwise it can evict itself and deadlock). Keep a small managed node group (2 × m6i.large, across 2 AZs) for the controller and other system components.

Prerequisites: the controller IAM role (IRSA or EKS Pod Identity), the node IAM role, and the SQS interruption queue. If you use the official CloudFormation/Terraform modules, they create all of it.

export CLUSTER=prod-eks
export AWS_ACCOUNT=111122223333

helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version 1.1.1 \
  --namespace kube-system \
  --set "settings.clusterName=${CLUSTER}" \
  --set "settings.interruptionQueue=karpenter-${CLUSTER}" \
  --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=arn:aws:iam::${AWS_ACCOUNT}:role/KarpenterController-${CLUSTER}" \
  --set "nodeSelector.karpenter\.sh/controller=true" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.memory=1Gi \
  --wait
Enter fullscreen mode Exit fullscreen mode

Tag the subnets and security groups Karpenter must discover — this is the #1 cause of "no instance type satisfied the requirements" errors:

aws ec2 create-tags --resources subnet-0a1b2c3d subnet-0e4f5a6b \
  --tags Key=karpenter.sh/discovery,Value=${CLUSTER}

aws ec2 create-tags --resources sg-0123456789abcdef0 \
  --tags Key=karpenter.sh/discovery,Value=${CLUSTER}
Enter fullscreen mode Exit fullscreen mode

Writing your EC2NodeClass and NodePool

The EC2NodeClass is the AWS-specific part: AMI, subnets, security groups, disks, IMDS, kubelet. The NodePool is the Kubernetes-facing part: scheduling constraints, limits, disruption policy.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: KarpenterNodeRole-prod-eks
  amiSelectorTerms:
    # Pin the AMI version. "al2023@latest" is convenient but makes your
    # fleet drift the day AWS publishes a new AMI.
    - alias: al2023@v20241024
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: prod-eks
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: prod-eks
  metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpTokens: required          # IMDSv2 only
    httpPutResponseHopLimit: 1    # containers cannot reach IMDS
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 80Gi
        volumeType: gp3
        iops: 3000
        throughput: 125
        encrypted: true
        kmsKeyID: alias/eks-node-volumes
        deleteOnTermination: true
  kubelet:
    maxPods: 110
    systemReserved:
      cpu: 100m
      memory: 300Mi
      ephemeral-storage: 2Gi
    evictionHard:
      memory.available: 5%
      nodefs.available: 10%
  tags:
    Environment: production
    ManagedBy: karpenter
    CostCenter: platform
Enter fullscreen mode Exit fullscreen mode 
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  weight: 50
  limits:
    cpu: "2000"
    memory: 8000Gi
  template:
    metadata:
      labels:
        workload-class: general
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 336h          # force node rotation every 14 days
      terminationGracePeriod: 2h # hard ceiling on drain time
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.k8s.aws/instance-size
          operator: NotIn
          values: ["nano", "micro", "small", "metal"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m
    budgets:
      - nodes: "15%"                 # baseline: never churn more than 15%
      - nodes: "0"                   # freeze voluntary disruption in business hours
        schedule: "0 8 * * mon-fri"
        duration: 10h
        reasons: ["Underutilized", "Drifted"]
Enter fullscreen mode Exit fullscreen mode

Two things matter here. First, be generous with instance types: the more candidates, the cheaper the result and the lower the Spot interruption rate. Restricting to one family is the most common self-inflicted cost problem. Second, weight lets you layer NodePools — a weight-100 pool with Reserved Instance-matching shapes, falling back to a weight-50 general pool.

For Spot-only workloads with a Spot/on-demand fallback, use two NodePools:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-batch
spec:
  weight: 100
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      taints:
        - key: workload
          value: batch
          effect: NoSchedule
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-cpu
          operator: In
          values: ["8", "16", "32"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s
Enter fullscreen mode Exit fullscreen mode

The progressive migration, step by step

1. Verify Karpenter provisions correctly

Deploy a throwaway workload with a nodeSelector that only Karpenter can satisfy:

kubectl create deployment inflate --image=public.ecr.aws/eks-distro/kubernetes/pause:3.7 --replicas=0
kubectl set resources deployment inflate --requests=cpu=1,memory=1.5Gi
kubectl patch deployment inflate -p '{"spec":{"template":{"spec":{"nodeSelector":{"workload-class":"general"}}}}}'
kubectl scale deployment inflate --replicas=12

kubectl get nodeclaims -w
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -f --tail=50
Enter fullscreen mode Exit fullscreen mode

You should see a NodeClaim, then a node, in under 90 seconds. Scale back to 0 and confirm the node disappears after consolidateAfter.

2. Stop CA from growing

Do not delete CA yet. Just freeze the managed node groups by setting maxSize == desiredCapacity, so new capacity can only come from Karpenter:

aws eks update-nodegroup-config --cluster-name prod-eks \
  --nodegroup-name workers-a \
  --scaling-config minSize=0,maxSize=6,desiredSize=6
Enter fullscreen mode Exit fullscreen mode

3. Drain node groups in waves

Never drain everything at once. Take 20–25% of the old nodes per wave, and check your PDBs first:

# List the legacy nodes (no Karpenter label)
kubectl get nodes -l '!karpenter.sh/nodepool' -o name

# Check PDBs that could block the drain
kubectl get pdb -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed

for node in $(kubectl get nodes -l '!karpenter.sh/nodepool' -o name | head -n 4); do
  kubectl cordon "$node"
done

for node in $(kubectl get nodes -l '!karpenter.sh/nodepool' --field-selector spec.unschedulable=true -o name | head -n 4); do
  kubectl drain "$node" \
    --ignore-daemonsets \
    --delete-emptydir-data \
    --grace-period=120 \
    --timeout=10m
done
Enter fullscreen mode Exit fullscreen mode

Pods become pending, Karpenter provisions new capacity, and you shrink the ASG:

aws eks update-nodegroup-config --cluster-name prod-eks \
  --nodegroup-name workers-a \
  --scaling-config minSize=0,maxSize=6,desiredSize=2
Enter fullscreen mode Exit fullscreen mode

Repeat until only the controller node group remains. Then scale CA to zero replicas, observe for 48h, and remove it.

kubectl scale deployment cluster-autoscaler -n kube-system --replicas=0
Enter fullscreen mode Exit fullscreen mode

Consolidation pitfalls

Consolidation is where the savings come from — and where incidents come from if you skip the guardrails.

Pods without a controller block consolidation. A bare Pod (no Deployment/Job owner) makes Karpenter refuse to disrupt the node. Good: it protects stateful one-offs. Bad: debug pods left running pin an entire node for days.

PodDisruptionBudgets are mandatory, not optional. Karpenter honours PDBs during drain. Without them, a two-replica Deployment on the same node can go to zero during a consolidation.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
  namespace: shop
spec:
  minAvailable: 70%
  selector:
    matchLabels:
      app: checkout-api
Enter fullscreen mode Exit fullscreen mode

Long-running jobs need an opt-out. Use the annotation on the pod (or the node for a manual freeze):

apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-reindex
spec:
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"
    spec:
      restartPolicy: Never
      containers:
        - name: reindex
          image: registry.example.com/reindex:1.4.2
Enter fullscreen mode Exit fullscreen mode

Spot-to-Spot consolidation is conservative by design. Karpenter only replaces a Spot node with a cheaper Spot node when at least 15 cheaper instance types remain available, to avoid pushing the fleet into low-capacity pools. If your requirements are narrow, you will never see Spot consolidation — widen instance-category and instance-generation.

Spot interruptions are handled, not avoided. With settings.interruptionQueue set, Karpenter consumes EC2 Spot interruption notices, rebalance recommendations and scheduled maintenance events from SQS, cordons and drains the node immediately. You get ~110 seconds of drain — make sure terminationGracePeriod on your pods is below that for Spot workloads, and never put a database on Spot.

Topology spread still applies. Karpenter is topology-aware, but whenUnsatisfiable: DoNotSchedule on topology.kubernetes.io/zone combined with EBS zonal volumes can generate one node per AZ for 3 pods. Expect it, and size minDomains accordingly.

Measuring the real gain

Do not trust "it feels cheaper". Instrument it:

# Node count and capacity type distribution
kubectl get nodes -L karpenter.sh/capacity-type,node.kubernetes.io/instance-type,karpenter.sh/nodepool

# Cluster-wide request vs allocatable ratio
kubectl describe nodes | grep -A5 "Allocated resources" | grep -E "cpu|memory"
Enter fullscreen mode Exit fullscreen mode

Karpenter exposes Prometheus metrics on port 8080: karpenter_nodeclaims_created_total, karpenter_nodeclaims_disrupted_total, karpenter_voluntary_disruption_decisions_total, karpenter_pods_startup_duration_seconds. Track the last one closely: if p99 pod startup degrades after migration, your NodePool requirements are too narrow.

On the AWS side, activate karpenter.sh/nodepool and your own CostCenter tag as cost allocation tags, then compare EC2 spend week-over-week in Cost Explorer. The two numbers that actually move: average CPU request/allocatable ratio (expect 45% → 75%) and Spot share of vCPU hours (expect 0–20% → 60–80% on stateless clusters).

Security best practices

  • IMDSv2 with hop limit 1. httpTokens: required + httpPutResponseHopLimit: 1 prevents a compromised container from reaching the instance metadata endpoint and stealing node credentials. This is the single highest-value setting in the EC2NodeClass.
  • Least-privilege node role. The node IAM role should carry only AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly and the CNI policy. Application permissions go through IRSA or EKS Pod Identity, never through the node role.
  • Scope the controller role. The Karpenter controller policy must restrict ec2:RunInstances / ec2:CreateTags with a condition on aws:RequestTag/kubernetes.io/cluster/<cluster> and iam:PassRole to the single node role ARN. Otherwise anyone with NodePool write access can launch arbitrary instances with arbitrary roles.
  • Treat NodePool/EC2NodeClass as privileged objects. They are cluster-scoped and define userData executed as root on every node. RBAC on karpenter.sh and karpenter.k8s.aws API groups belongs to platform admins only; manage them through GitOps with mandatory review.
  • Pin AMIs, use drift to roll them. alias: al2023@latest means AWS controls when your fleet rotates. Pin a version, bump it in Git; Karpenter detects drift and replaces nodes within your disruption budgets.
  • Encrypt volumes with a CMK and keep deleteOnTermination: true so ephemeral node disks never outlive the node.
  • Cap node lifetime. expireAfter: 336h guarantees no node runs an unpatched kernel for months — a free hardening win Cluster Autoscaler never gave you.
  • Private subnets only. Select subnets by tag, and verify no public subnet carries the karpenter.sh/discovery tag; a mis-tagged subnet silently puts workloads on public IPs.

Key takeaways

  • Karpenter replaces N node groups with one or two NodePools and picks instance types per pod requirement set — that alone recovers the 25–30% of capacity fragmentation CA leaves behind.
  • The migration is safe because both autoscalers coexist: freeze the managed node groups, drain in waves of 20–25% with kubectl drain --ignore-daemonsets --delete-emptydir-data, shrink the ASG, repeat.
  • Savings come from breadth: wide instance-category/instance-generation requirements, Spot plus on-demand fallback, and consolidationPolicy: WhenEmptyOrUnderutilized.
  • Consolidation needs guardrails: PDBs on every production Deployment, karpenter.sh/do-not-disrupt on long jobs, and disruption budgets with a business-hours freeze window.
  • Harden the EC2NodeClass on day one: IMDSv2 hop limit 1, encrypted gp3 with a CMK, pinned AMI, minimal node role, private subnets.
  • Measure with karpenter_pods_startup_duration_seconds, request/allocatable ratio and Spot vCPU share before claiming the win.

Top comments (1)

Collapse
 
abhinavs1920 profile image
Abhinav Sharma •

The homogeneous-ASG constraint is the quiet cost driver, it blocks Spot diversification, which is where most of the savings actually come from. Karpenter reasoning from pod requirements instead of a predeclared shape catalog is the real model change, consolidation is just the visible part.