DEV Community

Cover image for Enabling Karpenter on EKS with Terraform: What It Is, Why It's Worth It, and How to Set It Up
Ajinkya
Ajinkya

Posted on Originally published at Medium

Enabling Karpenter on EKS with Terraform: What It Is, Why It's Worth It, and How to Set It Up

#aws #kubernetes #terraform #karpenter

Karpenter replaces the old "guess an instance type, set a min/max, wait for Cluster Autoscaler" model with something simpler: watch for pods that can't schedule, and launch exactly the EC2 capacity they need. This guide covers how it actually works, why teams adopt it, and the Terraform you need to wire up to run it on EKS.

Prerequisites

Before you begin, make sure you have:

  • A running Amazon EKS cluster (created through either Terraform, eksctl, or the console)
  • Terraform (v1.11+ recommended)
  • An existing VPC and subnets
  • Existing security groups
  • Helm (v3+)
  • kubectl configured to access your EKS cluster
  • AWS CLI with credentials configured

How Karpenter Actually Works

Traditional autoscaling works in two disconnected layers: the Kubernetes scheduler decides where pods should go, and a separate Cluster Autoscaler watches Auto Scaling Groups and adds nodes when they're full. The ASG only knows one instance type per group, so it's constantly a step behind what the scheduler actually needs.

Karpenter collapses those two layers into one loop:

  1. A pod goes Pending because nothing can schedule it.
  2. Karpenter looks at that pod's actual requirements CPU, memory, architecture, zone, GPU, whatever's in its spec not a pre-defined group.
  3. It picks the cheapest EC2 instance type across the entire fleet that satisfies those requirements, and calls the EC2 Fleet API directly to launch it.
  4. The node joins the cluster, the pod schedules, done typically in under a minute.
  5. On the way back down, Karpenter continuously watches for nodes that are empty or underutilized and consolidates workloads to shut them down.

There's no ASG in this loop at all. Karpenter talks to EC2 directly, which is what makes it fast and lets it pick from hundreds of instance types instead of the two or three you'd hand-pick for a node group.

Why This Is Beneficial

  • Right-sized capacity, every time. Nodes are chosen per-pod's actual needs instead of forcing every workload into whatever instance type a node group happened to use.
  • Faster scaling. Direct EC2 Fleet calls skip the ASG polling loop, so new capacity shows up in seconds to a couple of minutes instead of several minutes.
  • Less to manage. One NodePool and EC2NodeClass replace a sprawl of hand-tuned node groups, one per instance-type-and-AZ combination.
  • Safer use of Spot. Built-in interruption handling means Spot stops being "risky" and becomes just cheaper capacity with a warning light.
  • Self-healing infrastructure. Nodes expire and get replaced automatically, so patching and drift correction happen on a schedule instead of a ticket.

How the Cost-Effective Strategies Are Implemented

Karpenter's cost savings aren't a side effect they come from specific, configurable behaviors:

Spot-first, mixed with On-Demand. The NodePool allows both capacity types, and Karpenter defaults to the cheapest option that satisfies the pod's constraints usually Spot, at up to 90% off On-Demand pricing:

requirements:
  - key: karpenter.sh/capacity-type
    operator: In
    values: ["spot", "on-demand"]
Enter fullscreen mode Exit fullscreen mode

Consolidation. Karpenter continuously re-evaluates the cluster. If pods on an underused node could be bin-packed onto existing capacity, it moves them and terminates the now-empty node:

disruption:
  consolidationPolicy: WhenEmptyOrUnderutilized
  consolidateAfter: 1m
Enter fullscreen mode Exit fullscreen mode

Broad instance selection. By allowing whole instance categories (c, m, r) instead of a single type, Karpenter can pick whichever specific instance is cheapest and most available in that family at that moment rather than you guessing one in advance.

Scheduled node expiry. Every node is replaced after a fixed lifetime, which doubles as automatic AMI patching no cost from stale, oversized, or drifted long-lived nodes:

expireAfter: 720h
Enter fullscreen mode Exit fullscreen mode

Graceful Spot interruption handling. Losing a Spot instance mid-job is what makes teams avoid Spot in the first place. Karpenter listens for the 2-minute interruption warning via SQS and EventBridge, drains the node in time, and replaces it so you get the discount without the risk of a workload dying uncleanly.

Setting It Up

1. IAM Roles

Karpenter needs two separate roles because two separate things assume them: the controller pod, and the EC2 instances it creates.

Controller role: trusted by your OIDC provider (IRSA), scoped to the karpenter service account:

resource "aws_iam_role" "karpenter_controller" {
  name = "KarpenterControllerRole-${var.cluster_name}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = aws_iam_openid_connect_provider.eks.arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${local.oidc_issuer}:sub" = "system:serviceaccount:karpenter:karpenter"
          "${local.oidc_issuer}:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}
Enter fullscreen mode Exit fullscreen mode

Node role: trusted by EC2, attached to every node Karpenter launches:

resource "aws_iam_role" "karpenter_node" {
  name = "KarpenterNodeRole-${var.cluster_name}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect    = "Allow",
      Principal = { Service = "ec2.amazonaws.com" },
      Action    = "sts:AssumeRole"
    }]
  })

  tags = var.tags
}

resource "aws_iam_role_policy_attachment" "node_policies" {
  for_each = toset([
    "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
    "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
    "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly",
    "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
  ])
  role       = aws_iam_role.karpenter_node.name
  policy_arn = each.value
}
Enter fullscreen mode Exit fullscreen mode

2. The Controller Policy

The controller role above only has a trust policy so far it says who can assume it, not what it's allowed to do. That comes from a separate inline policy granting ec2:RunInstances, ec2:CreateFleet, ec2:TerminateInstances, instance-profile management, and iam:PassRole on the node role.

Don't hand-write this from scratch Karpenter's maintainers publish and maintain the canonical version as part of the official Getting Started CloudFormation template:

  • Official reference: karpenter.sh/docs/reference/cloudformation
  • Raw policy source: https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml

Pull the KarpenterControllerPolicy statements out of that template into a local JSON file, templatize the account-specific values, and render it with templatefile():

resource "aws_iam_role_policy" "karpenter_controller_policy" {
  name = "KarpenterControllerPolicy-${var.cluster_name}"
  role = aws_iam_role.karpenter_controller.id
  policy = templatefile("${path.module}/policies/karpenter-controller-policy.json", {
    cluster_name            = var.cluster_name
    region                  = var.region
    karpenter_node_role_arn = aws_iam_role.karpenter_node.arn
    eks_cluster_arn         = aws_eks_cluster.eks.arn
    sqs_queue_arn           = aws_sqs_queue.karpenter_interruption.arn
  })
}
Enter fullscreen mode Exit fullscreen mode

Templating it this way means the policy JSON stays identical to the upstream version — only the account, region, cluster, role, and queue ARNs are substituted per environment so pulling in a future upstream policy update is a diff, not a rewrite.

3. Let the Nodes Actually Join the Cluster

If your cluster uses the older aws-auth ConfigMap, you'd map the node role there. On newer clusters with authentication_mode = "API", you skip the ConfigMap entirely and create an access entry instead:

resource "aws_eks_access_entry" "karpenter_node" {
  cluster_name  = var.cluster_name
  principal_arn = aws_iam_role.karpenter_node.arn
  type          = "EC2_LINUX"
}
Enter fullscreen mode Exit fullscreen mode

That's the entire equivalent of the old mapRoles block — one resource, no YAML.

4. Make Subnets and Security Groups Discoverable

Karpenter doesn't take a list of subnet IDs. It finds them by tag. Tag only your private subnets, and Karpenter will never place a node in a public one:

tags = {
  "karpenter.sh/discovery" = var.cluster_name
}
Enter fullscreen mode Exit fullscreen mode

Put the same tag on the node security group. Two tags, and Karpenter's entire networking config is done.

5. SQS + EventBridge for Spot Interruptions

This is the piece that makes Spot safe to use. AWS gives a 2-minute warning before reclaiming a Spot instance — but only if something is listening. EventBridge catches that warning and drops it on a queue; Karpenter polls the queue and drains the node before AWS pulls it.

resource "aws_sqs_queue" "karpenter_interrupt" {
  name                      = "${var.cluster_name}-karpenter-spot-events"
  message_retention_seconds = 300
}

resource "aws_cloudwatch_event_rule" "spot_interruption" {
  name = "${var.cluster_name}-spot-interruption"
  event_pattern = jsonencode({
    source      = ["aws.ec2"]
    detail-type = ["EC2 Spot Instance Interruption Warning"]
  })
}

resource "aws_cloudwatch_event_target" "spot_interruption_to_sqs" {
  rule = aws_cloudwatch_event_rule.spot_interruption.name
  arn  = aws_sqs_queue.karpenter_interrupt.arn
}
Enter fullscreen mode Exit fullscreen mode

Repeat the rule/target pair for Rebalance Recommendation, Instance State-change Notification, and AWS Health Event — four small rules, all feeding the same queue. Karpenter needs sqs:ReceiveMessage, DeleteMessage, and GetQueueAttributes on it, granted through the controller policy above.

6. Install Karpenter with Helm

Install the CRDs and the controller as two separate releases — this keeps future CRD upgrades from breaking against an immutable-field error:

resource "kubernetes_namespace_v1" "karpenter" {
  metadata {
    name = var.karpenter_namespace
  }
}

resource "helm_release" "karpenter_crds" {
  name             = "karpenter-crd"
  namespace        = var.karpenter_namespace
  repository       = "oci://public.ecr.aws/karpenter"
  chart            = "karpenter-crd"
  version          = var.karpenter_version
  create_namespace = false

  depends_on = [
    kubernetes_namespace_v1.karpenter
  ]
}
Enter fullscreen mode Exit fullscreen mode

Then the controller itself, wired up to the queue, the IRSA role, and — critically — pinned to your existing managed node group:

resource "helm_release" "karpenter" {
  name             = "karpenter"
  namespace        = var.karpenter_namespace
  repository       = "oci://public.ecr.aws/karpenter"
  chart            = "karpenter"
  version          = var.karpenter_version
  create_namespace = false
  wait             = true
  cleanup_on_fail  = true
  timeout          = 600
  set = [
    {
      name  = "settings.clusterName"
      value = local.cluster_name
    },
    {
      name  = "settings.interruptionQueue"
      value = local.karpenter_queue_name
    },
    {
      name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
      value = local.karpenter_controller_role_arn
    },
    {
      name  = "controller.resources.requests.cpu"
      value = "1"
    },
    {
      name  = "controller.resources.requests.memory"
      value = "1Gi"
    },
    {
      name  = "controller.resources.limits.cpu"
      value = "1"
    },
    {
      name  = "controller.resources.limits.memory"
      value = "1Gi"
    },
    # Pin Karpenter pods to the existing managed node group
    {
      name  = "affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key"
      value = "eks.amazonaws.com/nodegroup"
    },
    {
      name  = "affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].operator"
      value = "In"
    },
    {
      name  = "affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].values[0]"
      value = local.node_group_name
    }
  ]
  depends_on = [
    kubernetes_namespace_v1.karpenter,
    helm_release.karpenter_crds
  ]
}
Enter fullscreen mode Exit fullscreen mode

Why the node affinity block matters: Karpenter's job is to launch nodes for pods that can't schedule — but Karpenter itself runs as a pod. If Karpenter is allowed to schedule onto nodes it provisions, you get a chicken-and-egg deadlock: no Karpenter-provisioned nodes exist yet, so Karpenter can't start, so no nodes ever get created. The fix is to explicitly schedule Karpenter onto your base, pre-existing managed node group — the one created in the EKS/Terraform bootstrap, not anything Karpenter itself spins up. That node group's only job is to keep a stable home for cluster-critical controllers; everything else in the cluster is fair game for Karpenter to provision, resize, and recycle.

7. Tell It What to Launch: NodePool + EC2NodeClass

EC2NodeClass: the instance blueprint:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: KarpenterNodeRole-my-cluster
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags: { karpenter.sh/discovery: my-cluster }
  securityGroupSelectorTerms:
    - tags: { karpenter.sh/discovery: my-cluster }
Enter fullscreen mode Exit fullscreen mode

NodePool: the rules for when and what:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
Enter fullscreen mode Exit fullscreen mode

Wrapping Up

Two roles plus one templated controller policy, one access entry, two tags, one queue with four EventBridge rules, two Helm releases pinned to your base node group, and two YAML manifests. Nothing here is complicated on its own — it's just a lot of small pieces that all have to point at each other correctly. Once they do, scaling stops being something you configure per-workload and becomes something the cluster just does, at whatever price the market currently offers.

Originally published on Medium.

Top comments (0)