DEV Community: Ajinkya

The Anatomy of Elastic CI: GitHub Actions ARC EKS Karpenter

Ajinkya — Mon, 07 Sep 2026 07:38:47 +0000

Zero idle CI runners. That's the point.

No jobs = zero runners = zero wasted spend.
Traffic shows up. Infrastructure follows.

The flow

Workflow triggers → ARC creates a runner pod → pod lands on a Karpenter NodePool → if capacity is missing, it goes Pending → Karpenter provisions exactly what's needed → job runs → pod disappears → node consolidates.

No standing fleet. No idle EC2 between builds. No capacity planning spreadsheet. Just compute that exists for exactly the minutes a job needs it.

What makes the cost math real

Guaranteed QoS

CPU requests = limits. Memory requests = limits.

This gives Karpenter a clean bin-packing target instead of over-allocating — Karpenter sizes the node off what you ask for, so a sloppy request means a bigger, pricier instance than the job actually needs.

Storage reclaim policy matters

Reclaim policy should be Delete, not Retain.

With Retain, per-job PVCs and their backing EBS volumes can remain after ephemeral runners disappear. The runners scale to zero. The storage bill doesn't.

This is the single easiest detail to overlook when evaluating the true cost of an ephemeral CI architecture — check it before you trust the "zero idle spend" pitch.

Spot for most jobs, On-Demand for critical ones

It doesn't have to be a cluster-wide decision. Use different Karpenter NodePools and target the appropriate runner/workload based on the job.

Spot → cost optimization
On-Demand → workloads that can't tolerate interruption

Same elasticity. Different reliability and cost trade-off — decided per job, not once for the whole cluster.

Cold starts are the trade-off

With minRunners: 0, there are no warm runners waiting for jobs. The first job after an idle period needs to wait for:

Pod scheduling → Karpenter provisioning → EC2 startup → runner initialization.

And in kubernetes mode, a PVC may also need to be provisioned before the job even starts.

You're trading idle compute cost for startup latency. Fine for most CI workloads. Not ideal when you need sub-minute pipeline startup times.

Not a drop-in swap for `ubuntu-latest`

The container mode you choose determines how much your existing workflow needs to change.

dind mode — The runner gets a real Docker daemon sidecar. Most existing workflows can remain largely unchanged. Trade-off: a privileged pod, plus additional startup overhead on every job.

kubernetes mode — Every job/action runs as its own unprivileged Kubernetes pod. More isolation, but also more constraints:

No Docker daemon anywhere
Every job needs an explicit container: configuration
Image builds and pushes require daemonless tooling
Bash-specific steps can break if the container's default shell isn't Bash

Why this architecture works

Each layer has one responsibility:

ARC → Runner lifecycle
Kubernetes → Scheduling
Karpenter → Provisioning + consolidation
Spot / On-Demand → Cost vs. reliability

Stack them together and the infrastructure behaves almost like a function:

Input → Compute appears → Work executes → Compute disappears

How to actually build it

The architecture above isn't theoretical — here's the exact install path, end to end, for a Karpenter-backed, spot-first, kubernetes-mode runner scale set on EKS.

1. Namespaces + controller

One controller per cluster, installed once:

kubectl create namespace arc-systems
kubectl create namespace arc-runners

helm install arc \
  --namespace arc-systems --create-namespace \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

kubectl get pods -n arc-systems
# expect: arc-gha-runner-scale-set-controller-xxxx   1/1   Running

2. A PAT scoped to exactly one repo

Fine-grained token → Repository access: Only select repositories → the one target repo, nothing org-wide:

Permission	Access level	Why
Actions	Read and write	Register/deregister runners, read the job queue
Administration	Read and write	Create/remove self-hosted runners on the repo
Metadata	Read-only	Required baseline

kubectl create secret generic eks-spot-amd-gh-secret \
  --namespace arc-runners \
  --from-literal=github_token=<YOUR_PAT>

3. The values file — where the cost math actually lives

# values-eks-spot-amd.yaml
githubConfigUrl: "https://github.com/<your-username>/<your-repo>"
githubConfigSecret: eks-spot-amd-gh-secret
runnerScaleSetName: "eks-spot-amd"

minRunners: 0     # zero idle runners — this is the whole point
maxRunners: 3     # keep tight for a single repo/workflow

containerMode:
  type: "kubernetes"
  kubernetesModeWorkVolumeClaim:
    accessModes: [ReadWriteOnce]
    storageClassName: gp3
    resources:
      requests:
        storage: 5Gi

template:
  spec:
    securityContext:
      fsGroup: 1001
    nodeSelector:
      node-pool: spot-amd64
      capacity-type: spot
      arch: amd64
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        resources:
          requests: { cpu: "2", memory: "4Gi" }   # requests == limits → Guaranteed QoS
          limits:   { cpu: "2", memory: "4Gi" }

requests == limits is not stylistic — it's the line that actually produces Guaranteed QoS and gives Karpenter a clean bin-packing target instead of an over-provisioned guess.

Install:

helm install eks-spot-amd \
  --namespace arc-runners \
  -f values-eks-spot-amd.yaml \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

kubectl get pods -n arc-runners
# expect: eks-spot-amd-xxxx-listener   1/1   Running   (idle, long-poll)

4. Confirm the reclaim policy before you trust the cost story

This is the one-line check that separates "zero idle spend" from "zero idle spend, plus a slowly growing EBS bill nobody notices":

kubectl get storageclass gp3 -o jsonpath='{.reclaimPolicy}'

If it says Retain instead of Delete, fix it before going further — otherwise every ephemeral job leaves a volume behind.

5. Route jobs to Spot or On-Demand per NodePool

Rather than a cluster-wide choice, tag NodePools separately and target them with nodeSelector per runner scale set — Spot for the bulk of jobs, a second On-Demand scale set for anything interruption-sensitive (release builds, deploy gates). Same controller, same chart, two values files, two nodeSelector blocks.

6. Adjust the workflow for your container mode

If you're on kubernetes mode, every job needs an explicit container:, and anything touching Docker (builds, registry logins, local-scan patterns) needs daemonless tooling instead — no /var/run/docker.sock exists anywhere in this mode by design.

jobs:
  build:
    runs-on: eks-spot-amd
    container:
      image: ghcr.io/catthehacker/ubuntu:act-22.04   # glibc-based — musl breaks JS actions
    defaults:
      run:
        shell: bash   # confirm the image's default shell isn't /bin/sh
    steps:
      - uses: actions/checkout@v4
      - run: make build

If most of your existing workflows assume a live daemon and rewriting them isn't practical yet, start on dind mode instead — same cost model, none of the container-mode constraints, at the cost of a privileged pod.

You also don't have to pick one mode for the whole cluster. Run two runner scale sets side by side — one on kubernetes mode for the bulk of your jobs, one on dind mode scoped to just the jobs that genuinely need a Docker daemon (image builds, registry pushes, container actions). Same controller, same chart, two values-*.yaml files, two runnerScaleSetNames. Point the daemon-dependent jobs at the dind scale set's runs-on: and leave everything else on kubernetes:

jobs:
  test:
    runs-on: eks-spot-amd            # kubernetes mode — no daemon needed
    container:
      image: ghcr.io/catthehacker/ubuntu:act-22.04
    steps:
      - uses: actions/checkout@v4
      - run: make test

  build-and-push:
    runs-on: eks-spot-amd-dind       # dind mode — needs a real daemon
    steps:
      - uses: actions/checkout@v4
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: registry/image:${{ github.sha }}

You get the smaller, unprivileged blast radius everywhere it's free to have it, and you contain the privileged trade-off to exactly the jobs that can't avoid it — instead of either rewriting your entire Docker-based pipeline around daemonless tooling, or running every job privileged just because one of them needs to be.

7. Validate the full elastic loop

kubectl get pods -n arc-runners -w                                           # ephemeral pod appears on trigger
kubectl get pod <pod> -n arc-runners -o jsonpath='{.spec.nodeName}'           # landed on the intended NodePool?
kubectl get node <node> --show-labels | grep -o 'capacity-type=[a-z]*'        # spot or on-demand, as intended?
kubectl get pod <pod> -n arc-runners -o jsonpath='{.status.qosClass}'         # actually Guaranteed?
kubectl get pvc -n arc-runners                                                # gone after the job completes?

Trigger a real workflow and watch the whole function execute: Input → Compute appears → Work executes → Compute disappears.

If the PVC and node are both gone a few minutes after the job finishes, the cost math in this post is real for your cluster, not just in theory.

If you're running ARC on EKS, I'd love to hear what NodePool split or container mode you landed on — drop a comment below.

Enabling Karpenter on EKS with Terraform: What It Is, Why It's Worth It, and How to Set It Up

Ajinkya — Sun, 06 Sep 2026 05:51:35 +0000

Karpenter replaces the old "guess an instance type, set a min/max, wait for Cluster Autoscaler" model with something simpler: watch for pods that can't schedule, and launch exactly the EC2 capacity they need. This guide covers how it actually works, why teams adopt it, and the Terraform you need to wire up to run it on EKS.

Prerequisites

Before you begin, make sure you have:

A running Amazon EKS cluster (created through either Terraform, eksctl, or the console)
Terraform (v1.11+ recommended)
An existing VPC and subnets
Existing security groups
Helm (v3+)
kubectl configured to access your EKS cluster
AWS CLI with credentials configured

How Karpenter Actually Works

Traditional autoscaling works in two disconnected layers: the Kubernetes scheduler decides where pods should go, and a separate Cluster Autoscaler watches Auto Scaling Groups and adds nodes when they're full. The ASG only knows one instance type per group, so it's constantly a step behind what the scheduler actually needs.

Karpenter collapses those two layers into one loop:

A pod goes Pending because nothing can schedule it.
Karpenter looks at that pod's actual requirements CPU, memory, architecture, zone, GPU, whatever's in its spec not a pre-defined group.
It picks the cheapest EC2 instance type across the entire fleet that satisfies those requirements, and calls the EC2 Fleet API directly to launch it.
The node joins the cluster, the pod schedules, done typically in under a minute.
On the way back down, Karpenter continuously watches for nodes that are empty or underutilized and consolidates workloads to shut them down.

There's no ASG in this loop at all. Karpenter talks to EC2 directly, which is what makes it fast and lets it pick from hundreds of instance types instead of the two or three you'd hand-pick for a node group.

Why This Is Beneficial

Right-sized capacity, every time. Nodes are chosen per-pod's actual needs instead of forcing every workload into whatever instance type a node group happened to use.
Faster scaling. Direct EC2 Fleet calls skip the ASG polling loop, so new capacity shows up in seconds to a couple of minutes instead of several minutes.
Less to manage. One NodePool and EC2NodeClass replace a sprawl of hand-tuned node groups, one per instance-type-and-AZ combination.
Safer use of Spot. Built-in interruption handling means Spot stops being "risky" and becomes just cheaper capacity with a warning light.
Self-healing infrastructure. Nodes expire and get replaced automatically, so patching and drift correction happen on a schedule instead of a ticket.

How the Cost-Effective Strategies Are Implemented

Karpenter's cost savings aren't a side effect they come from specific, configurable behaviors:

Spot-first, mixed with On-Demand. The NodePool allows both capacity types, and Karpenter defaults to the cheapest option that satisfies the pod's constraints usually Spot, at up to 90% off On-Demand pricing:

requirements:
  - key: karpenter.sh/capacity-type
    operator: In
    values: ["spot", "on-demand"]

Consolidation. Karpenter continuously re-evaluates the cluster. If pods on an underused node could be bin-packed onto existing capacity, it moves them and terminates the now-empty node:

disruption:
  consolidationPolicy: WhenEmptyOrUnderutilized
  consolidateAfter: 1m

Broad instance selection. By allowing whole instance categories (c, m, r) instead of a single type, Karpenter can pick whichever specific instance is cheapest and most available in that family at that moment rather than you guessing one in advance.

Scheduled node expiry. Every node is replaced after a fixed lifetime, which doubles as automatic AMI patching no cost from stale, oversized, or drifted long-lived nodes:

expireAfter: 720h

Graceful Spot interruption handling. Losing a Spot instance mid-job is what makes teams avoid Spot in the first place. Karpenter listens for the 2-minute interruption warning via SQS and EventBridge, drains the node in time, and replaces it so you get the discount without the risk of a workload dying uncleanly.

Setting It Up

1. IAM Roles

Karpenter needs two separate roles because two separate things assume them: the controller pod, and the EC2 instances it creates.

Controller role: trusted by your OIDC provider (IRSA), scoped to the karpenter service account:

resource "aws_iam_role" "karpenter_controller" {
  name = "KarpenterControllerRole-${var.cluster_name}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = aws_iam_openid_connect_provider.eks.arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${local.oidc_issuer}:sub" = "system:serviceaccount:karpenter:karpenter"
          "${local.oidc_issuer}:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}

Node role: trusted by EC2, attached to every node Karpenter launches:

resource "aws_iam_role" "karpenter_node" {
  name = "KarpenterNodeRole-${var.cluster_name}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect    = "Allow",
      Principal = { Service = "ec2.amazonaws.com" },
      Action    = "sts:AssumeRole"
    }]
  })

  tags = var.tags
}

resource "aws_iam_role_policy_attachment" "node_policies" {
  for_each = toset([
    "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
    "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
    "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly",
    "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
  ])
  role       = aws_iam_role.karpenter_node.name
  policy_arn = each.value
}

2. The Controller Policy

The controller role above only has a trust policy so far it says who can assume it, not what it's allowed to do. That comes from a separate inline policy granting ec2:RunInstances, ec2:CreateFleet, ec2:TerminateInstances, instance-profile management, and iam:PassRole on the node role.

Don't hand-write this from scratch Karpenter's maintainers publish and maintain the canonical version as part of the official Getting Started CloudFormation template:

Official reference: karpenter.sh/docs/reference/cloudformation
Raw policy source: https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml

Pull the KarpenterControllerPolicy statements out of that template into a local JSON file, templatize the account-specific values, and render it with templatefile():

resource "aws_iam_role_policy" "karpenter_controller_policy" {
  name = "KarpenterControllerPolicy-${var.cluster_name}"
  role = aws_iam_role.karpenter_controller.id
  policy = templatefile("${path.module}/policies/karpenter-controller-policy.json", {
    cluster_name            = var.cluster_name
    region                  = var.region
    karpenter_node_role_arn = aws_iam_role.karpenter_node.arn
    eks_cluster_arn         = aws_eks_cluster.eks.arn
    sqs_queue_arn           = aws_sqs_queue.karpenter_interruption.arn
  })
}

Templating it this way means the policy JSON stays identical to the upstream version — only the account, region, cluster, role, and queue ARNs are substituted per environment so pulling in a future upstream policy update is a diff, not a rewrite.

3. Let the Nodes Actually Join the Cluster

If your cluster uses the older aws-auth ConfigMap, you'd map the node role there. On newer clusters with authentication_mode = "API", you skip the ConfigMap entirely and create an access entry instead:

resource "aws_eks_access_entry" "karpenter_node" {
  cluster_name  = var.cluster_name
  principal_arn = aws_iam_role.karpenter_node.arn
  type          = "EC2_LINUX"
}

That's the entire equivalent of the old mapRoles block — one resource, no YAML.

4. Make Subnets and Security Groups Discoverable

Karpenter doesn't take a list of subnet IDs. It finds them by tag. Tag only your private subnets, and Karpenter will never place a node in a public one:

tags = {
  "karpenter.sh/discovery" = var.cluster_name
}

Put the same tag on the node security group. Two tags, and Karpenter's entire networking config is done.

5. SQS + EventBridge for Spot Interruptions

This is the piece that makes Spot safe to use. AWS gives a 2-minute warning before reclaiming a Spot instance — but only if something is listening. EventBridge catches that warning and drops it on a queue; Karpenter polls the queue and drains the node before AWS pulls it.

resource "aws_sqs_queue" "karpenter_interrupt" {
  name                      = "${var.cluster_name}-karpenter-spot-events"
  message_retention_seconds = 300
}

resource "aws_cloudwatch_event_rule" "spot_interruption" {
  name = "${var.cluster_name}-spot-interruption"
  event_pattern = jsonencode({
    source      = ["aws.ec2"]
    detail-type = ["EC2 Spot Instance Interruption Warning"]
  })
}

resource "aws_cloudwatch_event_target" "spot_interruption_to_sqs" {
  rule = aws_cloudwatch_event_rule.spot_interruption.name
  arn  = aws_sqs_queue.karpenter_interrupt.arn
}

Repeat the rule/target pair for Rebalance Recommendation, Instance State-change Notification, and AWS Health Event — four small rules, all feeding the same queue. Karpenter needs sqs:ReceiveMessage, DeleteMessage, and GetQueueAttributes on it, granted through the controller policy above.

6. Install Karpenter with Helm

Install the CRDs and the controller as two separate releases — this keeps future CRD upgrades from breaking against an immutable-field error:

resource "kubernetes_namespace_v1" "karpenter" {
  metadata {
    name = var.karpenter_namespace
  }
}

resource "helm_release" "karpenter_crds" {
  name             = "karpenter-crd"
  namespace        = var.karpenter_namespace
  repository       = "oci://public.ecr.aws/karpenter"
  chart            = "karpenter-crd"
  version          = var.karpenter_version
  create_namespace = false

  depends_on = [
    kubernetes_namespace_v1.karpenter
  ]
}

Then the controller itself, wired up to the queue, the IRSA role, and — critically — pinned to your existing managed node group:

resource "helm_release" "karpenter" {
  name             = "karpenter"
  namespace        = var.karpenter_namespace
  repository       = "oci://public.ecr.aws/karpenter"
  chart            = "karpenter"
  version          = var.karpenter_version
  create_namespace = false
  wait             = true
  cleanup_on_fail  = true
  timeout          = 600
  set = [
    {
      name  = "settings.clusterName"
      value = local.cluster_name
    },
    {
      name  = "settings.interruptionQueue"
      value = local.karpenter_queue_name
    },
    {
      name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
      value = local.karpenter_controller_role_arn
    },
    {
      name  = "controller.resources.requests.cpu"
      value = "1"
    },
    {
      name  = "controller.resources.requests.memory"
      value = "1Gi"
    },
    {
      name  = "controller.resources.limits.cpu"
      value = "1"
    },
    {
      name  = "controller.resources.limits.memory"
      value = "1Gi"
    },
    # Pin Karpenter pods to the existing managed node group
    {
      name  = "affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key"
      value = "eks.amazonaws.com/nodegroup"
    },
    {
      name  = "affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].operator"
      value = "In"
    },
    {
      name  = "affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].values[0]"
      value = local.node_group_name
    }
  ]
  depends_on = [
    kubernetes_namespace_v1.karpenter,
    helm_release.karpenter_crds
  ]
}

Why the node affinity block matters: Karpenter's job is to launch nodes for pods that can't schedule — but Karpenter itself runs as a pod. If Karpenter is allowed to schedule onto nodes it provisions, you get a chicken-and-egg deadlock: no Karpenter-provisioned nodes exist yet, so Karpenter can't start, so no nodes ever get created. The fix is to explicitly schedule Karpenter onto your base, pre-existing managed node group — the one created in the EKS/Terraform bootstrap, not anything Karpenter itself spins up. That node group's only job is to keep a stable home for cluster-critical controllers; everything else in the cluster is fair game for Karpenter to provision, resize, and recycle.

7. Tell It What to Launch: NodePool + EC2NodeClass

EC2NodeClass: the instance blueprint:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: KarpenterNodeRole-my-cluster
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags: { karpenter.sh/discovery: my-cluster }
  securityGroupSelectorTerms:
    - tags: { karpenter.sh/discovery: my-cluster }

NodePool: the rules for when and what:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m

Wrapping Up

Two roles plus one templated controller policy, one access entry, two tags, one queue with four EventBridge rules, two Helm releases pinned to your base node group, and two YAML manifests. Nothing here is complicated on its own — it's just a lot of small pieces that all have to point at each other correctly. Once they do, scaling stops being something you configure per-workload and becomes something the cluster just does, at whatever price the market currently offers.

Originally published on Medium.

Deploying Envoy Gateway on AWS EKS: The Right Way

Ajinkya — Thu, 18 Jun 2026 16:33:49 +0000

Some context first: we were running on GKE Autopilot, where the Gateway API just works out of the box. Google manages the CRDs and the underlying load balancer controller for you create a Gateway, and it gets an external IP without you ever thinking about CRD lifecycle.

Moving that same ingress layer to EKS meant none of that was ready to use anymore. The first real decision wasn't about Envoy Gateway's configuration at all it was about how to install its CRDs without them colliding with the Gateway API CRDs, or with each other, during future migrations.

Installing the Gateway API CRDs

We start by installing the Gateway API CRDs first:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml

What this installs:

GatewayClass CRD
Gateway CRD
HTTPRoute CRD
ReferenceGrant CRD

Verify with:

kubectl get crd | grep gateway.networking.k8s.io

Installing Envoy Gateway's CRDs and Controller via ArgoCD

Next, install Envoy Gateway's own CRDs and the controller itself as two separate ArgoCD Applications, on two separate sync waves:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: envoy-gateway-crds
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  project: default
  source:
    repoURL: oci://docker.io/envoyproxy
    chart: gateway-crds-helm
    targetRevision: v1.8.0
    helm:
      values: |
        crds:
          gatewayAPI:
            enabled: false       # Gateway API CRDs managed separately
            channel: experimental
          envoyGateway:
            enabled: true        # Only Envoy-specific CRDs
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=false
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: envoy-gateway
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  project: default
  source:
    repoURL: oci://docker.io/envoyproxy
    chart: gateway-helm
    targetRevision: v1.8.0
    helm:
      skipCrds: true             # CRDs managed by envoy-gateway-crds app
  destination:
    server: https://kubernetes.default.svc
    namespace: envoy-gateway-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true

If you're not using ArgoCD, the equivalent Helm commands are:

helm install envoy-gateway-crds oci://docker.io/envoyproxy/gateway-crds-helm \
  --version v1.8.0 \
  --namespace default \
  --server-side \
  --set crds.gatewayAPI.enabled=false \
  --set crds.gatewayAPI.channel=experimental \
  --set crds.envoyGateway.enabled=true

helm install envoy-gateway oci://docker.io/envoyproxy/gateway-helm \
  --version v1.8.0 \
  --namespace envoy-gateway-system \
  --create-namespace \
  --skip-crds \
  --server-side

A few things matter here:

gatewayAPI.enabled: false the shared Gateway API CRDs (GatewayClass, Gateway, HTTPRoute, ReferenceGrant) aren't installed by this chart. They're installed once, separately, by their own Application, independent of any controller.
envoyGateway.enabled: true this chart installs only Envoy Gateway's own CRDs, including EnvoyProxy, on sync-wave -1, before the controller exists.
skipCrds: true on the envoy-gateway chart (wave 0) the controller deployment goes in after its CRDs already exist, and never touches CRD lifecycle itself.
ServerSideApply=true on both field-level ownership instead of whole-object ownership, so multiple Applications can touch overlapping CRDs without one overwriting the other.

Both Applications are templated as part of our cluster-bootstrap ApplicationSet, so every environment gets Envoy Gateway's CRDs-then-controller order automatically no manual sequencing per cluster for this part of the stack.

Architecture at a Glance

Internet
   │
   ▼
AWS NLB  (provisioned by AWS Load Balancer Controller)
   │
   ▼
Envoy Proxy Pods  (managed by Envoy Gateway, autoscaled by HPA)
   │
   ▼
Application Services  (via HTTPRoute rules)

The LoadBalancer Pending Trap

With the CRDs and controller in place, the next thing that breaks on a fresh EKS setup is the Service Envoy Gateway generates for its proxy deployment. By default, it's type LoadBalancer, and Kubernetes' in-tree cloud controller tries to provision a Classic Load Balancer for it.

On modern EKS clusters, that fails silently. No CLB gets created, no useful error appears in events, and the Service just sits at <pending> indefinitely.

The fix doesn't live on the Gateway object — Envoy Gateway generates its own Service internally, so there's nothing on Gateway.metadata to annotate. The fix has to go into the EnvoyProxy CRD, the same CRD installed separately in the -1 sync wave above, via envoyService.annotations:

envoyService:
  annotations:
    # Stops the in-tree CLB provisioner - AWS Load Balancer Controller
    # takes over and creates an NLB instead.
    service.beta.kubernetes.io/aws-load-balancer-type: "external"

    # Public-facing NLB. Use "internal" for private traffic only.
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"

    # Route NLB traffic directly to Pod IPs via VPC CNI -
    # bypasses kube-proxy and NodePort.
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

    # NLB health check hits Envoy's admin port. /healthz returns 200
    # only once Envoy is fully ready, so the NLB never routes to a
    # pod that's still starting or draining.
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: HTTP
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "19002"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/healthz"

The single annotation that actually breaks the deadlock is aws-load-balancer-type: "external" it tells the in-tree controller to back off, and hands the Service to the AWS Load Balancer Controller, which then provisions a real NLB and writes its hostname back to gateway.status.addresses. The rest of the block (scheme, target type, health check) is what makes that NLB actually production-ready rather than just "not pending."

Putting It Together: GatewayClass, Gateway, and EnvoyProxy

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: external-gateway
  namespace: gateway
spec:
  gatewayClassName: eg
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: external-proxy-config
  listeners:
    - name: http
      protocol: HTTP
      port: 80

And the EnvoyProxy CRD that ties resources, autoscaling, and the LB fix together in one place:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: external-proxy-config
  namespace: gateway
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        patch:
          type: StrategicMerge
          value:
            spec:
              template:
                spec:
                  containers:
                    - name: shutdown-manager
                      lifecycle:
                        preStop:
                          exec:
                            command: ["/bin/sh", "-c", "sleep 120"]
        container:
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              memory: 1Gi

      envoyHpa:
        minReplicas: 1
        maxReplicas: 5
        metrics:
          - type: Resource
            resource:
              name: cpu
              target:
                type: Utilization
                averageUtilization: 60

      envoyService:
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-type: "external"
          service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: HTTP
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "19002"
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/healthz"

Applying the Gateway triggers the chain reaction: Envoy Gateway creates the proxy deployment, the LoadBalancer Service with the annotations above, and an HPA — then AWS LBC provisions the NLB. HTTPRoute objects attach to the Gateway afterward and define per-service routing, owned by app teams.

Autoscaling Explained

The envoyHpa block creates a standard Kubernetes HPA against the proxy deployment. minReplicas: 1 keeps cost down during idle periods, at the cost of zero redundancy for ~15-30s if that pod dies. averageUtilization: 60 (150m of the 250m request) triggers scale-out early enough that new pods are healthy before latency degrades. For zero-downtime guarantees, minReplicas: 2 or 3 is the move.

Final Thoughts

Running Envoy Gateway on Amazon EKS isn't just about deploying another ingress controller — it's about understanding where the responsibilities are split.

Unlike managed Kubernetes offerings where the Gateway API experience is largely invisible, EKS gives you the flexibility to control every layer. That also means you own the lifecycle of the Gateway API CRDs, the Envoy Gateway CRDs, the controller installation, and the integration with the AWS Load Balancer Controller.

Separating CRDs from the controller, using ArgoCD sync waves to guarantee deployment order, and configuring the EnvoyProxy resource as the single place for infrastructure concerns makes the setup predictable and GitOps-friendly. It also avoids one of the most common migration issues: LoadBalancer Services remaining in a perpetual Pending state because the wrong controller is trying to provision them.

Why We Moved from GKE to EKS

Ajinkya — Sat, 25 Apr 2026 20:47:53 +0000

Why We Moved from GKE to EKS

When we initially adopted Kubernetes, Google Kubernetes Engine (GKE) Autopilot seemed like the perfect choice — fully managed, minimal operational overhead, and quick to get started.

But as our workloads matured, three major challenges started to surface:

Rising and unpredictable costs
Compliance constraints
The need for deeper infrastructure control

This blog walks through why we migrated to Amazon Elastic Kubernetes Service (EKS) with Karpenter, the architectural changes we made, and the lessons we learned running production workloads post-migration.

⚠️ Why GKE Autopilot Started Falling Short

1. Cost Inefficiencies at Scale

GKE Autopilot pricing is based on requested resources, not actual usage. This sounds fine at small scale — but as traffic grows, the gaps between requested and actual usage start to compound.

Problems we observed:

Over-provisioned workloads leading to higher bills
No access to Spot/Preemptible node strategies with the same level of flexibility
Very few cost optimization knobs to tune

As traffic grew, costs increased almost linearly with no meaningful way to optimize without restructuring our entire workload configuration.

2. Compliance and Governance Constraints

Operating in a regulated environment required:

Fine-grained IAM control at the workload level
Strict network isolation between services
Audit-level visibility into infrastructure activity

With GKE Autopilot, several configurations are abstracted away or restricted by design. This made it harder to enforce organization-wide security policies and satisfy compliance requirements from auditors. Specifically:

Enforcing per-pod IAM permissions cleanly was non-trivial
Network policy enforcement had gaps in our specific setup
Generating audit-ready logs tied to individual workload actions required workarounds

We needed something that gave us first-class integration with cloud-native IAM and security tooling — without layering on custom solutions.

3. Limited Infrastructure Control

When performance-sensitive services started hitting bottlenecks, the inability to choose instance types became a real blocker. We had no control over:

CPU vs. memory-optimized instance selection
ARM-based workloads on Graviton processors
Custom AMIs or low-level networking tuning

For teams running general-purpose workloads, this abstraction is a feature. For us, it was a ceiling.

🎯 Why We Chose EKS + Karpenter

Full Infrastructure Control

Moving to EKS gave us direct control over:

Instance families — CPU-optimized, memory-optimized, ARM (Graviton)
Custom AMIs — hardened images meeting our internal security baseline
Networking — VPC-native networking with fine-grained subnet and security group control

This unlocked workload-specific performance tuning that simply wasn't possible before.

Advanced Cost Optimization with Karpenter

Karpenter is not your traditional cluster autoscaler. Instead of scaling pre-defined node groups, it:

Watches for unschedulable pods in real time
Selects the right-sized instance based on actual pod requirements
Prioritizes Spot instances where workloads allow, falling back to On-Demand seamlessly
Bin-packs nodes efficiently, reducing idle capacity

The result: faster scaling reactions and a dramatically lower compute bill — without sacrificing reliability.

Compliance Alignment

AWS gave us the compliance story we needed:

IRSA (IAM Roles for Service Accounts) — precise, per-pod IAM permissions with no shared credentials
VPC-level isolation — full control over ingress, egress, and inter-service communication
CloudTrail integration — every API call, every node action, fully auditable out of the box
AWS Config + Security Hub — continuous compliance checks against CIS benchmarks and custom rules

This made our next compliance audit significantly smoother. Auditors got clear, traceable logs without us having to build custom instrumentation.

🏗️ Target Architecture

Here's what the high-level migration looked like architecturally:

Component	Before (GKE)	After (EKS)
Cluster	GKE Autopilot	EKS (Managed Node Groups + Karpenter)
Autoscaling	Built-in Autopilot scaling	Karpenter
Spot Strategy	NO Control	Karpenter Spot-first provisioning
IAM	GCP Workload Identity	AWS IRSA
Audit Logging	Cloud Audit Logs	CloudTrail + CloudWatch
Networking	GKE VPC-native	AWS VPC with custom subnets

⚙️ Karpenter Setup — The Game Changer

Karpenter replaced our traditional Cluster Autoscaler, and the difference was immediately visible.

How we configured it:

 apiVersion: karpenter.sh/v1
    kind: NodePool
    metadata:
      name: spot-arm64
    spec:
      template:
        metadata:
          labels:
            # -----------------------------------------------
            # These labels land on the EC2 node.
            # Your pod affinity rules match against these.
            # -----------------------------------------------
            node-pool: spot-arm64
            capacity-type: spot
            arch: arm64
            workload-class: standard
        spec:
          requirements:
            - key: kubernetes.io/arch
              operator: In
              values: ["arm64"]
            - key: kubernetes.io/os
              operator: In
              values: ["linux"]
            - key: karpenter.sh/capacity-type
              operator: In
              values: ["spot"]
            - key: karpenter.k8s.aws/instance-category
              operator: In
              values: ["c", "m", "r"]
              # c6g, c7g, c8g — compute optimized Graviton
              # m6g, m7g, m8g — general purpose Graviton
              # r6g, r7g, r8g — memory optimized Graviton
            - key: karpenter.k8s.aws/instance-generation
              operator: Gt
              values: ["5"]             # Graviton2+ only (gen 6,7,8)
          nodeClassRef:
            group: karpenter.k8s.aws
            kind: EC2NodeClass
            name: default
          expireAfter: 168h             # 7 days — shorter for spot nodes


      limits:
        cpu: 500
        memory: 2000Gi
      disruption:
        consolidationPolicy: WhenEmptyOrUnderutilized
        consolidateAfter: 2m
      weight: 100

Key decisions we made:

Spot-first provisioning — workloads that tolerate interruptions run on Spot; stateful services stay on On-Demand
Multiple instance families — Karpenter picks the cheapest right-sized option across families
Interruption handling — we use the Karpenter interruption queue (SQS) to gracefully drain Spot nodes before AWS reclaims them
consolidateAfter: 2m — nodes deprovision 2m seconds after going idle, eliminating ghost capacity

📊 Results After Migration

💰 Cost

Compute costs dropped significantly. The main drivers:

Spot instances covering the majority of our non-critical workloads
Karpenter's bin-packing eliminating idle node waste
Right-sized instances instead of over-provisioned static node groups

⚡ Performance

Faster pod scheduling — Karpenter provisions new nodes in under 60 seconds in most cases
Better workload isolation through custom node selectors and taints
Graviton (ARM) instances for compatible workloads gave us a meaningful price-performance improvement

🔐 Compliance

Audit reports now generated directly from CloudTrail without custom tooling
IRSA eliminated shared IAM credential risks
Security Hub provides continuous posture monitoring against our compliance framework

⚠️ Challenges We Faced

Being honest here — this is what makes a migration story actually useful.

1. Karpenter Learning Curve

Karpenter's provisioning model is fundamentally different from Cluster Autoscaler. Debugging why a node wasn't provisioned — or why Karpenter chose a specific instance type — required understanding its internal decision logic. The logs are verbose but not always immediately readable.

What helped: Running Karpenter in dry-run mode first, and adding structured logging to correlate provisioning decisions with pod events.

2. Networking Model Differences

GKE's VPC-native networking and AWS VPC behave differently in non-obvious ways — especially around CIDR planning, secondary IP ranges, and how pod IPs are allocated. We had to redesign our subnet layout and revisit some service-to-service communication assumptions.

3. IAM Complexity

IRSA is powerful but requires careful role design. Mapping GCP Workload Identity bindings to AWS IRSA role assumptions took time, especially for services that assumed broad IAM permissions under GCP that needed to be tightened properly.

🧠 Key Lessons Learned

Managed ≠ always optimal at scale. Autopilot is excellent for getting started, but production-grade platforms eventually need control surfaces that fully managed offerings deliberately hide.
Cost optimization requires infrastructure access. You can't tune what you can't see.
Autoscaling strategy matters more than cluster size. Karpenter's approach of provisioning for the pod rather than scaling a group changed how we think about capacity planning entirely.
Compliance is easier when the platform is designed for it. AWS's native compliance tooling removed a category of work that we were previously solving with custom scripts and log forwarding pipelines.
Migration should always be incremental. Parallel environment, gradual DNS cutover, canary deployments — this approach meant we caught issues in staging before they became production incidents.

🏁 Conclusion

GKE Autopilot is an excellent choice for teams that want Kubernetes without the operational overhead — and we'd still recommend it for that use case.

But for production environments that require cost control at scale, fine-grained compliance posture, and workload-specific infrastructure decisions, EKS with Karpenter provided a more flexible and efficient platform.

The migration wasn't trivial, but the control, visibility, and cost profile on the other side made it worth it.

Have you gone through a similar migration? Or are you evaluating EKS vs GKE for your stack? Drop your questions in the comments — happy to dig into specifics.

Tags: kubernetes aws devops cloud karpenter

DEV Community: Ajinkya

The Anatomy of Elastic CI: GitHub Actions ARC EKS Karpenter

The flow

What makes the cost math real

Guaranteed QoS

Storage reclaim policy matters

Spot for most jobs, On-Demand for critical ones

Cold starts are the trade-off

Not a drop-in swap for ubuntu-latest

Why this architecture works

How to actually build it

1. Namespaces + controller

2. A PAT scoped to exactly one repo

3. The values file — where the cost math actually lives

4. Confirm the reclaim policy before you trust the cost story

5. Route jobs to Spot or On-Demand per NodePool

6. Adjust the workflow for your container mode

7. Validate the full elastic loop

Enabling Karpenter on EKS with Terraform: What It Is, Why It's Worth It, and How to Set It Up

Prerequisites

How Karpenter Actually Works

Why This Is Beneficial

How the Cost-Effective Strategies Are Implemented

Setting It Up

1. IAM Roles

2. The Controller Policy

3. Let the Nodes Actually Join the Cluster

4. Make Subnets and Security Groups Discoverable

5. SQS + EventBridge for Spot Interruptions

6. Install Karpenter with Helm

7. Tell It What to Launch: NodePool + EC2NodeClass

Wrapping Up

Deploying Envoy Gateway on AWS EKS: The Right Way

Installing the Gateway API CRDs

Installing Envoy Gateway's CRDs and Controller via ArgoCD

Architecture at a Glance

The LoadBalancer Pending Trap

Putting It Together: GatewayClass, Gateway, and EnvoyProxy

Autoscaling Explained

Final Thoughts

Why We Moved from GKE to EKS

Why We Moved from GKE to EKS

⚠️ Why GKE Autopilot Started Falling Short

1. Cost Inefficiencies at Scale

2. Compliance and Governance Constraints

3. Limited Infrastructure Control

🎯 Why We Chose EKS + Karpenter

Full Infrastructure Control

Advanced Cost Optimization with Karpenter

Compliance Alignment

🏗️ Target Architecture

⚙️ Karpenter Setup — The Game Changer

📊 Results After Migration

💰 Cost

⚡ Performance

🔐 Compliance

⚠️ Challenges We Faced

1. Karpenter Learning Curve

2. Networking Model Differences

3. IAM Complexity

🧠 Key Lessons Learned

🏁 Conclusion

Not a drop-in swap for `ubuntu-latest`