DEV Community: Ajinkya

Stop Storing Access Keys in GitHub Secrets: Use OIDC for AWS

Ajinkya — Fri, 02 Oct 2026 08:09:31 +0000

If your GitHub Actions pipeline deploys to AWS, there's a good chance you've seen (or written) this:

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

It works. It's also a quiet liability sitting in your repo settings.

There's a better way: no static keys, no rotation headaches, no credentials sitting in GitHub Secrets. Let's look at why the old way hurts, and how OIDC fixes it.

Index

The problem with static AWS credentials
What is AWS OIDC?
Heads up: GitHub just changed the token format
- Why GitHub changed it
Build it with Terraform
- 1. Get your IDs
- 2. Variables
- 3. Provider and subject
- 4. Trust policy, role, and permissions
The workflow
Debugging: see what GitHub actually sends
Why this is secure
Wrapping up

The problem with static AWS credentials

A traditional workflow looks like this:

GitHub Actions
      |
      | AWS_ACCESS_KEY_ID
      | AWS_SECRET_ACCESS_KEY
      v
     AWS

Those two values are long-lived credentials. They don't expire when the job ends. They don't care which repo, branch, or machine uses them. Whoever holds them is your pipeline, today, next month, and next year.

Even when they're stored "securely" as GitHub Secrets, you still have to:

Rotate them periodically (and remember to)
Manage their lifecycle across every repo that uses them
Make sure they never leak, whether in logs, forks, a pasted screenshot, or a compromised dependency
Revoke them when a project or teammate is gone
Keep the IAM user's permissions tight, forever

Think about what a leak looks like: a key copied into a chat, printed by a debug step, or lifted by a malicious action. It works from any laptop on the planet until someone notices. Many keys never get rotated at all.

Here's the thing, though: a CI/CD pipeline doesn't need permanent credentials. It needs credentials that exist only while the pipeline is running.

That's exactly what OIDC gives you.

What is AWS OIDC?

OpenID Connect (OIDC) lets GitHub Actions prove its identity to AWS and receive temporary credentials, with no stored AWS keys.

                    GitHub
                      |
                GitHub Actions
                      |
                      | OIDC Token
                      v
                AWS IAM / STS
                      |
                      | AssumeRoleWithWebIdentity
                      v
                 IAM Role
                      |
                      | Temporary Credentials
                      v
                 AWS Resources

GitHub never hands you an access key to manage. Instead, for every workflow run, it generates a signed OIDC token that says who is asking: which repo, which branch, which run. AWS verifies the signature, checks your role's trust policy, and returns credentials that expire on their own (one hour by default).

Static access keys	OIDC
Stored in GitHub Secrets	Nothing stored
Valid until rotated (often never)	Expire in about an hour
A leak works from anywhere	Token only works for your repo and branch
Manual rotation	Nothing to rotate

The API behind it is AssumeRoleWithWebIdentity. The name is historical (it was built for web and mobile logins), but it's the same call used for GitHub, GitLab, and EKS. The caller needs no AWS credentials at all, because the signed token is the proof.

Heads up: GitHub just changed the token format

This is the part that's breaking older tutorials. The sub (subject) claim in the token used to look like this:

repo:octo-org/octo-repo:ref:refs/heads/main

For repositories created after July 15, 2026, it now includes immutable numeric IDs:

repo:octo-org@123456/octo-repo@456789:ref:refs/heads/main

123456 is the owner ID and 456789 is the repo ID. The @ separator is used because it can't appear in a GitHub username or repo name.

New repos get the new format automatically.
Renamed or transferred repos after that date also move to it.
Older repos keep the old format unless you opt in (Settings → Actions → OIDC).
GitHub Enterprise Server isn't part of the rollout.

If your trust policy expects the old format and your repo sends the new one, AWS answers with Not authorized to perform sts:AssumeRoleWithWebIdentity. Most guides online still show the old format, so this catches a lot of people.

Why GitHub changed it

The old sub was built from names, and names can be reused:

You own alice/deploy-tools, and your AWS role trusts that exact name.
You delete or rename the repo.
An attacker registers the same name and creates the same repo.
Their workflows produce the same sub as yours, and your role accepts them.

Numeric IDs are assigned once and never reused. A recreated repo gets different IDs, so its sub no longer matches your trust policy.

Build it with Terraform

1. Get your IDs

gh api repos/YOUR_USERNAME/YOUR_REPO --jq '{repo_id: .id, owner_id: .owner.id}'

Or open https://api.github.com/repos/YOUR_USERNAME/YOUR_REPO in a browser (public repos) and read id and owner.id.

2. Variables

variable "github_username" { type = string }
variable "github_repo"     { type = string }

variable "github_branch" {
  type    = string
  default = "main"
}

variable "github_owner_id" {
  type    = string
  default = ""
}

variable "github_repo_id" {
  type    = string
  default = ""
}

variable "create_oidc_provider" {
  type    = bool
  default = true
}

3. Provider and subject

resource "aws_iam_openid_connect_provider" "github" {
  count          = var.create_oidc_provider ? 1 : 0
  url            = "https://token.actions.githubusercontent.com"
  client_id_list = ["sts.amazonaws.com"]
}

data "aws_iam_openid_connect_provider" "github" {
  count = var.create_oidc_provider ? 0 : 1
  url   = "https://token.actions.githubusercontent.com"
}

locals {
  oidc_provider_arn = var.create_oidc_provider ? aws_iam_openid_connect_provider.github[0].arn : data.aws_iam_openid_connect_provider.github[0].arn

  use_ids = var.github_owner_id != "" && var.github_repo_id != ""

  github_sub = local.use_ids ? (
    "repo:${var.github_username}@${var.github_owner_id}/${var.github_repo}@${var.github_repo_id}:ref:refs/heads/${var.github_branch}"
  ) : (
    "repo:${var.github_username}/${var.github_repo}:ref:refs/heads/${var.github_branch}"
  )
}

AWS allows only one GitHub OIDC provider per account, so set create_oidc_provider = false if yours already exists. The thumbprint can be omitted on AWS provider 5.81 or newer. The conditional builds the new format when you pass both IDs and falls back to the old one otherwise.

4. Trust policy, role, and permissions

data "aws_iam_policy_document" "trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [local.oidc_provider_arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values   = [local.github_sub]
    }
  }
}

resource "aws_iam_role" "github_actions" {
  name               = "github-actions-deploy"
  assume_role_policy = data.aws_iam_policy_document.trust.json
}

resource "aws_iam_role_policy_attachment" "s3" {
  role       = aws_iam_role.github_actions.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}

output "github_actions_role_arn" {
  value = aws_iam_role.github_actions.arn
}

output "allowed_subject" {
  value = local.github_sub
}

The trust policy answers who can become the role. The policy attachment answers what the role can do. They're separate on purpose.

AmazonS3FullAccess is fine for a demo, but it allows every S3 action on every bucket. For real use, swap it for a policy scoped to the your need and the resource you need.

terraform init
terraform apply
terraform output

The workflow

Save the role ARN as a repository secret or variable named AWS_ROLE_ARN. Notice what's missing: no access key, no secret key.

name: OIDC Test

on:
  workflow_dispatch: {}

permissions:
  id-token: write      # without this, no token, no login
  contents: read

jobs:
  OIDC-Test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: ap-south-1

      - run: aws sts get-caller-identity

The classic mistake is forgetting id-token: write.

Debugging: see what GitHub actually sends

If you hit Not authorized to perform sts:AssumeRoleWithWebIdentity, add this step before the AWS step:

      - name: Print OIDC claims
        run: |
          TOKEN=$(curl -sS -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
            "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com" | jq -r .value)
          payload=$(echo "$TOKEN" | cut -d. -f2 | tr '_-' '/+')
          while [ $(( ${#payload} % 4 )) -ne 0 ]; do payload="${payload}="; done
          echo "$payload" | base64 -d | jq '{iss, aud, sub, repository}'

Compare the printed sub with terraform output allowed_subject, character for character. That's how I found my own bug: my token showed repo:USER@191673596/REPO@1400242638:... while my trust policy expected the old format. Adding the IDs fixed it instantly.

Other quick checks:

Case: usernames and repo names are case-sensitive.
Branch: a manual run uses whichever branch you picked in the dropdown.
Credentials could not be loaded: role-to-assume was empty. Use secrets. for secrets and vars. for variables.
Assuming role with OIDC repeated: the action is retrying a rejected call. The real error is on the line after.

Why this is secure

Signed token: AWS verifies the signature, so it can't be forged.
Audience check: tokens meant for other services are rejected.
Exact sub match: only your repo and branch get in.
Immutable IDs: a recycled name can't impersonate you.
Short-lived credentials: they expire on their own.
Nothing stored: nothing to steal from settings, logs, or forks.

A few habits make it stronger. Never use wildcards like repo:OWNER/* in the sub. Use one role per purpose. Protect production with a GitHub Environment and required reviewers. And remember that every new repo needs its own IDs in its trust policy.

Wrapping up

Static keys in GitHub Secrets were always a compromise. With OIDC your pipeline gets credentials only when it runs, only for the repo and branch you chose, and they vanish on their own. Just watch the new subject format, because older tutorials won't match what your new repos send.

If you're still pasting access keys into secrets, try this on one repo this week. You'll wonder why you waited.

Sources: GitHub Docs (OIDC reference, "Immutable subject claims") and the GitHub Changelog announcement on immutable subject claims.

Upgrade EKS Nodes Without Downtime: The Bridge Add-on Version Trick

Ajinkya — Mon, 28 Sep 2026 11:01:27 +0000

Move your add-ons to versions that work on both Kubernetes releases, and the node upgrade stops being the scary part.

The problem with "nodes first, add-ons after"

The AWS upgrade guide gives the order as: control plane → nodes → add-ons.

That works on paper, but it may leaves a gap. While your nodes roll from 1.35 to 1.36, you are running new nodes with old add-on versions, and those versions were only validated against the old Kubernetes version. The add-ons that hurt most are the ones every pod depends on:

vpc-cni hands out pod IPs on every node
CoreDNS resolves names for every workload
aws-ebs-csi-driver attaches volumes for stateful pods

If one of them misbehaves on a freshly rolled node, you find out in the middle of the rollout, with pods rescheduling all around you.

The idea: bridge versions

A bridge version is the newest add-on version that is supported on both your current and your target Kubernetes version.

If every add-on is on its bridge version before the nodes roll:

Old nodes (1.35) run an add-on version that supports 1.35.
New nodes (1.36) run the same add-on version, which also supports 1.36.
No node ever runs an add-on that wasn't validated for its Kubernetes version.

Because a bridge version is valid on both releases, you can apply it at any point before the node upgrade: at the very start, on 1.35, or right after the control plane moves to 1.36. Both work.

The order I use:

EKS control plane → bridge add-ons → nodes → (optional) default add-on versions.

This doesn't contradict the AWS order. It adds one safe step in front of the node roll.

Step 0: Find the bridge version for each add-on

EKS can list every version of an add-on for a given Kubernetes version. The bridge is the newest version that appears in both lists.

#!/usr/bin/env bash

OLD_K8S="1.35"
NEW_K8S="1.36"

get_versions() {
  aws eks describe-addon-versions \
    --addon-name "$1" \
    --kubernetes-version "$2" \
    --query 'addons[0].addonVersions[].addonVersion' \
    --output text 2>/dev/null | tr '\t' '\n' | tr -d '\r' | sort -V
}

for addon in vpc-cni coredns kube-proxy aws-ebs-csi-driver metrics-server; do
  old_versions=$(get_versions "$addon" "$OLD_K8S")
  new_versions=$(get_versions "$addon" "$NEW_K8S")

  echo "===== $addon ====="

  if [ -z "$old_versions" ] || [ -z "$new_versions" ]; then
    echo "WARNING: no versions returned - check addon name / region"
    echo
    continue
  fi

  old_latest=$(echo "$old_versions" | tail -1)
  new_latest=$(echo "$new_versions" | tail -1)

  common_version=$(grep -Fxf <(echo "$old_versions") <(echo "$new_versions") | sort -V | tail -1)

  echo "$OLD_K8S latest : $old_latest"
  echo "$NEW_K8S latest : $new_latest"

  if [ -z "$common_version" ]; then
    echo "No overlapping version found - you may need an intermediate hop"
  else
    echo "Bridge version (newest common to both): $common_version"
  fi
  echo
done

Output from my cluster (1.35 → 1.36)

===== vpc-cni =====
1.35 latest : v1.23.1-eksbuild.1
1.36 latest : v1.23.1-eksbuild.1
Bridge version (newest common to both): v1.23.1-eksbuild.1

===== coredns =====
1.35 latest : v1.14.6-eksbuild.4
1.36 latest : v1.14.6-eksbuild.4
Bridge version (newest common to both): v1.14.6-eksbuild.4

===== kube-proxy =====
1.35 latest : v1.35.3-eksbuild.29
1.36 latest : v1.36.0-eksbuild.25
Bridge version (newest common to both): v1.35.3-eksbuild.29

===== aws-ebs-csi-driver =====
1.35 latest : v1.66.0-eksbuild.1
1.36 latest : v1.66.0-eksbuild.1
Bridge version (newest common to both): v1.66.0-eksbuild.1

===== metrics-server =====
1.35 latest : v0.9.0-eksbuild.11
1.36 latest : v0.9.0-eksbuild.11
Bridge version (newest common to both): v0.9.0-eksbuild.11

How to read it:

For vpc-cni, coredns, aws-ebs-csi-driver and metrics-server, the latest version is identical on 1.35 and 1.36, so the bridge is simply the latest build.
kube-proxy is the exception. Its version tracks the Kubernetes minor, so the 1.36 latest (v1.36.0-eksbuild.25) is not available on 1.35, and the bridge is a 1.35.x build (v1.35.3-eksbuild.29). Keep it there until the nodes are on 1.36.

How the Terraform is wired

The EKS cluster (control plane)

The control plane version comes from a single variable:

resource "aws_eks_cluster" "eks" {
  name     = var.cluster_name
  role_arn = aws_iam_role.eks_cluster.arn
  version  = var.cluster_version

  vpc_config {
    # Pass both public and private subnets so the EKS control plane
    # ENIs are spread across AZs inside the VPC.
    subnet_ids              = concat(aws_subnet.public[*].id, aws_subnet.private[*].id)
    security_group_ids      = [aws_security_group.eks_cluster.id]
    endpoint_public_access  = true  # keeps kubectl working from your machine
    endpoint_private_access = true  # nodes in private subnets reach the API internally
  }

  access_config {
    authentication_mode                         = "API"
    bootstrap_cluster_creator_admin_permissions = true
  }

  tags = merge(var.tags, {
    Name = var.cluster_name
  })

  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy
  ]
}

What matters for the upgrade:

version = var.cluster_version is the only line you change. Going from "1.35" to "1.36" makes Terraform update the control plane in place. EKS upgrades one minor version at a time and there is no rollback, so this is the step to plan carefully.
vpc_config and access_config stay the same during an upgrade. The public and private endpoints keep kubectl and the nodes talking to the API, and authentication_mode = "API" keeps your access entries working.
The nodes and add-ons don't change when this resource changes. That is why the control plane can be upgraded on its own with -target, as Step 1 shows.

The node group

The managed node group reads the cluster version and rolls when it changes:

resource "aws_eks_node_group" "ondemand-node" {
  cluster_name  = aws_eks_cluster.eks.name
  node_role_arn = aws_iam_role.node.arn
  subnet_ids    = aws_subnet.private[*].id
  capacity_type = "ON_DEMAND"
  ami_type      = var.ami_type

  version              = var.cluster_version  # ties node AMI to control plane version
  force_update_version = true                 # forces rolling update when version changes

  update_config {
    max_unavailable = 1 # only 1 of nodes can be unavailable at once
  }
  ...
}

What each line does:

version = var.cluster_version sets the Kubernetes version of the nodes. With a managed node group and an ami_type, EKS picks the matching EKS-optimized AMI for that version. Change the variable and the node group rolls onto the new AMI.
force_update_version = true lets the update go through even when pods can't be drained because of a PodDisruptionBudget. Without it, a blocking PDB makes the update fail. With it, EKS eventually proceeds anyway, so pods can be evicted despite the PDB. That keeps upgrades from getting stuck, but it is also the one setting that can cost you availability, so make sure critical workloads have enough replicas.
max_unavailable = 1 rolls one node at a time, so the rest of the group keeps serving traffic.

The add-ons

Add-on versions come from a list variable, so bumping them is a data change:

resource "aws_eks_addon" "eks-addons" {
  for_each      = { for idx, addon in var.addons : idx => addon }
  cluster_name  = aws_eks_cluster.eks.name
  addon_name    = each.value.name
  addon_version = each.value.version

  # attach role ONLY for EBS CSI
  service_account_role_arn = each.value.name == "aws-ebs-csi-driver" ? aws_iam_role.ebs_csi.arn : null

  depends_on = [
    aws_eks_node_group.ondemand-node,
    aws_iam_role_policy_attachment.ebs_csi_policy
  ]
}

Gotcha: `-target` pulls in dependencies

-target applies the resource you name plus everything it depends on. The add-on resource above has depends_on = [aws_eks_node_group.ondemand-node]. If the node group shares var.cluster_version with the control plane, then after you bump that variable, terraform apply -target=aws_eks_addon.eks-addons drags the node group in and rolls your nodes before the bridge add-ons land. That defeats the whole point.

The fix is a separate variable for the node group, so you can hold nodes on 1.35 while you upgrade the control plane and add-ons, then move them last (nodes are allowed to run behind the control plane):

  version              = var.node_group_version  # was var.cluster_version
  force_update_version = true

The rest of this post assumes that split.

The upgrade, step by step with `-target`

Before each step, preview it with terraform plan using the same -target flags and check that only the resources you expect will change.

Step 1: Upgrade the EKS control plane

Set cluster_version = "1.36" and leave node_group_version = "1.35".

terraform plan  -target=aws_eks_cluster.eks
terraform apply -target=aws_eks_cluster.eks

EKS only allows one minor version at a time and there is no rollback, so run your pre-checks first: deprecated APIs, PodDisruptionBudgets, and a supported Karpenter version for 1.36 if you use it.

Step 2: Apply the bridge add-on versions

Set var.addons to the bridge versions from Step 0:

addons = [
  { name = "vpc-cni",            version = "v1.23.1-eksbuild.1" },
  { name = "coredns",            version = "v1.14.6-eksbuild.4" },
  { name = "kube-proxy",         version = "v1.35.3-eksbuild.29" },
  { name = "aws-ebs-csi-driver", version = "v1.66.0-eksbuild.1" },
  { name = "metrics-server",     version = "v0.9.0-eksbuild.11" },
]

terraform plan  -target=aws_eks_addon.eks-addons
terraform apply -target=aws_eks_addon.eks-addons

You can also target a single add-on. Because of for_each, the address includes the key:

terraform apply -target='aws_eks_addon.eks-addons["1"]'   # just coredns, for example

Check health before moving on:

kubectl get pods -n kube-system

You don't have to wait for Step 1. Since bridge versions work on both releases, you can run this step first, on the old 1.35 control plane, and then upgrade the control plane. Steps 1 and 2 are interchangeable. The only rule is that the bridge add-ons land before the nodes.

Step 3: Roll the nodes

Set node_group_version = "1.36":

terraform plan  -target=aws_eks_node_group.ondemand-node
terraform apply -target=aws_eks_node_group.ondemand-node

The node group rolls one node at a time. Every new node comes up on 1.36 running add-on versions that already support 1.36.

If you run Karpenter, its nodes are outside the managed node group. Check that your Karpenter release supports 1.36 before Step 1, then let drift replacement roll them onto the new AMI while respecting PDBs and disruption budgets. Watch progress with:

kubectl get nodes -o wide

Step 4 (optional): Move to the default 1.36 add-on versions

Once all nodes are on 1.36, you can move add-ons to the versions EKS marks as default for 1.36. This command shows them:

K8S_VERSION="1.36"

for addon in vpc-cni coredns kube-proxy aws-ebs-csi-driver metrics-server; do
  echo "===== $addon ====="

  aws eks describe-addon-versions \
    --addon-name "$addon" \
    --kubernetes-version "$K8S_VERSION" \
    --query 'addons[0].addonVersions[?compatibilities[?defaultVersion==`true`]].{Version:addonVersion,Default:compatibilities[?defaultVersion==`true`].defaultVersion}' \
    --output table

  echo
done

The --query keeps only the versions whose compatibility entry for 1.36 has defaultVersion set to true, and prints them as a table. The default is the version EKS would install if you created the add-on fresh on a 1.36 cluster. It can differ from the newest build, so compare it with the latest list from Step 0 before choosing.

Put the versions you pick into var.addons and apply:

terraform apply -target=aws_eks_addon.eks-addons

This step is optional. A bridge version is already supported on 1.36, so you can stay on it and upgrade add-ons later on your own schedule. The one add-on to move promptly is kube-proxy, so it matches the node version again.

Step 5: A final plain apply

-target is meant for exceptional cases, and it can leave the rest of your state unreconciled. Finish with a normal run and confirm it reports nothing unexpected:

terraform plan
terraform apply

The order at a glance

Step	Action	Result
0	Find bridge versions	nothing changed
1	Upgrade control plane (`-target=aws_eks_cluster.eks`)	cluster 1.36, nodes 1.35
2	Apply bridge add-ons (`-target=aws_eks_addon.eks-addons`)	add-ons valid on 1.35 and 1.36
3	Roll nodes (`-target=aws_eks_node_group.ondemand-node`)	nodes 1.36
4	Optional: default 1.36 add-on versions	add-ons on EKS defaults for 1.36
5	Plain `terraform apply`	state fully reconciled

Steps 1 and 2 are interchangeable.

Caveats

A bridge version reduces upgrade risk. It does not remove all of it. You still need multiple replicas for critical workloads, sensible PodDisruptionBudgets, and a check for deprecated APIs.
force_update_version = true can override a PDB during the node roll. It avoids stuck upgrades, but it also means the PDB is not an absolute guarantee.
If the script shows no overlapping version for an add-on, you'll need an intermediate hop for that one.
Read the release notes for each add-on. A version being listed as compatible is not a guarantee that nothing changed.

Takeaways

The AWS order (control plane, nodes, add-ons) leaves a window where new nodes run old add-ons.
A bridge version closes that window because it works on both Kubernetes versions, so it can go in before or after the control plane upgrade, as long as it lands before the nodes.
Use -target to control the order, but watch depends_on: give the node group its own version variable, and finish with a plain terraform apply.

Building Elastic GitHub Actions Runners on EKS with ARC & Karpenter

Ajinkya — Mon, 07 Sep 2026 07:38:47 +0000

Zero idle CI runners. That's the point.

No jobs = zero runners = zero wasted spend.
Traffic shows up. Infrastructure follows.

The flow

Workflow triggers → ARC creates a runner pod → pod lands on a Karpenter NodePool → if capacity is missing, it goes Pending → Karpenter provisions exactly what's needed → job runs → pod disappears → node consolidates.

No standing fleet. No idle EC2 between builds. No capacity planning spreadsheet. Just compute that exists for exactly the minutes a job needs it.

What makes the cost math real

Guaranteed QoS

CPU requests = limits. Memory requests = limits.

This gives Karpenter a clean bin-packing target instead of over-allocating — Karpenter sizes the node off what you ask for, so a sloppy request means a bigger, pricier instance than the job actually needs.

Storage reclaim policy matters

Reclaim policy should be Delete, not Retain.

With Retain, per-job PVCs and their backing EBS volumes can remain after ephemeral runners disappear. The runners scale to zero. The storage bill doesn't.

This is the single easiest detail to overlook when evaluating the true cost of an ephemeral CI architecture — check it before you trust the "zero idle spend" pitch.

Spot for most jobs, On-Demand for critical ones

It doesn't have to be a cluster-wide decision. Use different Karpenter NodePools and target the appropriate runner/workload based on the job.

Spot → cost optimization
On-Demand → workloads that can't tolerate interruption

Same elasticity. Different reliability and cost trade-off — decided per job, not once for the whole cluster.

Cold starts are the trade-off

With minRunners: 0, there are no warm runners waiting for jobs. The first job after an idle period needs to wait for:

Pod scheduling → Karpenter provisioning → EC2 startup → runner initialization.

And in kubernetes mode, a PVC may also need to be provisioned before the job even starts.

You're trading idle compute cost for startup latency. Fine for most CI workloads. Not ideal when you need sub-minute pipeline startup times.

Not a drop-in swap for `ubuntu-latest`

The container mode you choose determines how much your existing workflow needs to change.

dind mode — The runner gets a real Docker daemon sidecar. Most existing workflows can remain largely unchanged. Trade-off: a privileged pod, plus additional startup overhead on every job.

kubernetes mode — Every job/action runs as its own unprivileged Kubernetes pod. More isolation, but also more constraints:

No Docker daemon anywhere
Every job needs an explicit container: configuration
Image builds and pushes require daemonless tooling
Bash-specific steps can break if the container's default shell isn't Bash

Why this architecture works

Each layer has one responsibility:

ARC → Runner lifecycle
Kubernetes → Scheduling
Karpenter → Provisioning + consolidation
Spot / On-Demand → Cost vs. reliability

Stack them together and the infrastructure behaves almost like a function:

Input → Compute appears → Work executes → Compute disappears

How to actually build it

The architecture above isn't theoretical — here's the exact install path, end to end, for a Karpenter-backed, spot-first, kubernetes-mode runner scale set on EKS.

1. Namespaces + controller

One controller per cluster, installed once:

kubectl create namespace arc-systems
kubectl create namespace arc-runners

helm install arc \
  --namespace arc-systems --create-namespace \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

kubectl get pods -n arc-systems
# expect: arc-gha-runner-scale-set-controller-xxxx   1/1   Running

2. A PAT scoped to exactly one repo

Fine-grained token → Repository access: Only select repositories → the one target repo, nothing org-wide:

Permission	Access level	Why
Actions	Read and write	Register/deregister runners, read the job queue
Administration	Read and write	Create/remove self-hosted runners on the repo
Metadata	Read-only	Required baseline

kubectl create secret generic eks-spot-amd-gh-secret \
  --namespace arc-runners \
  --from-literal=github_token=<YOUR_PAT>

3. The values file — where the cost math actually lives

# values-eks-spot-amd.yaml
githubConfigUrl: "https://github.com/<your-username>/<your-repo>"
githubConfigSecret: eks-spot-amd-gh-secret
runnerScaleSetName: "eks-spot-amd"

minRunners: 0     # zero idle runners — this is the whole point
maxRunners: 3     # keep tight for a single repo/workflow

containerMode:
  type: "kubernetes"
  kubernetesModeWorkVolumeClaim:
    accessModes: [ReadWriteOnce]
    storageClassName: gp3
    resources:
      requests:
        storage: 5Gi

template:
  spec:
    securityContext:
      fsGroup: 1001
    nodeSelector:
      node-pool: spot-amd64
      capacity-type: spot
      arch: amd64
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        resources:
          requests: { cpu: "2", memory: "4Gi" }   # requests == limits → Guaranteed QoS
          limits:   { cpu: "2", memory: "4Gi" }

requests == limits is not stylistic — it's the line that actually produces Guaranteed QoS and gives Karpenter a clean bin-packing target instead of an over-provisioned guess.

Install:

helm install eks-spot-amd \
  --namespace arc-runners \
  -f values-eks-spot-amd.yaml \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

kubectl get pods -n arc-runners
# expect: eks-spot-amd-xxxx-listener   1/1   Running   (idle, long-poll)

4. Confirm the reclaim policy before you trust the cost story

This is the one-line check that separates "zero idle spend" from "zero idle spend, plus a slowly growing EBS bill nobody notices":

kubectl get storageclass gp3 -o jsonpath='{.reclaimPolicy}'

If it says Retain instead of Delete, fix it before going further — otherwise every ephemeral job leaves a volume behind.

5. Route jobs to Spot or On-Demand per NodePool

Rather than a cluster-wide choice, tag NodePools separately and target them with nodeSelector per runner scale set — Spot for the bulk of jobs, a second On-Demand scale set for anything interruption-sensitive (release builds, deploy gates). Same controller, same chart, two values files, two nodeSelector blocks.

6. Adjust the workflow for your container mode

If you're on kubernetes mode, every job needs an explicit container:, and anything touching Docker (builds, registry logins, local-scan patterns) needs daemonless tooling instead — no /var/run/docker.sock exists anywhere in this mode by design.

jobs:
  build:
    runs-on: eks-spot-amd
    container:
      image: ghcr.io/catthehacker/ubuntu:act-22.04   # glibc-based — musl breaks JS actions
    defaults:
      run:
        shell: bash   # confirm the image's default shell isn't /bin/sh
    steps:
      - uses: actions/checkout@v4
      - run: make build

If most of your existing workflows assume a live daemon and rewriting them isn't practical yet, start on dind mode instead — same cost model, none of the container-mode constraints, at the cost of a privileged pod.

You also don't have to pick one mode for the whole cluster. Run two runner scale sets side by side — one on kubernetes mode for the bulk of your jobs, one on dind mode scoped to just the jobs that genuinely need a Docker daemon (image builds, registry pushes, container actions). Same controller, same chart, two values-*.yaml files, two runnerScaleSetNames. Point the daemon-dependent jobs at the dind scale set's runs-on: and leave everything else on kubernetes:

jobs:
  test:
    runs-on: eks-spot-amd            # kubernetes mode — no daemon needed
    container:
      image: ghcr.io/catthehacker/ubuntu:act-22.04
    steps:
      - uses: actions/checkout@v4
      - run: make test

  build-and-push:
    runs-on: eks-spot-amd-dind       # dind mode — needs a real daemon
    steps:
      - uses: actions/checkout@v4
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: registry/image:${{ github.sha }}

You get the smaller, unprivileged blast radius everywhere it's free to have it, and you contain the privileged trade-off to exactly the jobs that can't avoid it — instead of either rewriting your entire Docker-based pipeline around daemonless tooling, or running every job privileged just because one of them needs to be.

7. Validate the full elastic loop

kubectl get pods -n arc-runners -w                                           # ephemeral pod appears on trigger
kubectl get pod <pod> -n arc-runners -o jsonpath='{.spec.nodeName}'           # landed on the intended NodePool?
kubectl get node <node> --show-labels | grep -o 'capacity-type=[a-z]*'        # spot or on-demand, as intended?
kubectl get pod <pod> -n arc-runners -o jsonpath='{.status.qosClass}'         # actually Guaranteed?
kubectl get pvc -n arc-runners                                                # gone after the job completes?

Trigger a real workflow and watch the whole function execute: Input → Compute appears → Work executes → Compute disappears.

If the PVC and node are both gone a few minutes after the job finishes, the cost math in this post is real for your cluster, not just in theory.

If you're running ARC on EKS, I'd love to hear what NodePool split or container mode you landed on — drop a comment below.

Enabling Karpenter on EKS with Terraform: What It Is, Why It's Worth It, and How to Set It Up

Ajinkya — Sun, 06 Sep 2026 05:51:35 +0000

Karpenter replaces the old "guess an instance type, set a min/max, wait for Cluster Autoscaler" model with something simpler: watch for pods that can't schedule, and launch exactly the EC2 capacity they need. This guide covers how it actually works, why teams adopt it, and the Terraform you need to wire up to run it on EKS.

Prerequisites

Before you begin, make sure you have:

A running Amazon EKS cluster (created through either Terraform, eksctl, or the console)
Terraform (v1.11+ recommended)
An existing VPC and subnets
Existing security groups
Helm (v3+)
kubectl configured to access your EKS cluster
AWS CLI with credentials configured

How Karpenter Actually Works

Traditional autoscaling works in two disconnected layers: the Kubernetes scheduler decides where pods should go, and a separate Cluster Autoscaler watches Auto Scaling Groups and adds nodes when they're full. The ASG only knows one instance type per group, so it's constantly a step behind what the scheduler actually needs.

Karpenter collapses those two layers into one loop:

A pod goes Pending because nothing can schedule it.
Karpenter looks at that pod's actual requirements CPU, memory, architecture, zone, GPU, whatever's in its spec not a pre-defined group.
It picks the cheapest EC2 instance type across the entire fleet that satisfies those requirements, and calls the EC2 Fleet API directly to launch it.
The node joins the cluster, the pod schedules, done typically in under a minute.
On the way back down, Karpenter continuously watches for nodes that are empty or underutilized and consolidates workloads to shut them down.

There's no ASG in this loop at all. Karpenter talks to EC2 directly, which is what makes it fast and lets it pick from hundreds of instance types instead of the two or three you'd hand-pick for a node group.

Why This Is Beneficial

Right-sized capacity, every time. Nodes are chosen per-pod's actual needs instead of forcing every workload into whatever instance type a node group happened to use.
Faster scaling. Direct EC2 Fleet calls skip the ASG polling loop, so new capacity shows up in seconds to a couple of minutes instead of several minutes.
Less to manage. One NodePool and EC2NodeClass replace a sprawl of hand-tuned node groups, one per instance-type-and-AZ combination.
Safer use of Spot. Built-in interruption handling means Spot stops being "risky" and becomes just cheaper capacity with a warning light.
Self-healing infrastructure. Nodes expire and get replaced automatically, so patching and drift correction happen on a schedule instead of a ticket.

How the Cost-Effective Strategies Are Implemented

Karpenter's cost savings aren't a side effect they come from specific, configurable behaviors:

Spot-first, mixed with On-Demand. The NodePool allows both capacity types, and Karpenter defaults to the cheapest option that satisfies the pod's constraints usually Spot, at up to 90% off On-Demand pricing:

requirements:
  - key: karpenter.sh/capacity-type
    operator: In
    values: ["spot", "on-demand"]

Consolidation. Karpenter continuously re-evaluates the cluster. If pods on an underused node could be bin-packed onto existing capacity, it moves them and terminates the now-empty node:

disruption:
  consolidationPolicy: WhenEmptyOrUnderutilized
  consolidateAfter: 1m

Broad instance selection. By allowing whole instance categories (c, m, r) instead of a single type, Karpenter can pick whichever specific instance is cheapest and most available in that family at that moment rather than you guessing one in advance.

Scheduled node expiry. Every node is replaced after a fixed lifetime, which doubles as automatic AMI patching no cost from stale, oversized, or drifted long-lived nodes:

expireAfter: 720h

Graceful Spot interruption handling. Losing a Spot instance mid-job is what makes teams avoid Spot in the first place. Karpenter listens for the 2-minute interruption warning via SQS and EventBridge, drains the node in time, and replaces it so you get the discount without the risk of a workload dying uncleanly.

Setting It Up

1. IAM Roles

Karpenter needs two separate roles because two separate things assume them: the controller pod, and the EC2 instances it creates.

Controller role: trusted by your OIDC provider (IRSA), scoped to the karpenter service account:

resource "aws_iam_role" "karpenter_controller" {
  name = "KarpenterControllerRole-${var.cluster_name}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = aws_iam_openid_connect_provider.eks.arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${local.oidc_issuer}:sub" = "system:serviceaccount:karpenter:karpenter"
          "${local.oidc_issuer}:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}

Node role: trusted by EC2, attached to every node Karpenter launches:

resource "aws_iam_role" "karpenter_node" {
  name = "KarpenterNodeRole-${var.cluster_name}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect    = "Allow",
      Principal = { Service = "ec2.amazonaws.com" },
      Action    = "sts:AssumeRole"
    }]
  })

  tags = var.tags
}

resource "aws_iam_role_policy_attachment" "node_policies" {
  for_each = toset([
    "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
    "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
    "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly",
    "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
  ])
  role       = aws_iam_role.karpenter_node.name
  policy_arn = each.value
}

2. The Controller Policy

The controller role above only has a trust policy so far it says who can assume it, not what it's allowed to do. That comes from a separate inline policy granting ec2:RunInstances, ec2:CreateFleet, ec2:TerminateInstances, instance-profile management, and iam:PassRole on the node role.

Don't hand-write this from scratch Karpenter's maintainers publish and maintain the canonical version as part of the official Getting Started CloudFormation template:

Official reference: karpenter.sh/docs/reference/cloudformation
Raw policy source: https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml

Pull the KarpenterControllerPolicy statements out of that template into a local JSON file, templatize the account-specific values, and render it with templatefile():

resource "aws_iam_role_policy" "karpenter_controller_policy" {
  name = "KarpenterControllerPolicy-${var.cluster_name}"
  role = aws_iam_role.karpenter_controller.id
  policy = templatefile("${path.module}/policies/karpenter-controller-policy.json", {
    cluster_name            = var.cluster_name
    region                  = var.region
    karpenter_node_role_arn = aws_iam_role.karpenter_node.arn
    eks_cluster_arn         = aws_eks_cluster.eks.arn
    sqs_queue_arn           = aws_sqs_queue.karpenter_interruption.arn
  })
}

Templating it this way means the policy JSON stays identical to the upstream version — only the account, region, cluster, role, and queue ARNs are substituted per environment so pulling in a future upstream policy update is a diff, not a rewrite.

3. Let the Nodes Actually Join the Cluster

If your cluster uses the older aws-auth ConfigMap, you'd map the node role there. On newer clusters with authentication_mode = "API", you skip the ConfigMap entirely and create an access entry instead:

resource "aws_eks_access_entry" "karpenter_node" {
  cluster_name  = var.cluster_name
  principal_arn = aws_iam_role.karpenter_node.arn
  type          = "EC2_LINUX"
}

That's the entire equivalent of the old mapRoles block — one resource, no YAML.

4. Make Subnets and Security Groups Discoverable

Karpenter doesn't take a list of subnet IDs. It finds them by tag. Tag only your private subnets, and Karpenter will never place a node in a public one:

tags = {
  "karpenter.sh/discovery" = var.cluster_name
}

Put the same tag on the node security group. Two tags, and Karpenter's entire networking config is done.

5. SQS + EventBridge for Spot Interruptions

This is the piece that makes Spot safe to use. AWS gives a 2-minute warning before reclaiming a Spot instance — but only if something is listening. EventBridge catches that warning and drops it on a queue; Karpenter polls the queue and drains the node before AWS pulls it.

resource "aws_sqs_queue" "karpenter_interrupt" {
  name                      = "${var.cluster_name}-karpenter-spot-events"
  message_retention_seconds = 300
}

resource "aws_cloudwatch_event_rule" "spot_interruption" {
  name = "${var.cluster_name}-spot-interruption"
  event_pattern = jsonencode({
    source      = ["aws.ec2"]
    detail-type = ["EC2 Spot Instance Interruption Warning"]
  })
}

resource "aws_cloudwatch_event_target" "spot_interruption_to_sqs" {
  rule = aws_cloudwatch_event_rule.spot_interruption.name
  arn  = aws_sqs_queue.karpenter_interrupt.arn
}

Repeat the rule/target pair for Rebalance Recommendation, Instance State-change Notification, and AWS Health Event — four small rules, all feeding the same queue. Karpenter needs sqs:ReceiveMessage, DeleteMessage, and GetQueueAttributes on it, granted through the controller policy above.

6. Install Karpenter with Helm

Install the CRDs and the controller as two separate releases — this keeps future CRD upgrades from breaking against an immutable-field error:

resource "kubernetes_namespace_v1" "karpenter" {
  metadata {
    name = var.karpenter_namespace
  }
}

resource "helm_release" "karpenter_crds" {
  name             = "karpenter-crd"
  namespace        = var.karpenter_namespace
  repository       = "oci://public.ecr.aws/karpenter"
  chart            = "karpenter-crd"
  version          = var.karpenter_version
  create_namespace = false

  depends_on = [
    kubernetes_namespace_v1.karpenter
  ]
}

Then the controller itself, wired up to the queue, the IRSA role, and — critically — pinned to your existing managed node group:

resource "helm_release" "karpenter" {
  name             = "karpenter"
  namespace        = var.karpenter_namespace
  repository       = "oci://public.ecr.aws/karpenter"
  chart            = "karpenter"
  version          = var.karpenter_version
  create_namespace = false
  wait             = true
  cleanup_on_fail  = true
  timeout          = 600
  set = [
    {
      name  = "settings.clusterName"
      value = local.cluster_name
    },
    {
      name  = "settings.interruptionQueue"
      value = local.karpenter_queue_name
    },
    {
      name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
      value = local.karpenter_controller_role_arn
    },
    {
      name  = "controller.resources.requests.cpu"
      value = "1"
    },
    {
      name  = "controller.resources.requests.memory"
      value = "1Gi"
    },
    {
      name  = "controller.resources.limits.cpu"
      value = "1"
    },
    {
      name  = "controller.resources.limits.memory"
      value = "1Gi"
    },
    # Pin Karpenter pods to the existing managed node group
    {
      name  = "affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key"
      value = "eks.amazonaws.com/nodegroup"
    },
    {
      name  = "affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].operator"
      value = "In"
    },
    {
      name  = "affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].values[0]"
      value = local.node_group_name
    }
  ]
  depends_on = [
    kubernetes_namespace_v1.karpenter,
    helm_release.karpenter_crds
  ]
}

Why the node affinity block matters: Karpenter's job is to launch nodes for pods that can't schedule — but Karpenter itself runs as a pod. If Karpenter is allowed to schedule onto nodes it provisions, you get a chicken-and-egg deadlock: no Karpenter-provisioned nodes exist yet, so Karpenter can't start, so no nodes ever get created. The fix is to explicitly schedule Karpenter onto your base, pre-existing managed node group — the one created in the EKS/Terraform bootstrap, not anything Karpenter itself spins up. That node group's only job is to keep a stable home for cluster-critical controllers; everything else in the cluster is fair game for Karpenter to provision, resize, and recycle.

7. Tell It What to Launch: NodePool + EC2NodeClass

EC2NodeClass: the instance blueprint:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: KarpenterNodeRole-my-cluster
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags: { karpenter.sh/discovery: my-cluster }
  securityGroupSelectorTerms:
    - tags: { karpenter.sh/discovery: my-cluster }

NodePool: the rules for when and what:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m

Wrapping Up

Two roles plus one templated controller policy, one access entry, two tags, one queue with four EventBridge rules, two Helm releases pinned to your base node group, and two YAML manifests. Nothing here is complicated on its own — it's just a lot of small pieces that all have to point at each other correctly. Once they do, scaling stops being something you configure per-workload and becomes something the cluster just does, at whatever price the market currently offers.

Originally published on Medium.

Deploying Envoy Gateway on AWS EKS: The Right Way

Ajinkya — Thu, 18 Jun 2026 16:33:49 +0000

Some context first: we were running on GKE Autopilot, where the Gateway API just works out of the box. Google manages the CRDs and the underlying load balancer controller for you create a Gateway, and it gets an external IP without you ever thinking about CRD lifecycle.

Moving that same ingress layer to EKS meant none of that was ready to use anymore. The first real decision wasn't about Envoy Gateway's configuration at all it was about how to install its CRDs without them colliding with the Gateway API CRDs, or with each other, during future migrations.

Installing the Gateway API CRDs

We start by installing the Gateway API CRDs first:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml

What this installs:

GatewayClass CRD
Gateway CRD
HTTPRoute CRD
ReferenceGrant CRD

Verify with:

kubectl get crd | grep gateway.networking.k8s.io

Installing Envoy Gateway's CRDs and Controller via ArgoCD

Next, install Envoy Gateway's own CRDs and the controller itself as two separate ArgoCD Applications, on two separate sync waves:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: envoy-gateway-crds
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  project: default
  source:
    repoURL: oci://docker.io/envoyproxy
    chart: gateway-crds-helm
    targetRevision: v1.8.0
    helm:
      values: |
        crds:
          gatewayAPI:
            enabled: false       # Gateway API CRDs managed separately
            channel: experimental
          envoyGateway:
            enabled: true        # Only Envoy-specific CRDs
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=false
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: envoy-gateway
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  project: default
  source:
    repoURL: oci://docker.io/envoyproxy
    chart: gateway-helm
    targetRevision: v1.8.0
    helm:
      skipCrds: true             # CRDs managed by envoy-gateway-crds app
  destination:
    server: https://kubernetes.default.svc
    namespace: envoy-gateway-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - CreateNamespace=true

If you're not using ArgoCD, the equivalent Helm commands are:

helm install envoy-gateway-crds oci://docker.io/envoyproxy/gateway-crds-helm \
  --version v1.8.0 \
  --namespace default \
  --server-side \
  --set crds.gatewayAPI.enabled=false \
  --set crds.gatewayAPI.channel=experimental \
  --set crds.envoyGateway.enabled=true

helm install envoy-gateway oci://docker.io/envoyproxy/gateway-helm \
  --version v1.8.0 \
  --namespace envoy-gateway-system \
  --create-namespace \
  --skip-crds \
  --server-side

A few things matter here:

gatewayAPI.enabled: false the shared Gateway API CRDs (GatewayClass, Gateway, HTTPRoute, ReferenceGrant) aren't installed by this chart. They're installed once, separately, by their own Application, independent of any controller.
envoyGateway.enabled: true this chart installs only Envoy Gateway's own CRDs, including EnvoyProxy, on sync-wave -1, before the controller exists.
skipCrds: true on the envoy-gateway chart (wave 0) the controller deployment goes in after its CRDs already exist, and never touches CRD lifecycle itself.
ServerSideApply=true on both field-level ownership instead of whole-object ownership, so multiple Applications can touch overlapping CRDs without one overwriting the other.

Both Applications are templated as part of our cluster-bootstrap ApplicationSet, so every environment gets Envoy Gateway's CRDs-then-controller order automatically no manual sequencing per cluster for this part of the stack.

Architecture at a Glance

Internet
   │
   ▼
AWS NLB  (provisioned by AWS Load Balancer Controller)
   │
   ▼
Envoy Proxy Pods  (managed by Envoy Gateway, autoscaled by HPA)
   │
   ▼
Application Services  (via HTTPRoute rules)

The LoadBalancer Pending Trap

With the CRDs and controller in place, the next thing that breaks on a fresh EKS setup is the Service Envoy Gateway generates for its proxy deployment. By default, it's type LoadBalancer, and Kubernetes' in-tree cloud controller tries to provision a Classic Load Balancer for it.

On modern EKS clusters, that fails silently. No CLB gets created, no useful error appears in events, and the Service just sits at <pending> indefinitely.

The fix doesn't live on the Gateway object — Envoy Gateway generates its own Service internally, so there's nothing on Gateway.metadata to annotate. The fix has to go into the EnvoyProxy CRD, the same CRD installed separately in the -1 sync wave above, via envoyService.annotations:

envoyService:
  annotations:
    # Stops the in-tree CLB provisioner - AWS Load Balancer Controller
    # takes over and creates an NLB instead.
    service.beta.kubernetes.io/aws-load-balancer-type: "external"

    # Public-facing NLB. Use "internal" for private traffic only.
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"

    # Route NLB traffic directly to Pod IPs via VPC CNI -
    # bypasses kube-proxy and NodePort.
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

    # NLB health check hits Envoy's admin port. /healthz returns 200
    # only once Envoy is fully ready, so the NLB never routes to a
    # pod that's still starting or draining.
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: HTTP
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "19002"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/healthz"

The single annotation that actually breaks the deadlock is aws-load-balancer-type: "external" it tells the in-tree controller to back off, and hands the Service to the AWS Load Balancer Controller, which then provisions a real NLB and writes its hostname back to gateway.status.addresses. The rest of the block (scheme, target type, health check) is what makes that NLB actually production-ready rather than just "not pending."

Putting It Together: GatewayClass, Gateway, and EnvoyProxy

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: external-gateway
  namespace: gateway
spec:
  gatewayClassName: eg
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: external-proxy-config
  listeners:
    - name: http
      protocol: HTTP
      port: 80

And the EnvoyProxy CRD that ties resources, autoscaling, and the LB fix together in one place:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: external-proxy-config
  namespace: gateway
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        patch:
          type: StrategicMerge
          value:
            spec:
              template:
                spec:
                  containers:
                    - name: shutdown-manager
                      lifecycle:
                        preStop:
                          exec:
                            command: ["/bin/sh", "-c", "sleep 120"]
        container:
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              memory: 1Gi

      envoyHpa:
        minReplicas: 1
        maxReplicas: 5
        metrics:
          - type: Resource
            resource:
              name: cpu
              target:
                type: Utilization
                averageUtilization: 60

      envoyService:
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-type: "external"
          service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: HTTP
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "19002"
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/healthz"

Applying the Gateway triggers the chain reaction: Envoy Gateway creates the proxy deployment, the LoadBalancer Service with the annotations above, and an HPA — then AWS LBC provisions the NLB. HTTPRoute objects attach to the Gateway afterward and define per-service routing, owned by app teams.

Autoscaling Explained

The envoyHpa block creates a standard Kubernetes HPA against the proxy deployment. minReplicas: 1 keeps cost down during idle periods, at the cost of zero redundancy for ~15-30s if that pod dies. averageUtilization: 60 (150m of the 250m request) triggers scale-out early enough that new pods are healthy before latency degrades. For zero-downtime guarantees, minReplicas: 2 or 3 is the move.

Final Thoughts

Running Envoy Gateway on Amazon EKS isn't just about deploying another ingress controller — it's about understanding where the responsibilities are split.

Unlike managed Kubernetes offerings where the Gateway API experience is largely invisible, EKS gives you the flexibility to control every layer. That also means you own the lifecycle of the Gateway API CRDs, the Envoy Gateway CRDs, the controller installation, and the integration with the AWS Load Balancer Controller.

Separating CRDs from the controller, using ArgoCD sync waves to guarantee deployment order, and configuring the EnvoyProxy resource as the single place for infrastructure concerns makes the setup predictable and GitOps-friendly. It also avoids one of the most common migration issues: LoadBalancer Services remaining in a perpetual Pending state because the wrong controller is trying to provision them.

Why We Moved from GKE to EKS

Ajinkya — Sat, 25 Apr 2026 20:47:53 +0000

Why We Moved from GKE to EKS

When we initially adopted Kubernetes, Google Kubernetes Engine (GKE) Autopilot seemed like the perfect choice — fully managed, minimal operational overhead, and quick to get started.

But as our workloads matured, three major challenges started to surface:

Rising and unpredictable costs
Compliance constraints
The need for deeper infrastructure control

This blog walks through why we migrated to Amazon Elastic Kubernetes Service (EKS) with Karpenter, the architectural changes we made, and the lessons we learned running production workloads post-migration.

⚠️ Why GKE Autopilot Started Falling Short

1. Cost Inefficiencies at Scale

GKE Autopilot pricing is based on requested resources, not actual usage. This sounds fine at small scale — but as traffic grows, the gaps between requested and actual usage start to compound.

Problems we observed:

Over-provisioned workloads leading to higher bills
No access to Spot/Preemptible node strategies with the same level of flexibility
Very few cost optimization knobs to tune

As traffic grew, costs increased almost linearly with no meaningful way to optimize without restructuring our entire workload configuration.

2. Compliance and Governance Constraints

Operating in a regulated environment required:

Fine-grained IAM control at the workload level
Strict network isolation between services
Audit-level visibility into infrastructure activity

With GKE Autopilot, several configurations are abstracted away or restricted by design. This made it harder to enforce organization-wide security policies and satisfy compliance requirements from auditors. Specifically:

Enforcing per-pod IAM permissions cleanly was non-trivial
Network policy enforcement had gaps in our specific setup
Generating audit-ready logs tied to individual workload actions required workarounds

We needed something that gave us first-class integration with cloud-native IAM and security tooling — without layering on custom solutions.

3. Limited Infrastructure Control

When performance-sensitive services started hitting bottlenecks, the inability to choose instance types became a real blocker. We had no control over:

CPU vs. memory-optimized instance selection
ARM-based workloads on Graviton processors
Custom AMIs or low-level networking tuning

For teams running general-purpose workloads, this abstraction is a feature. For us, it was a ceiling.

🎯 Why We Chose EKS + Karpenter

Full Infrastructure Control

Moving to EKS gave us direct control over:

Instance families — CPU-optimized, memory-optimized, ARM (Graviton)
Custom AMIs — hardened images meeting our internal security baseline
Networking — VPC-native networking with fine-grained subnet and security group control

This unlocked workload-specific performance tuning that simply wasn't possible before.

Advanced Cost Optimization with Karpenter

Karpenter is not your traditional cluster autoscaler. Instead of scaling pre-defined node groups, it:

Watches for unschedulable pods in real time
Selects the right-sized instance based on actual pod requirements
Prioritizes Spot instances where workloads allow, falling back to On-Demand seamlessly
Bin-packs nodes efficiently, reducing idle capacity

The result: faster scaling reactions and a dramatically lower compute bill — without sacrificing reliability.

Compliance Alignment

AWS gave us the compliance story we needed:

IRSA (IAM Roles for Service Accounts) — precise, per-pod IAM permissions with no shared credentials
VPC-level isolation — full control over ingress, egress, and inter-service communication
CloudTrail integration — every API call, every node action, fully auditable out of the box
AWS Config + Security Hub — continuous compliance checks against CIS benchmarks and custom rules

This made our next compliance audit significantly smoother. Auditors got clear, traceable logs without us having to build custom instrumentation.

🏗️ Target Architecture

Here's what the high-level migration looked like architecturally:

Component	Before (GKE)	After (EKS)
Cluster	GKE Autopilot	EKS (Managed Node Groups + Karpenter)
Autoscaling	Built-in Autopilot scaling	Karpenter
Spot Strategy	NO Control	Karpenter Spot-first provisioning
IAM	GCP Workload Identity	AWS IRSA
Audit Logging	Cloud Audit Logs	CloudTrail + CloudWatch
Networking	GKE VPC-native	AWS VPC with custom subnets

⚙️ Karpenter Setup — The Game Changer

Karpenter replaced our traditional Cluster Autoscaler, and the difference was immediately visible.

How we configured it:

 apiVersion: karpenter.sh/v1
    kind: NodePool
    metadata:
      name: spot-arm64
    spec:
      template:
        metadata:
          labels:
            # -----------------------------------------------
            # These labels land on the EC2 node.
            # Your pod affinity rules match against these.
            # -----------------------------------------------
            node-pool: spot-arm64
            capacity-type: spot
            arch: arm64
            workload-class: standard
        spec:
          requirements:
            - key: kubernetes.io/arch
              operator: In
              values: ["arm64"]
            - key: kubernetes.io/os
              operator: In
              values: ["linux"]
            - key: karpenter.sh/capacity-type
              operator: In
              values: ["spot"]
            - key: karpenter.k8s.aws/instance-category
              operator: In
              values: ["c", "m", "r"]
              # c6g, c7g, c8g — compute optimized Graviton
              # m6g, m7g, m8g — general purpose Graviton
              # r6g, r7g, r8g — memory optimized Graviton
            - key: karpenter.k8s.aws/instance-generation
              operator: Gt
              values: ["5"]             # Graviton2+ only (gen 6,7,8)
          nodeClassRef:
            group: karpenter.k8s.aws
            kind: EC2NodeClass
            name: default
          expireAfter: 168h             # 7 days — shorter for spot nodes


      limits:
        cpu: 500
        memory: 2000Gi
      disruption:
        consolidationPolicy: WhenEmptyOrUnderutilized
        consolidateAfter: 2m
      weight: 100

Key decisions we made:

Spot-first provisioning — workloads that tolerate interruptions run on Spot; stateful services stay on On-Demand
Multiple instance families — Karpenter picks the cheapest right-sized option across families
Interruption handling — we use the Karpenter interruption queue (SQS) to gracefully drain Spot nodes before AWS reclaims them
consolidateAfter: 2m — nodes deprovision 2m seconds after going idle, eliminating ghost capacity

📊 Results After Migration

💰 Cost

Compute costs dropped significantly. The main drivers:

Spot instances covering the majority of our non-critical workloads
Karpenter's bin-packing eliminating idle node waste
Right-sized instances instead of over-provisioned static node groups

⚡ Performance

Faster pod scheduling — Karpenter provisions new nodes in under 60 seconds in most cases
Better workload isolation through custom node selectors and taints
Graviton (ARM) instances for compatible workloads gave us a meaningful price-performance improvement

🔐 Compliance

Audit reports now generated directly from CloudTrail without custom tooling
IRSA eliminated shared IAM credential risks
Security Hub provides continuous posture monitoring against our compliance framework

⚠️ Challenges We Faced

Being honest here — this is what makes a migration story actually useful.

1. Karpenter Learning Curve

Karpenter's provisioning model is fundamentally different from Cluster Autoscaler. Debugging why a node wasn't provisioned — or why Karpenter chose a specific instance type — required understanding its internal decision logic. The logs are verbose but not always immediately readable.

What helped: Running Karpenter in dry-run mode first, and adding structured logging to correlate provisioning decisions with pod events.

2. Networking Model Differences

GKE's VPC-native networking and AWS VPC behave differently in non-obvious ways — especially around CIDR planning, secondary IP ranges, and how pod IPs are allocated. We had to redesign our subnet layout and revisit some service-to-service communication assumptions.

3. IAM Complexity

IRSA is powerful but requires careful role design. Mapping GCP Workload Identity bindings to AWS IRSA role assumptions took time, especially for services that assumed broad IAM permissions under GCP that needed to be tightened properly.

🧠 Key Lessons Learned

Managed ≠ always optimal at scale. Autopilot is excellent for getting started, but production-grade platforms eventually need control surfaces that fully managed offerings deliberately hide.
Cost optimization requires infrastructure access. You can't tune what you can't see.
Autoscaling strategy matters more than cluster size. Karpenter's approach of provisioning for the pod rather than scaling a group changed how we think about capacity planning entirely.
Compliance is easier when the platform is designed for it. AWS's native compliance tooling removed a category of work that we were previously solving with custom scripts and log forwarding pipelines.
Migration should always be incremental. Parallel environment, gradual DNS cutover, canary deployments — this approach meant we caught issues in staging before they became production incidents.

🏁 Conclusion

GKE Autopilot is an excellent choice for teams that want Kubernetes without the operational overhead — and we'd still recommend it for that use case.

But for production environments that require cost control at scale, fine-grained compliance posture, and workload-specific infrastructure decisions, EKS with Karpenter provided a more flexible and efficient platform.

The migration wasn't trivial, but the control, visibility, and cost profile on the other side made it worth it.

Have you gone through a similar migration? Or are you evaluating EKS vs GKE for your stack? Drop your questions in the comments — happy to dig into specifics.

Tags: kubernetes aws devops cloud karpenter

DEV Community: Ajinkya

Stop Storing Access Keys in GitHub Secrets: Use OIDC for AWS

Index

The problem with static AWS credentials

What is AWS OIDC?

Heads up: GitHub just changed the token format

Why GitHub changed it

Build it with Terraform

1. Get your IDs

2. Variables

3. Provider and subject

4. Trust policy, role, and permissions

The workflow

Debugging: see what GitHub actually sends

Why this is secure

Wrapping up

Upgrade EKS Nodes Without Downtime: The Bridge Add-on Version Trick

The problem with "nodes first, add-ons after"

The idea: bridge versions

Step 0: Find the bridge version for each add-on

Output from my cluster (1.35 → 1.36)

How the Terraform is wired

The EKS cluster (control plane)

The node group

The add-ons

Gotcha: -target pulls in dependencies

The upgrade, step by step with -target

Step 1: Upgrade the EKS control plane

Step 2: Apply the bridge add-on versions

Step 3: Roll the nodes

Step 4 (optional): Move to the default 1.36 add-on versions

Step 5: A final plain apply

The order at a glance

Caveats

Takeaways

Building Elastic GitHub Actions Runners on EKS with ARC & Karpenter

The flow

What makes the cost math real

Guaranteed QoS

Storage reclaim policy matters

Spot for most jobs, On-Demand for critical ones

Cold starts are the trade-off

Not a drop-in swap for ubuntu-latest

Why this architecture works

How to actually build it

1. Namespaces + controller

2. A PAT scoped to exactly one repo

3. The values file — where the cost math actually lives

4. Confirm the reclaim policy before you trust the cost story

5. Route jobs to Spot or On-Demand per NodePool

6. Adjust the workflow for your container mode

7. Validate the full elastic loop

Enabling Karpenter on EKS with Terraform: What It Is, Why It's Worth It, and How to Set It Up

Prerequisites

How Karpenter Actually Works

Why This Is Beneficial

How the Cost-Effective Strategies Are Implemented

Setting It Up

1. IAM Roles

2. The Controller Policy

3. Let the Nodes Actually Join the Cluster

4. Make Subnets and Security Groups Discoverable

5. SQS + EventBridge for Spot Interruptions

6. Install Karpenter with Helm

7. Tell It What to Launch: NodePool + EC2NodeClass

Wrapping Up

Deploying Envoy Gateway on AWS EKS: The Right Way

Installing the Gateway API CRDs

Installing Envoy Gateway's CRDs and Controller via ArgoCD

Architecture at a Glance

The LoadBalancer Pending Trap

Putting It Together: GatewayClass, Gateway, and EnvoyProxy

Autoscaling Explained

Final Thoughts

Why We Moved from GKE to EKS

Why We Moved from GKE to EKS

⚠️ Why GKE Autopilot Started Falling Short

1. Cost Inefficiencies at Scale

2. Compliance and Governance Constraints

3. Limited Infrastructure Control

Gotcha: `-target` pulls in dependencies

The upgrade, step by step with `-target`

Not a drop-in swap for `ubuntu-latest`