The Anatomy of Elastic CI: GitHub Actions ARC EKS Karpenter

Zero idle CI runners. That's the point.

No jobs = zero runners = zero wasted spend.
Traffic shows up. Infrastructure follows.

---

The flow

Workflow triggers → ARC creates a runner pod → pod lands on a Karpenter NodePool → if capacity is missing, it goes Pending → Karpenter provisions exactly what's needed → job runs → pod disappears → node consolidates.

No standing fleet. No idle EC2 between builds. No capacity planning spreadsheet. Just compute that exists for exactly the minutes a job needs it.

---

What makes the cost math real

Guaranteed QoS

CPU requests = limits. Memory requests = limits.

This gives Karpenter a clean bin-packing target instead of over-allocating — Karpenter sizes the node off what you ask for, so a sloppy request means a bigger, pricier instance than the job actually needs.

Storage reclaim policy matters

Reclaim policy should be Delete, not Retain.

With Retain, per-job PVCs and their backing EBS volumes can remain after ephemeral runners disappear. The runners scale to zero. The storage bill doesn't.

This is the single easiest detail to overlook when evaluating the true cost of an ephemeral CI architecture — check it before you trust the "zero idle spend" pitch.

Spot for most jobs, On-Demand for critical ones

It doesn't have to be a cluster-wide decision. Use different Karpenter NodePools and target the appropriate runner/workload based on the job.

• Spot → cost optimization
• On-Demand → workloads that can't tolerate interruption

Same elasticity. Different reliability and cost trade-off — decided per job, not once for the whole cluster.

Cold starts are the trade-off

With minRunners: 0, there are no warm runners waiting for jobs. The first job after an idle period needs to wait for:

Pod scheduling → Karpenter provisioning → EC2 startup → runner initialization.

And in kubernetes mode, a PVC may also need to be provisioned before the job even starts.

You're trading idle compute cost for startup latency. Fine for most CI workloads. Not ideal when you need sub-minute pipeline startup times.

---

Not a drop-in swap for ubuntu-latest

The container mode you choose determines how much your existing workflow needs to change.

dind mode — The runner gets a real Docker daemon sidecar. Most existing workflows can remain largely unchanged. Trade-off: a privileged pod, plus additional startup overhead on every job.

kubernetes mode — Every job/action runs as its own unprivileged Kubernetes pod. More isolation, but also more constraints:

• No Docker daemon anywhere
• Every job needs an explicit container: configuration
• Image builds and pushes require daemonless tooling
• Bash-specific steps can break if the container's default shell isn't Bash

---

Why this architecture works

Each layer has one responsibility:

• ARC → Runner lifecycle
• Kubernetes → Scheduling
• Karpenter → Provisioning + consolidation
• Spot / On-Demand → Cost vs. reliability

Stack them together and the infrastructure behaves almost like a function:

Input → Compute appears → Work executes → Compute disappears

---

How to actually build it

The architecture above isn't theoretical — here's the exact install path, end to end, for a Karpenter-backed, spot-first, kubernetes-mode runner scale set on EKS.

1. Namespaces + controller

One controller per cluster, installed once:

kubectl create namespace arc-systems
kubectl create namespace arc-runners

helm install arc \
  --namespace arc-systems --create-namespace \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

kubectl get pods -n arc-systems
# expect: arc-gha-runner-scale-set-controller-xxxx   1/1   Running

2. A PAT scoped to exactly one repo

Fine-grained token → Repository access: Only select repositories → the one target repo, nothing org-wide:

Permission	Access level	Why
Actions	Read and write	Register/deregister runners, read the job queue
Administration	Read and write	Create/remove self-hosted runners on the repo
Metadata	Read-only	Required baseline

kubectl create secret generic eks-spot-amd-gh-secret \
  --namespace arc-runners \
  --from-literal=github_token=<YOUR_PAT>

3. The values file — where the cost math actually lives

# values-eks-spot-amd.yaml
githubConfigUrl: "https://github.com/<your-username>/<your-repo>"
githubConfigSecret: eks-spot-amd-gh-secret
runnerScaleSetName: "eks-spot-amd"

minRunners: 0     # zero idle runners — this is the whole point
maxRunners: 3     # keep tight for a single repo/workflow

containerMode:
  type: "kubernetes"
  kubernetesModeWorkVolumeClaim:
    accessModes: [ReadWriteOnce]
    storageClassName: gp3
    resources:
      requests:
        storage: 5Gi

template:
  spec:
    securityContext:
      fsGroup: 1001
    nodeSelector:
      node-pool: spot-amd64
      capacity-type: spot
      arch: amd64
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        resources:
          requests: { cpu: "2", memory: "4Gi" }   # requests == limits → Guaranteed QoS
          limits:   { cpu: "2", memory: "4Gi" }

requests == limits is not stylistic — it's the line that actually produces Guaranteed QoS and gives Karpenter a clean bin-packing target instead of an over-provisioned guess.

Install:

helm install eks-spot-amd \
  --namespace arc-runners \
  -f values-eks-spot-amd.yaml \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

kubectl get pods -n arc-runners
# expect: eks-spot-amd-xxxx-listener   1/1   Running   (idle, long-poll)

4. Confirm the reclaim policy before you trust the cost story

This is the one-line check that separates "zero idle spend" from "zero idle spend, plus a slowly growing EBS bill nobody notices":

kubectl get storageclass gp3 -o jsonpath='{.reclaimPolicy}'

If it says Retain instead of Delete, fix it before going further — otherwise every ephemeral job leaves a volume behind.

5. Route jobs to Spot or On-Demand per NodePool

Rather than a cluster-wide choice, tag NodePools separately and target them with nodeSelector per runner scale set — Spot for the bulk of jobs, a second On-Demand scale set for anything interruption-sensitive (release builds, deploy gates). Same controller, same chart, two values files, two nodeSelector blocks.

6. Adjust the workflow for your container mode

If you're on kubernetes mode, every job needs an explicit container:, and anything touching Docker (builds, registry logins, local-scan patterns) needs daemonless tooling instead — no /var/run/docker.sock exists anywhere in this mode by design.

jobs:
  build:
    runs-on: eks-spot-amd
    container:
      image: ghcr.io/catthehacker/ubuntu:act-22.04   # glibc-based — musl breaks JS actions
    defaults:
      run:
        shell: bash   # confirm the image's default shell isn't /bin/sh
    steps:
      - uses: actions/checkout@v4
      - run: make build

If most of your existing workflows assume a live daemon and rewriting them isn't practical yet, start on dind mode instead — same cost model, none of the container-mode constraints, at the cost of a privileged pod.

You also don't have to pick one mode for the whole cluster. Run two runner scale sets side by side — one on kubernetes mode for the bulk of your jobs, one on dind mode scoped to just the jobs that genuinely need a Docker daemon (image builds, registry pushes, container actions). Same controller, same chart, two values-*.yaml files, two runnerScaleSetNames. Point the daemon-dependent jobs at the dind scale set's runs-on: and leave everything else on kubernetes:

jobs:
  test:
    runs-on: eks-spot-amd            # kubernetes mode — no daemon needed
    container:
      image: ghcr.io/catthehacker/ubuntu:act-22.04
    steps:
      - uses: actions/checkout@v4
      - run: make test

  build-and-push:
    runs-on: eks-spot-amd-dind       # dind mode — needs a real daemon
    steps:
      - uses: actions/checkout@v4
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: registry/image:${{ github.sha }}

You get the smaller, unprivileged blast radius everywhere it's free to have it, and you contain the privileged trade-off to exactly the jobs that can't avoid it — instead of either rewriting your entire Docker-based pipeline around daemonless tooling, or running every job privileged just because one of them needs to be.

7. Validate the full elastic loop

kubectl get pods -n arc-runners -w                                           # ephemeral pod appears on trigger
kubectl get pod <pod> -n arc-runners -o jsonpath='{.spec.nodeName}'           # landed on the intended NodePool?
kubectl get node <node> --show-labels | grep -o 'capacity-type=[a-z]*'        # spot or on-demand, as intended?
kubectl get pod <pod> -n arc-runners -o jsonpath='{.status.qosClass}'         # actually Guaranteed?
kubectl get pvc -n arc-runners                                                # gone after the job completes?

Trigger a real workflow and watch the whole function execute: Input → Compute appears → Work executes → Compute disappears.

If the PVC and node are both gone a few minutes after the job finishes, the cost math in this post is real for your cluster, not just in theory.

---

If you're running ARC on EKS, I'd love to hear what NodePool split or container mode you landed on — drop a comment below.