DEV Community: Sathpal Singh

Running GitHub Actions Runner Controller on AKS Automatic: the blog post, executed end to end

Sathpal Singh — Mon, 21 Sep 2026 17:37:04 +0000

On 17 September 2026 the AKS engineering blog published Running GitHub Actions Runner Controller on AKS Automatic by Steve Griffith. It is a clean guide: create an AKS Automatic cluster, install Actions Runner Controller (ARC), register a runner scale set against a GitHub repository, run a workflow, watch it scale.

I followed it from a subscription that had never run AKS Automatic, and this post is what happened, command by command, with the real output. Most of it went exactly as written. Three things did not, and those are the useful parts.

Repo: github.com/sathpal/arc-aks-automatic-demo. One make target per step, a screenshot of each, a troubleshooting section for everything that went wrong, and the validation workflow. Fork it, set your GitHub user in .env, and the runners register against your fork.

What AKS Automatic changes

A standard AKS cluster gives you a Kubernetes API and lets you decide the rest. AKS Automatic decides for you: node auto-provisioning (Karpenter) instead of hand-sized node pools, Azure RBAC for Kubernetes with local accounts disabled, managed Prometheus and Grafana, and Deployment Safeguards, which are Gatekeeper policies that block or warn on common anti-patterns such as pods without resource requests or images tagged latest.

ARC is a good test of that. It has a controller, a listener per runner set, and ephemeral runner pods that appear when a job is queued and vanish when it finishes. The runner pods are exactly the kind of bursty workload node auto-provisioning is for, and the charts are exactly the kind of upstream Helm charts that Safeguards will have opinions about.

Step 0: two things the blog assumes you already have

A current Azure CLI. The guide's first command is az extension add --name aks-preview --upgrade. On my machine that pulled an aks-preview build that needs a newer CLI than the 2.72 I had, and every az aks command then died with a Python traceback. The --sku automatic flag is GA and lives in the core CLI, so the fix was not the extension but the CLI itself:

az extension remove -n aks-preview
brew upgrade azure-cli          # 2.72.0 -> 2.90.0
az aks create --help | grep -A2 -- --sku

Sixteen vCPUs of quota in one VM family that AKS Automatic likes. My first create attempt failed after a minute with:

AKS Automatic could not find a suitable VM size. The subscription may not have the
required quota of '16' vCPUs, may have restrictions, or location 'australiaeast' may
not support three availability zones for the following VM sizes: 'standard_d4lds_v5,
standard_d4ads_v5, standard_d4ds_v5, standard_d4d_v5, standard_d4d_v4, standard_ds3_v2,
standard_ds12_v2, standard_d4alds_v6, standard_d4lds_v6, standard_d4alds_v5'

Every D-family in every region I checked was at the default limit of 10 cores. The fix was a quota request through the CLI, which was approved instantly for one family:

az extension add -n quota
az quota update \
  --resource-name standardDaldv6Family \
  --scope "/subscriptions/<sub-id>/providers/Microsoft.Compute/locations/australiaeast" \
  --limit-object value=48 --resource-type dedicated

The same request for the DSv5 family came back "ContactSupport", so pick a family that is cheap and new; the v6 AMD sizes were the ones that went through for me.

Step 1: create the cluster

export LOCATION=australiaeast RG=rg-arc-auto-lab CLUSTER=arc-auto-lab
az group create --name $RG --location $LOCATION
az aks create --resource-group $RG --name $CLUSTER --location $LOCATION \
  --sku automatic --no-ssh-key

This took 33 minutes, not the ten I had budgeted from experience with standard AKS. Automatic creates a three-node system pool spread across zones, a hosted pool for its own add-ons, Karpenter, managed monitoring, and the policy stack, then waits for all of it to be healthy.

Local accounts are disabled, so you give yourself a Kubernetes RBAC role through Azure and use kubelogin:

AKS_ID=$(az aks show -g $RG -n $CLUSTER --query id -o tsv)
ME=$(az ad signed-in-user show --query id -o tsv)
az role assignment create --assignee $ME \
  --role "Azure Kubernetes Service RBAC Cluster Admin" --scope $AKS_ID
az aks get-credentials -g $RG -n $CLUSTER --format exec --overwrite-existing
kubectl get nodes -o wide

Six nodes on day one: three nodepool1 system nodes and three hostedpool nodes that AKS runs for its managed components. All on Azure Linux 3.0.

Step 2: install the controller

The controller chart has no resource requests by default, and Deployment Safeguards will refuse a pod without them. The blog's values file fixes that:

# arc-controller-values.yaml
resources:
  requests: { cpu: 100m, memory: 128Mi }
  limits:   { cpu: 500m, memory: 512Mi }

helm upgrade --install arc \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
  --namespace arc-systems --create-namespace --wait --timeout 10m \
  -f arc-controller-values.yaml

It installs, but Safeguards has more to say than the blog mentions. The install printed warnings that the controller container has no liveness or readiness probe, and that anti-affinity and topology spread constraints were added to the deployment by mutation. Those are warn-level policies, so the install goes through. The ones in deny mode, such as resource requests, are the ones that would have stopped it:

If you want the probe warning gone, the controller chart accepts livenessProbe and readinessProbe values. I left it as the blog has it so the output matches.

Step 3: token and runner scale set

kubectl create namespace arc-runners --dry-run=client -o yaml | kubectl apply -f -
printf '%s' "$GITHUB_TOKEN" | kubectl create secret generic github-pat \
  --namespace arc-runners --from-file=github_token=/dev/stdin

A classic PAT with repo scope is enough for a repository-level runner set. The blog says, and I agree, that anything beyond a lab should use a GitHub App so the permission is narrow and rotation is not a person's job.

The runner set values need three things to be explicit for Safeguards: resource requests on the listener, resource requests on the runner, and a pinned runner image tag. The default chart uses ghcr.io/actions/actions-runner:latest, and Safeguards rejects floating tags.

# arc-runner-set-values.yaml
githubConfigUrl: https://github.com/<owner>/<repo>
githubConfigSecret: github-pat
minRunners: 0
maxRunners: 3
listenerTemplate:
  spec:
    containers:
      - name: listener
        resources:
          requests: { cpu: 100m, memory: 128Mi }
          limits:   { cpu: 500m, memory: 512Mi }
template:
  spec:
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:2.337.0
        command: ["/home/runner/run.sh"]
        resources:
          requests: { cpu: "2", memory: 4Gi }
          limits:   { cpu: "2", memory: 4Gi }

The blog pins 2.336.0. By the time I ran this, 2.337.0 was current, and GitHub refuses runners more than a few versions behind, so check the releases page before you copy a tag.

helm upgrade --install arc-auto-runners \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
  --namespace arc-runners --wait --timeout 10m -f arc-runner-set-values.yaml

A listener pod appears in arc-systems and long-polls GitHub's broker for jobs. With minRunners: 0 nothing else runs until a job arrives, and the repository's runner list stays empty. That is normal and briefly alarming.

Step 4: run a workflow and watch it scale

The validation workflow from the blog, unchanged, committed to .github/workflows/arc-automatic-validation.yml with runs-on: arc-auto-runners, which is the Helm release name and therefore the runner label.

gh workflow run arc-automatic-validation.yml --repo <owner>/<repo> --ref main

I polled the run status, the runner pods and the node count every ten seconds:

What happened in those 190 seconds:

t+19s the listener saw the queued job and created one ephemeral runner pod. It went Pending: the system nodes are tainted for system workloads and the one schedulable node did not have 4Gi free.
t+53s node auto-provisioning launched a new node for it. Karpenter nominated the pod onto the node claim before the VM existed.
t+121s the node was ready and the pod pulled the runner image, which is about a gigabyte and took 52 seconds.
t+173s the job ran. It took 8 seconds.
t+190s the run reported success, the pod was gone, and the runner count was back to zero.

The node claim output is the part worth staring at. Karpenter picked a Standard_D4as_v6 on demand for the runner, and within minutes of the job finishing it flagged that node as a consolidation candidate, launched a cheaper D4als_v6 replacement and moved the controller and listener onto it, quoting the saving. Nobody configured any of that.

On the GitHub side it looks like any other run:

Current runner version: '2.337.0'
Runner name: 'arc-auto-runners-j4kzc-runner-x8p8j'
ARC runner reached workflow execution
Linux arc-auto-runners-j4kzc-runner-x8p8j 6.6.150.1-1.azl3 #1 SMP x86_64 GNU/Linux
Docker version 29.7.2, build a7dcaa6
Validation complete

The total of 2 minutes 48 seconds is almost entirely cold start: a new VM and a cold image pull. A second job arriving while the node is still there would start in seconds. If that latency matters, set minRunners: 1, and accept that you now pay for one idle D4.

Step 5: clean up

helm uninstall arc-auto-runners -n arc-runners
helm uninstall arc -n arc-systems
kubectl delete namespace arc-runners arc-systems
az group delete --name rg-arc-auto-lab --yes --no-wait

The resource group delete also removes the node resource group AKS created.

One more thing I learned by doing it wrong. I moved the runner set from one repository to another with helm upgrade and a new githubConfigUrl. The listener crashed on every start with "No runner scale set found with identifier 1": the set had kept the ID it registered under the first repository, and the second one had no such ID. Uninstall and reinstall is the only clean path, and if a stale AutoscalingListener object survives that, delete it and the controller recreates it in seconds.

Where I landed

If someone on my team asked whether to run our Actions runners this way, I would say yes, with two caveats that are about Azure rather than ARC.

The ARC side is boring in the best sense. Steve's values files are exactly what the cluster demands, the charts install first time, and the runner behaves like a hosted runner from the workflow's point of view. I did not change a line of the workflow. What I spent my time on was getting a cluster at all: the quota round trip and the CLI upgrade took longer than everything after them combined, and neither is mentioned in the post because a Microsoft engineer's subscription does not have those problems. Yours might. Run az vm list-usage for your region before you type az aks create, and check the CLI version before you add any extension.

The thing that changed my mind about AKS Automatic was not the cluster creation, which is slow, or Safeguards, which mostly nag. It was watching Karpenter after the job finished. It had bought a D4as_v6 for the runner, noticed twenty minutes later that the node was underused, priced a cheaper D4als_v6, launched it, moved the pods, and deleted the original. It printed the saving in the event log. I have written that logic by hand for other clusters and never got it this tidy.

The caveat on the ARC side is the token. I used my own PAT because it was a lab and I wanted to get to the interesting part. Do not do that for a team. A GitHub App gives the runner set its own identity with the permissions it needs and nothing else, and rotation stops being someone's memory.

Try it

gh repo fork sathpal/arc-aks-automatic-demo --clone && cd arc-aks-automatic-demo
make tools && cp .env.example .env         # set GITHUB_OWNER to your user
make quota && make cluster && make access && make arc
export GITHUB_TOKEN=$(gh auth token) && make runners && make test
make cleanup

Same app, two base images: 178 CVEs vs 5. A hands-on Chainguard on AKS walkthrough

Sathpal Singh — Tue, 15 Sep 2026 18:24:22 +0000

I watched the Cloud Native Partner Showcase episode where Microsoft's David Giard talks to Hannah Hawken and Manfred Moser from Chainguard about secure-by-default container images on Azure Kubernetes Service. Good conversation, but I wanted numbers I produced myself. So I built the smallest possible demo that proves or disproves the pitch, and this post is that demo, step by step, with the actual output.

The destination is Azure Kubernetes Service. The first steps build the evidence on your machine so you can see the difference before it reaches a cluster; the later steps put both images on AKS behind public load balancers and a Kyverno admission policy.

Repo: github.com/sathpal/chainguard-aks-demo

What we are testing

The claim from the episode: Chainguard images are minimal, rebuilt continuously from source, signed, and ship with an SBOM, so most CVEs never reach your cluster in the first place.

The test: one 40-line FastAPI app, built two ways.

	upstream	chainguard
base image	`python:3.13-slim`	`cgr.dev/chainguard/python`
user	root	nonroot (65532)
shell / package manager	yes	no

Then scan both, inspect both, deploy both, and see what the difference actually is.

Prerequisites

brew install trivy grype syft cosign crane      # scanners, SBOM, signing
# also: docker, kubectl, helm, azure-cli

Step 1: the app

The app deliberately reports what it is running on: OS, uid, whether a shell exists, how many OS packages are installed. Same code in both images.

def facts() -> dict:
    return {
        "image_flavor": os.getenv("IMAGE_FLAVOR"),
        "os": os_release(),                     # /etc/os-release PRETTY_NAME
        "uid": os.getuid(),
        "shell_present": any(Path(p).exists() for p in ("/bin/sh", "/bin/bash")),
        "package_manager_present": any(Path(p).exists() for p in ("/usr/bin/apt", "/sbin/apk", "/usr/bin/pip")),
        "os_packages": package_count(),         # dpkg or apk database
    }

Step 2: two Dockerfiles

The upstream one is what most tutorials show you:

FROM python:3.13-slim
WORKDIR /app
COPY app/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/main.py .
ENV IMAGE_FLAVOR=upstream
CMD ["python", "main.py"]

The Chainguard one is multi-stage. The -dev tag has pip and a shell so you can build; the plain tag has neither, so you only copy the result in:

FROM cgr.dev/chainguard/python:latest-dev AS builder
WORKDIR /home/nonroot
RUN python -m venv venv
ENV PATH="/home/nonroot/venv/bin:$PATH"
COPY app/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM cgr.dev/chainguard/python:latest
WORKDIR /app
COPY --from=builder --chown=nonroot:nonroot /home/nonroot/venv /app/venv
COPY --chown=nonroot:nonroot app/main.py .
ENV PATH="/app/venv/bin:$PATH" IMAGE_FLAVOR=chainguard
USER 65532
ENTRYPOINT ["python", "main.py"]

Three things bite people here. There is no shell in the runtime image, so CMD python main.py (string form) fails; use the exec form. pip install must happen in the builder stage, never in the runtime stage. And the builder stage works inside /home/nonroot, a directory that already exists and belongs to the nonroot user. The -dev image runs as nonroot, and while BuildKit creates a new WORKDIR owned by the current user, the legacy builder that ACR Tasks still uses creates it as root, so WORKDIR /app followed by python -m venv venv fails with permission denied the moment you build in the cloud instead of on your laptop. I found that one the hard way later in this post.

Step 3: build

docker build -f docker/Dockerfile.upstream   -t demo-app:upstream .
docker build -f docker/Dockerfile.chainguard -t demo-app:chainguard .
docker images demo-app

Step 4: scan

grype demo-app:upstream   -o table
grype demo-app:chainguard -o table

A small script summarises the two JSON reports into one table:

Same app. 178 known CVEs versus 5. Zero critical on the Chainguard side, and the five that remain are Wolfi packages with a fix already queued (zlib) or with no upstream fix yet (python). Rebuild tomorrow and the count moves. That is the "continuously rebuilt" part of the pitch working as described.

An honest note: my first scan showed 12 on the Chainguard side, not 5. Seven of those were in starlette, because I had pinned an old FastAPI in requirements.txt. The base image cannot fix your dependency file. Application-level CVEs stay your job.

Step 5: poke around inside

docker run --rm -it demo-app:upstream sh -c "id; which apt-get; ls /bin | wc -l"
docker run --rm -it --entrypoint sh demo-app:chainguard

The upstream container runs as root with apt-get and 259 binaries available to anyone who gets code execution. The Chainguard container cannot even start a shell. For debugging you use kubectl debug with an ephemeral container, which is a workflow change worth planning for (see cons below).

Step 6: verify the base image is really from Chainguard

Chainguard signs every image with Sigstore keyless signing from their GitHub release workflow. You can check that without any keys:

cosign verify cgr.dev/chainguard/python:latest \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity https://github.com/chainguard-images/images/.github/workflows/release.yaml@refs/heads/main

The signature is in the public Rekor transparency log. This is the difference between "we pulled an image called python" and "we pulled the image Chainguard's release pipeline built".

Step 7: SBOMs, yours and theirs

syft demo-app:upstream   -o spdx-json > out/upstream.sbom.spdx.json
syft demo-app:chainguard -o spdx-json > out/chainguard.sbom.spdx.json

# the SBOM Chainguard already attested for the base image
cosign verify-attestation cgr.dev/chainguard/python:latest --type https://spdx.dev/Document \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity https://github.com/chainguard-images/images/.github/workflows/release.yaml@refs/heads/main

109 packages to account for versus 47. When someone from compliance asks "are we affected by CVE-X", the smaller list answers faster.

Step 8: run both and look

docker run -d -p 8081:8080 demo-app:upstream
docker run -d -p 8082:8080 demo-app:chainguard

And the one-page report the scan script generates, useful for a slide:

Step 9: Azure Container Registry and AKS

This part costs money. Two small nodes for an hour is about a dollar, but delete the resource group when done.

az login --tenant <tenant-id>
az group create -n rg-chainguard-demo -l centralindia
az acr create -n sathpalcgdemo -g rg-chainguard-demo --sku Basic
az aks create -n aks-chainguard-demo -g rg-chainguard-demo --node-count 2 \
  --node-vm-size Standard_D2s_v4 --attach-acr sathpalcgdemo --generate-ssh-keys
az aks get-credentials -n aks-chainguard-demo -g rg-chainguard-demo

az acr login -n sathpalcgdemo
docker buildx build --platform linux/amd64 -f docker/Dockerfile.chainguard \
  -t sathpalcgdemo.azurecr.io/demo-app:chainguard --push .

Attaching ACR to AKS means the nodes pull with managed identity, no image pull secrets. Build for linux/amd64 if you are on Apple Silicon, since the default AKS node pool is x86.

Or skip local Docker entirely and let the registry build it. My laptop's disk filled up halfway through this post and Docker Desktop stopped working, so the Chainguard image in the cluster was actually built by ACR Tasks:

az acr build -r sathpalcgdemo -f docker/Dockerfile.chainguard -t demo-app:chainguard --platform linux/amd64 .

That is also where the WORKDIR /home/nonroot gotcha from Step 2 bit me. ACR Tasks uses the legacy builder, and my original WORKDIR /app failed with Permission denied: '/app/venv' on the first cloud build.

Two Standard_D2s_v4 nodes came up in about six minutes. If your subscription refuses the VM size, az vm list-usage -l <region> -o table shows which families you have quota for; mine had zero for B-series in Central India.

$ kubectl get nodes
NAME                                STATUS   ROLES    AGE     VERSION
aks-nodepool1-13038109-vmss000000   Ready    <none>   7m17s   v1.35.7
aks-nodepool1-13038109-vmss000001   Ready    <none>   7m16s   v1.35.7

The Chainguard deployment can turn on every hardening knob because the image cooperates:

securityContext:
  runAsNonRoot: true
  runAsUser: 65532
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities: { drop: ["ALL"] }

Try the same block on the upstream image and the pod fails runAsNonRoot immediately, because the image runs as uid 0.

The runAsUser: 65532 line is not decoration. The Chainguard base image declares its user numerically, but my first Dockerfile overrode it with USER nonroot, a name, and the kubelet refuses to start a runAsNonRoot container when it cannot prove the user is non-root from a name alone. The Dockerfile above now says USER 65532, and the manifest pins the same uid so the check holds even if someone changes the image later. My first rollout sat in CreateContainerConfigError with exactly that message:

Error: container has runAsNonRoot and image has non-numeric user (nonroot),
cannot verify user is non-root

Give it the numeric uid and it starts. Both deployments, each behind its own LoadBalancer:

$ kubectl -n chainguard-demo get pods,svc
NAME                              READY   STATUS    RESTARTS   AGE
app-chainguard-6f776d7959-nprs9   1/1     Running   0          89s
app-upstream-574978df4f-4pf9g     1/1     Running   0          5m6s
NAME             TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
app-chainguard   LoadBalancer   10.0.67.88     98.70.244.97     80:30308/TCP   5m5s
app-upstream     LoadBalancer   10.0.126.245   20.204.187.119   80:30890/TCP   5m6s

And the app reporting on itself from inside the cluster, same code, two answers:

$ curl -s http://20.204.187.119/api
{
    "image_flavor": "upstream",
    "hostname": "app-upstream-574978df4f-4pf9g",
    "os": "Debian GNU/Linux 13 (trixie)",
    "python": "3.13.15",
    "uid": 0,
    "running_as_root": true,
    "shell_present": true,
    "package_manager_present": true,
    "os_packages": 87
}

$ curl -s http://98.70.244.97/api
{
    "image_flavor": "chainguard",
    "hostname": "app-chainguard-6f776d7959-nprs9",
    "os": "Wolfi",
    "python": "3.14.7+",
    "uid": 65532,
    "running_as_root": false,
    "shell_present": false,
    "package_manager_present": false,
    "os_packages": 26
}

The same page from Step 8, now served by AKS through a public load balancer:

And the on-call moment. kubectl exec into the upstream pod gives you a root shell with apt-get. The same command on the Chainguard pod has nothing to run:

Step 10: enforce it with Kyverno

Scanning tells you. Admission control stops you. Two Kyverno policies:

# only trusted registries in this namespace
validate:
  message: "Images must come from cgr.dev or *.azurecr.io"
  pattern:
    spec:
      containers:
        - image: "cgr.dev/* | *.azurecr.io/*"

# every cgr.dev image must carry Chainguard's keyless signature
verifyImages:
  - imageReferences: ["cgr.dev/chainguard/*"]
    attestors:
      - entries:
          - keyless:
              subject: "https://github.com/chainguard-images/images/.github/workflows/release.yaml@refs/heads/main"
              issuer: "https://token.actions.githubusercontent.com"
              rekor: { url: https://rekor.sigstore.dev }

helm upgrade --install kyverno kyverno/kyverno -n kyverno --create-namespace
kubectl apply -f k8s/policies/
kubectl -n chainguard-demo run bad --image=docker.io/library/nginx:latest   # rejected
kubectl apply -f k8s/pod-chainguard-nginx.yaml                              # admitted

The rejection, verbatim from the API server:

$ kubectl -n chainguard-demo run bad --image=docker.io/library/nginx:latest
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request: 
resource Pod/chainguard-demo/bad was blocked due to the following policies 
restrict-image-registries:
  allowed-registries: 'validation error: Images must come from cgr.dev or *.azurecr.io (trusted, signed sources). rule allowed-registries failed at path /spec/containers/0/image/'

The signed image goes straight through, and Kyverno checked the Rekor entry for cgr.dev/chainguard/nginx on the way in:

$ kubectl apply -f k8s/pod-chainguard-nginx.yaml
pod/nginx-signed created
$ kubectl -n chainguard-demo get pod nginx-signed
NAME           READY   STATUS    RESTARTS   AGE
nginx-signed   1/1     Running   0          8s

One more real-world note: the namespace also carries the Pod Security Standards labels (enforce: baseline, warn: restricted). Every kubectl apply for the upstream deployment prints a warning that it would violate restricted. The Chainguard deployment is silent. That warning line is the cheapest security audit you will ever run.

Step 11: clean up

Everything above lives in one resource group, including the node resource group AKS creates for itself. One command stops the meter:

az group delete -n rg-chainguard-demo --yes --no-wait     # or: make aks-down

The public IPs in this post are gone by the time you read it. The repo is not.

Step 12: the "agent skill" part

The episode ends on AI-assisted workflows. I turned the migration steps above into a Claude Code skill (.claude/skills/chainguard-migrate/SKILL.md) plus a base-image mapping table. In the repo you say:

use the chainguard-migrate skill on docker/Dockerfile.upstream

and the agent baselines the image with grype, rewrites the Dockerfile to the multi-stage Chainguard pattern, rebuilds, rescans, verifies the base image signature and prints the before/after table. The skill's last rule is the important one: never claim zero CVEs without a fresh scan in the transcript. Agents that write security claims need receipts.

Pros

The CVE delta is real, not marketing. 178 to 5 on the same app, with the remaining 5 traceable to two packages.
Smaller attack surface by construction. No shell, no package manager, nonroot by default. A large class of post-exploitation tricks simply has nothing to run.
Signatures and SBOMs are already there. You verify with public tooling, no vendor account needed for the free images.
Kubernetes hardening becomes possible. readOnlyRootFilesystem, drop: ALL, PSS restricted profile. These fail on most upstream images.
Less scanner noise. Fewer findings means the ones left get read. Security teams stop being the department of 178 tickets.
Fits Azure without glue. Push to ACR, attach to AKS, listed on Azure Marketplace for procurement.

Cons

Debugging changes. No shell means kubectl exec is gone. You need kubectl debug with ephemeral containers, and your on-call runbooks need updating.
Free tier is latest only. Version-pinned tags (python:3.12) and the full catalog need a paid Chainguard account. For reproducible enterprise builds that is a budget conversation.
Wolfi is not Debian. Different package names, apk not apt, paths differ. Every RUN apt-get install in your Dockerfiles needs rethinking.
Multi-stage is mandatory. Single-stage Dockerfiles with build tools in the runtime image will not work. Good discipline, but it is work.
Your dependencies are still yours. My first scan had 12 findings and 7 were my outdated FastAPI pin. The base image does not fix requirements.txt, package.json or pom.xml.
Vendor concentration. You are trusting one company's rebuild pipeline. The signatures let you verify that trust, but it is a new dependency in your supply chain.

Who benefits

Platform teams on AKS who own the golden-image catalog and are tired of re-patching Debian bases every week.
Regulated teams (finance, health, public sector) who need SBOMs, provenance and a credible answer to "how do you know what is in production".
Small teams with no security engineer. Swapping the base image is the highest-leverage security change per hour of effort I know of.
Anyone adopting Pod Security Standards restricted. You cannot pass it with root images; this is the shortcut.
Teams introducing AI coding agents. A skill like the one above turns "harden this container" into a repeatable, auditable action instead of a vibe.

Who should wait: teams whose apps genuinely need a shell or system packages at runtime, and teams that need pinned versions but have no budget for the paid catalog yet.

Try it

git clone https://github.com/sathpal/chainguard-aks-demo && cd chainguard-aks-demo
make tools && make demo                                  # build, scan, verify, sbom, report
cp .env.example .env                                     # set your registry name and region
make aks-up && make acr-push && make deploy && make policy   # the AKS part, then make aks-down

If your numbers differ from mine, that is expected. The whole point is that both images change every day, and only one of them changes in your favour.

Picking the Latest Kubernetes Release on AKS Without Shooting Yourself in the Foot

Sathpal Singh — Fri, 24 Jul 2026 07:41:22 +0000

Picking the Latest Kubernetes Release on AKS Without Shooting Yourself in the Foot

Every few months a platform engineer opens a Slack message that reads something like: "Hey, our AKS cluster is on a version that's being retired — what do we do?" The answer is never "just click upgrade," but it also shouldn't be a three-week fire drill. This post walks through how AKS versions work, how to find what's actually available in your region, and how to set a defensible upgrade posture — without repeating the node pool mechanics we've already covered in earlier posts in this series.

How AKS Manages Version Support Windows

AKS doesn't support every Kubernetes minor version forever. Microsoft maintains a sliding support window — commonly described as the three most recent minor versions (N, N-1, N-2) — and retires older releases on a rolling basis. ¹ NEEDS_VALIDATION: confirm the exact N/N-1/N-2 window against current Microsoft docs, as the supported count has varied historically.

What that means operationally: if 1.30 is the current stable release, you can expect 1.29 and 1.28 to remain supported, while 1.27 approaches end-of-life. Clusters still running a retired version don't immediately explode, but they stop receiving security patches and you lose Microsoft support SLA coverage — two things you do not want to explain to your CISO.

Patch versions (the third digit) within a supported minor are more granular. AKS regularly ships node OS patches and Kubernetes patch releases; these are where most of the day-to-day CVE fixes land.

Finding What's Available in Your Region

Version availability is per-region. A version that GA'd last week in East US may still be in preview in Southeast Asia. Always query your actual deployment region before planning an upgrade:

az aks get-versions --location eastus --output table

NEEDS_VALIDATION: confirm --output table renders usefully for this subcommand in current CLI versions; --output json is always safe if the table format shifts.

The output lists minor versions, their patch variants, and whether each is in preview or GA. Treat anything marked preview as off-limits for production unless you have a specific reason and the risk tolerance to match.

If you want the current default version AKS would pick if you didn't specify one:

az aks get-versions --location eastus --output json \
  | jq '[.values[] | select(.isDefault == true)]'

NEEDS_VALIDATION: the isDefault field name — verify against current CLI JSON schema.

Key insight: the AKS default is not necessarily the latest GA patch. ¹ Microsoft selects a "recommended" version that has had some soak time. If you want the latest, you have to ask for it explicitly.

Pinning a Version at Cluster Create Time

To be explicit about your version at cluster creation:

az aks create \
  --resource-group my-rg \
  --name my-cluster \
  --location eastus \
  --kubernetes-version 1.30.2 \
  --node-count 3 \
  --generate-ssh-keys

NEEDS_VALIDATION: substitute the actual latest GA patch for your region; 1.30.2 is illustrative only — do not copy-paste without checking az aks get-versions first.

Omitting --kubernetes-version hands the decision to Microsoft's recommended default. That's fine for dev clusters. For production, pin it, document why you chose that version, and put the upgrade decision in your change process.

Auto-Upgrade Channels: Declaring Upgrade Intent

Manually upgrading clusters is fine until you have fifteen of them. Auto-upgrade channels let you express intent rather than managing individual upgrade events. AKS supports the following channels ¹:

none — no automatic upgrades; you control everything manually.
patch — automatically upgrades to the latest patch within the current minor version.
stable — targets the latest patch on the N-1 minor version.
rapid — targets the latest patch on the latest supported minor version (N).
node-image — upgrades only the node OS image, not the Kubernetes version itself.

NEEDS_VALIDATION: confirm exact current semantics for stable and rapid against Microsoft docs — the mapping of "N" vs "N-1" has been adjusted in past releases.

The node-image channel is worth calling out explicitly: node image upgrades and Kubernetes version upgrades are decoupled. ¹ You can roll fresh OS images (security patches, kernel updates) without touching your Kubernetes minor or patch version. This is the channel most production clusters should be running continuously — there's little reason to let node images go stale.

Setting the channel on an existing cluster:

az aks update \
  --resource-group my-rg \
  --name my-cluster \
  --auto-upgrade-channel patch

Gating Upgrades with Planned Maintenance Windows

Auto-upgrade without a maintenance window means upgrades can fire at any time AKS decides conditions are met. That's a bad day when it happens at 2 PM on a Tuesday during peak traffic. Planned Maintenance lets you constrain when auto-upgrades are allowed to execute. ¹ NEEDS_VALIDATION: confirm current Planned Maintenance API capabilities and whether it gates both K8s and node-image upgrades or only one.

apiVersion: maintenance.azure.com/v1
kind: MaintenanceConfiguration
metadata:
  name: aksmaintenance
spec:
  timeInWeek:
    - day: Saturday
      hourSlots:
        - 2
        - 3
  notAllowedTime:
    - start: "2024-12-20T00:00:00Z"
      end: "2025-01-03T00:00:00Z"

NEEDS_VALIDATION: the above YAML structure is illustrative — validate the exact CRD schema against current AKS documentation before applying.

Alternatively via CLI:

az aks maintenanceconfiguration add \
  --resource-group my-rg \
  --cluster-name my-cluster \
  --name aksManagedAutoUpgradeSchedule \
  --weekday Saturday \
  --start-hour 2

NEEDS_VALIDATION: confirm subcommand name and flags for the current CLI version.

A Concrete Upgrade Decision Process

Here's the numbered checklist I'd run through before touching a production cluster's Kubernetes version:

Query available versions in your target region with az aks get-versions. Confirm the version you want is GA, not preview.
Check your current cluster version with az aks show --resource-group my-rg --name my-cluster --query kubernetesVersion.
Review the Kubernetes changelog for the target minor version. Focus on API deprecations — if your workloads use APIs removed in the target version, fix that first.
Run kubectl deprecations (via the kubent tool or similar) against your cluster to catch any in-use deprecated APIs before the upgrade window. NEEDS_VALIDATION: confirm kubent compatibility with your target version.
Upgrade control plane first, validate, then upgrade node pools — this is standard AKS sequencing. If you need the node pool mechanics, see our earlier posts in this series.
Set your auto-upgrade channel to patch at minimum so you don't fall behind on patch releases between planned minor upgrades.
Configure a Planned Maintenance window so automated patch upgrades don't surprise you in business hours.

The Version Skew Problem

One thing that catches teams off guard: AKS allows your control plane and node pools to differ by at most one minor version, with the control plane always ahead. ² If you're running control plane 1.30 and node pools at 1.28, you have a problem — the node pools need to come up through 1.29 before they can reach 1.30. You cannot skip minor versions on node pools.

This is why letting clusters drift is expensive. A cluster that hasn't been upgraded in a year may require sequential minor version upgrades, each with its own validation cycle.

Putting It Together

Version management on AKS is less about clicking the upgrade button and more about having a policy that makes the decision for you at the right time. Set the patch auto-upgrade channel, add a maintenance window, keep node images continuously current via node-image channel, and plan minor version upgrades quarterly. That's a posture you can defend, automate, and sleep through.

The clusters that cause incidents are the ones where version management was treated as a one-time task rather than a continuous process.

Sources

Architecture

flowchart TD
    A([Platform Engineer]) --> B[az aks get-versions\n--location region]
    B --> C{Version Selection\nStrategy}
    C -->|Pin explicit version| D[az aks create\n--kubernetes-version X.Y.Z]
    C -->|Declare upgrade intent| E[Set Auto-Upgrade Channel]
    E --> F{Channel Choice}
    F -->|Conservative| G[patch\nStay on current minor]
    F -->|Balanced| H[stable\nN-1 minor version]
    F -->|Aggressive| I[rapid\nLatest supported minor]
    D --> J[AKS Cluster]
    G --> J
    H --> J
    I --> J
    J --> K[Planned Maintenance\nWindow Gates Upgrades]
    K --> L{Upgrade Scope}
    L -->|K8s version| M[Control Plane\n+ Node Pools]
    L -->|OS patches only| N[Node Image Upgrade\nDecoupled from K8s]

AKS and Near-Bare-Metal Workloads: What Platform Teams Can Responsibly Plan For

Sathpal Singh — Thu, 23 Jul 2026 16:55:32 +0000

AKS and Near-Bare-Metal Workloads: What Platform Teams Can Responsibly Plan For

Every few months a team shows up with a workload that has opinions about hardware. GPU inference, high-frequency packet processing, latency-sensitive financial systems — something that makes a shared-tenant VM feel like a traffic jam. The conversation usually ends up at "can we get bare-metal nodes on AKS?"

The honest answer right now: partially, with caveats, and you should not trust anyone who gives you a confident full answer without pointing at current documentation. This post is about what you can design around today, what requires validation before you commit to it, and where the gaps are that will bite you if you skip due diligence.

A note on sources: The AKS core concepts and product overview documentation available at time of writing covers cluster and workload fundamentals¹² but does not address bare-metal or near-bare-metal node pool specifics. Claims in that category are marked NEEDS_VALIDATION throughout. Everything else is grounded in general AKS behavior that is broadly consistent across the platform.

Why "Bare-Metal on AKS" Is Complicated Terminology

AKS runs on Azure VMs. Azure VMs run on Azure's physical fleet. The question is how much of the hypervisor overhead and hardware sharing model you can escape.

Azure offers VM SKU families that are designed to minimize virtualization overhead — isolated VM sizes that occupy an entire physical host, giving you a single-tenant hardware environment. Whether specific SKUs in this category are available as AKS node pool targets depends on regional availability and AKS's supported VM size list. NEEDS_VALIDATION — check the current AKS supported VM sizes documentation for your target region before designing around a specific SKU.

What you get with isolated or near-bare-metal SKUs, if available:

Predictable NUMA topology — no neighbor noise on the physical host
Full access to hardware features — relevant for SR-IOV, DPDK, and accelerated networking scenarios (NEEDS_VALIDATION for specific feature compatibility with AKS CNI configurations)
Single-tenant physical isolation — meaningful for compliance frameworks that require it

What you do NOT automatically get:

A different version of Kubernetes
Different node lifecycle behavior
Escape from AKS's node auto-repair or OS upgrade mechanics

That last point is where platform teams usually get surprised.

Node Lifecycle Is Still AKS's Domain

AKS manages node health and upgrades at the node pool level¹. This does not change because you picked a specialty VM SKU. If auto-repair decides a node is unhealthy and reprovisioned it, the replacement VM must come from available capacity of the same SKU. For isolated VM families, that capacity is often constrained — regionally and numerically.

The practical consequence: your cluster auto-repair and upgrade assumptions need re-evaluation for specialty node pools.

For standard node pools on commodity SKUs, AKS node upgrade behavior is well-documented and we've covered it in depth in our prior upgrade series (#2, #42, #50). The short version is that AKS handles node image upgrades via cordon-drain-replace, and you control the surge buffer and max unavailable settings. That mechanical behavior is the same regardless of SKU. What changes is the risk profile:

Capacity risk: If your isolated SKU has low regional availability, a surge node during upgrade may not provision. The upgrade stalls.
Drain timing risk: Pods on high-performance hardware often have longer graceful termination windows. Make sure your terminationGracePeriodSeconds reflects that.
OS upgrade compatibility: Node OS image updates on specialty hardware SKUs may lag behind standard SKU rollouts. NEEDS_VALIDATION — check AKS release notes for your target SKU family.

Node Pool Isolation Strategy

If you're mixing standard workloads and hardware-sensitive workloads in the same cluster (which is a reasonable cost efficiency decision), node pool isolation is non-negotiable. This is standard AKS practice¹ and applies regardless of whether the specialty pool is "bare-metal adjacent" or not.

The pattern:

Separate node pools by workload class — one pool for standard platform services, one for hardware-intensive workloads
Taint specialty node pools — prevent accidental scheduling of standard workloads onto expensive hardware
Use node selectors and tolerations explicitly — don't rely on the absence of a node being "wrong enough" to avoid it
Set resource requests and limits that reflect actual hardware — on high-performance SKUs with full NUMA access, misconfigured limits can be worse than no limits

Here's a minimal node pool configuration pattern for a tainted specialty pool:

# Create a specialty node pool with a taint to prevent accidental scheduling
# Replace <SKU_NAME> with your validated target SKU
az aks nodepool add \
  --resource-group myRG \
  --cluster-name myCluster \
  --name specialtypool \
  --node-vm-size <SKU_NAME> \ # NEEDS_VALIDATION: confirm SKU is AKS-supported
  --node-count 2 \
  --node-taints workload-class=hardware-intensive:NoSchedule \
  --labels workload-class=hardware-intensive \
  --zones 1 2 3

And the corresponding pod spec side:

apiVersion: v1
kind: Pod
metadata:
  name: hardware-sensitive-workload
spec:
  tolerations:
    - key: "workload-class"
      operator: "Equal"
      value: "hardware-intensive"
      effect: "NoSchedule"
  nodeSelector:
    workload-class: hardware-intensive
  containers:
    - name: app
      image: your-registry/your-image # pin your digest, don't use latest
      resources:
        requests:
          cpu: "8"
          memory: "32Gi"
        limits:
          cpu: "8"
          memory: "32Gi"

Concrete Steps Before Committing to a Specialty Node Pool

Validate SKU availability in your target region. Run az vm list-skus --location <region> --size <sku-prefix> --output table and cross-reference against the AKS supported VM sizes list. Do not assume a SKU that works for standalone VMs is available for AKS node pools.
Check current AKS release notes for the SKU family. Look specifically for known issues with node auto-repair, OS image compatibility, or accelerated networking feature support. NEEDS_VALIDATION — this is release-cadence dependent.
Model your capacity headroom. For upgrade surge and auto-repair replacement, you need available quota and available physical capacity. For isolated SKUs, physical capacity is the harder constraint. File a support ticket to confirm capacity commitments if the workload is business-critical.
Test node pool upgrade behavior in a non-production cluster first. Don't discover that your specialty SKU has zero surge capacity during a production Kubernetes version upgrade. The upgrade series we've covered previously (#50 is the most current approved version) gives you the tooling — apply the same runbook to a test specialty pool before you need it for real.
Define your eviction and rescheduling strategy. If a node is auto-repaired and the replacement takes longer than expected (capacity contention), what happens to your workload? PodDisruptionBudgets should reflect the actual tolerance of the workload, not the default.
Align with your golden path guardrails. If your platform has golden path templates (covered in #25), specialty node pools should either be an explicit supported variant or explicitly out-of-scope. The worst outcome is an undocumented workaround that every team discovers independently and implements differently.

What Not To Do

Don't build production dependency on a feature that's in preview. AKS preview features can change or be dropped. If a specific bare-metal adjacent capability is currently in preview, treat it as R&D, not production infrastructure.
Don't assume SKU-level isolation means network-level isolation. Physical host isolation and network security are different axes. Your CNI configuration and network policies still matter.
Don't skip the PodDisruptionBudget. On hardware-intensive workloads, the instinct is often "this thing is stateful and complex, let's not let Kubernetes touch it." That instinct leads to nodes that can't be drained and upgrades that time out.

The Responsible Summary

AKS can host workloads on VM SKUs designed to reduce virtualization overhead and provide single-tenant hardware access. The exact SKUs available, their compatibility with specific AKS networking and acceleration features, and their behavior under node lifecycle operations all require validation against current documentation before you commit. NEEDS_VALIDATION on the specifics is not a cop-out — it's the difference between a platform that works and a platform that works until it doesn't.

What you can design now: isolation strategy, taint/toleration patterns, upgrade runbooks, and capacity planning processes. Get those right and the hardware specifics slot in cleanly once validated.

Sources

Architecture

flowchart TD
    A[Platform Team Evaluation] --> B{Workload Requirements}
    B --> C[Standard Latency\nGeneral Purpose VMs]
    B --> D[Low Latency / Hardware Intensive\nSpecialty VM SKUs]
    D --> E[Validate SKU Availability\nby Region]
    E --> F[Dedicated Node Pool\nIsolation Strategy]
    F --> G[Review Auto-Repair &\nOS Upgrade Behavior]
    G --> H[Node Pool Taints & Tolerations\nfor Workload Scheduling]
    C --> H
    H --> I[Mixed Cluster:\nStandard + Specialty Pools]
    I --> J[Ongoing: Monitor AKS\nRelease Notes & SKU Docs]

Karpenter on AKS in 2026: What Actually Works

Sathpal Singh — Sun, 31 May 2026 05:21:06 +0000

Karpenter on AKS in 2026: What Actually Works

Karpenter on AKS has gone from "interesting experiment" to "something you can actually run in production" with some caveats that will save you a weekend of pain if you read them now. This post is a field report, not a sales pitch.

The Short Version

If you're running homogeneous, predictable workloads and you're happy with cluster-autoscaler (CAS), stay there. CAS is boring, it works, and Azure supports it fully. If you're running GPU workloads, spot-heavy batch pipelines, or you need bin-packing that doesn't require you to pre-define a node pool for every VM SKU you might want, Karpenter is now worth the operational overhead.

What Karpenter Actually Does on AKS

Karpenter watches for unschedulable pods and provisions nodes directly via the Azure provider, no VMSS node pools required for every SKU combination. It provisions, consolidates, and terminates nodes based on pod requirements and your defined NodePool and AKSNodeClass resources.

The AKS provider for Karpenter (karpenter-provider-azure) is a separate project from the AWS provider. Same core Karpenter engine, different provider implementation. This matters because feature parity with AWS Karpenter is not guaranteed and the cadence of releases differs.

Prerequisites and Installation

You need:

An AKS cluster with --network-plugin azure or --network-plugin overlay (Azure CNI in either mode works; kubenet is not supported)
A managed identity with the right RBAC, the provider needs to create and delete VMs and manage NICs, disks, and NSGs
Workload identity enabled on the cluster
Karpenter installed via Helm

The official installation path uses Helm with values pulled from your cluster. Here's a stripped-down install sequence:

# Set environment variables: replace with your actual values
export CLUSTER_NAME="my-aks-cluster"
export RESOURCE_GROUP="my-rg"
export LOCATION="eastus2"
export KARPENTER_NAMESPACE="kube-system"

# Get cluster details needed for Karpenter config
export SUBSCRIPTION_ID=$(az account show --query id -o tsv)
export NODE_RESOURCE_GROUP=$(az aks show \
  --name "${CLUSTER_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --query nodeResourceGroup -o tsv)

# Install via Helm
# NEEDS_VALIDATION: confirm chart version and repo URL against
# https://github.com/Azure/karpenter-provider-azure at time of deployment
helm upgrade --install karpenter oci://mcr.microsoft.com/aks/karpenter/karpenter \
  --namespace "${KARPENTER_NAMESPACE}" \
  --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.location=${LOCATION}" \
  --set "settings.subscriptionID=${SUBSCRIPTION_ID}" \
  --set "settings.resourceGroup=${NODE_RESOURCE_GROUP}" \
  --wait

Defining Your First NodePool

This is where Karpenter's model diverges most from node pools. Instead of pre-creating a pool for every SKU you might want, you define constraints and let Karpenter pick:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    metadata:
      labels:
        workload-type: general
    spec:
      nodeClassRef:
        apiVersion: karpenter.azure.com/v1alpha2
        kind: AKSNodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.azure.com/sku-family
          operator: In
          values: ["D", "E"]
        - key: karpenter.azure.com/sku-version
          operator: Gt
          values: ["3"]
  limits:
    cpu: "200"
    memory: 800Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s
---
apiVersion: karpenter.azure.com/v1alpha2
kind: AKSNodeClass
metadata:
  name: default
spec:
  imageFamily: AzureLinux
  osDiskSizeGB: 128

Note on API versions: The karpenter.azure.com API group is versioned separately from upstream Karpenter. v1alpha2 was current as of early 2026 but NEEDS_VALIDATION check the CRD definitions in the installed chart before you copy this into a GitOps repo.

What Works Well

Spot consolidation is the headline win. When spot VMs get preempted or you have underutilized nodes, Karpenter's consolidation loop handles bin-packing without you writing any automation. With CAS you're responsible for node pool min/max sizing and the consolidation is coarse.

Multi-SKU scheduling is the other real win. A pod requesting 8 vCPU and 64 GiB RAM will cause Karpenter to search the allowed SKU families for a node that fits, rather than failing because your single pre-configured node pool is exhausted.

GPU node provisioning works, including time-slicing scenarios, as long as your AKSNodeClass uses an image family that ships the NVIDIA drivers. AzureLinux with GPU extensions does this. You still need to manage the device plugin separately.

What Is Still Rough

Node provisioning latency is higher than AWS because Azure VM creation is slower than EC2. Plan for 3–5 minutes from pod pending to node ready on cold starts. This isn't a Karpenter problem per se, but it affects how you design your buffer capacity and PodDisruptionBudgets.

Windows node pools are not supported by the Azure Karpenter provider. If you have Windows workloads, keep a static node pool managed by CAS or manual scaling.

Custom VNet/subnet selection requires care. The AKSNodeClass lets you specify subnet IDs, but if you're using private clusters with complex network topologies, test thoroughly before rolling to production. Subnet exhaustion errors surface late and are annoying to debug.

Observability is immature. Karpenter emits metrics to Prometheus and logs to stdout, but the AKS provider's specific actions (VM creation, NIC attachment) aren't surfaced as well as you'd want. You'll be reading kubectl logs more than you'd like.

Concrete Steps to a Safe Rollout

Start with a non-production cluster. Run Karpenter alongside CAS, not instead of it. CAS can manage your system node pool; Karpenter handles a workload node pool namespace.
Define limits on your NodePool. Without a CPU/memory ceiling, a scheduling bug or runaway HPA can provision hundreds of nodes before you notice. Set limits conservatively and raise them deliberately.
Set consolidateAfter to something sane for your workload. 30 seconds is aggressive for stateful apps. Use 5–10 minutes for anything with slow startup or persistent volumes.
Test spot preemption handling. Deploy a test workload on spot nodes and manually deallocate a VM. Verify that Karpenter reprovisioned within your acceptable window and that your pod disruption budgets held.
Add Karpenter's node labels to your monitoring dashboards. Specifically track karpenter.sh/capacity-type, karpenter.azure.com/sku-name, and karpenter.sh/nodepool as label dimensions so you can see cost and performance breakdown by node type.
Pin your Helm chart version in GitOps. The provider is still in active development. Uncontrolled upgrades have broken NodePool CRD schemas between minor versions. Treat upgrades as a planned event.

CAS vs. Karpenter: The Honest Comparison

Concern	CAS	Karpenter
Operational maturity on AKS	Production-grade	Production-capable with caveats
Multi-SKU bin-packing	Requires pre-defined pools	Native
Spot handling	Decent	Better
Windows nodes	Yes	No
Debug tooling	Mature	Developing
Azure support	First-party	Community + Microsoft OSS

Final Thoughts

Karpenter on AKS in 2026 is the right choice if you have heterogeneous compute requirements and engineering capacity to own the operational model. It is not yet the "set it and forget it" experience that CAS is for straightforward clusters.

The Azure team has been shipping at a reasonable pace and the GitHub issues backlog is actually getting shorter, which is a good sign. The API is stabilizing. The path from alpha to beta to stable is visible.

Just don't copy that YAML into production without validating the API versions first. I warned you.