Kubernetes upgrade preflight checklist for AKS, EKS and GKE

Managed Kubernetes makes the control-plane upgrade a button. The button does not tell you what will break in the workloads, and it does not tell you that the node-pool roll will stall at the first node with a bad PodDisruptionBudget. This is the checklist to run on a cluster (call it prod-eu-1) before you press it. The first four sections apply everywhere; the last covers what differs on AKS, EKS and GKE.

0. Read the release notes and pick a target

Upgrade one minor version at a time. Kubernetes does not support skipping minors on the control plane, and the managed providers enforce this in different ways. Read the upstream changelog for the target version, specifically the "Urgent upgrade notes" and "Deprecation" sections, and your provider's own release notes for that version.

1. APIs removed in the target version

Kubernetes removes beta API versions on a published schedule. The authoritative list is the upstream deprecated API migration guide at kubernetes.io. Some well-known removals:

• 1.22: extensions/v1beta1 and networking.k8s.io/v1beta1 Ingress, apiextensions.k8s.io/v1beta1 CRDs, admissionregistration.k8s.io/v1beta1 webhooks.
• 1.25: batch/v1beta1 CronJob, policy/v1beta1 PodDisruptionBudget and PodSecurityPolicy, autoscaling/v2beta1 HorizontalPodAutoscaler, discovery.k8s.io/v1beta1 EndpointSlice.
• 1.26: autoscaling/v2beta2 HorizontalPodAutoscaler, flowcontrol.apiserver.k8s.io/v1beta1.
• 1.27: storage.k8s.io/v1beta1 CSIStorageCapacity.
• 1.29: flowcontrol.apiserver.k8s.io/v1beta2.
• 1.32: flowcontrol.apiserver.k8s.io/v1beta3.

The list is illustrative; check the guide for your exact target.

Find what still uses a removed API

There are three places a removed API can hide, and none of them alone is sufficient.

Live requests. The API server counts calls to deprecated APIs. This is the best signal that something is actively using one:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

Each series carries group, version, resource and removed_release labels. A non-zero series with removed_release="1.32" means a client called that API recently. Metrics reset on API server restart and on managed services you may not be able to scrape /metrics at all, so treat an empty result as "not seen", not "not used".

What the cluster still serves. Discovery lists every group/version the server answers:

kubectl api-versions | sort
kubectl api-resources --verbs=list -o name | sort

A beta group/version in this list that your target removes is the thing to chase.

What is stored and what is applied. Objects are stored in one version and served in others, so kubectl get returns the preferred version regardless of how it was written. Manifests are where the old version lives: grep Helm charts, Kustomize overlays, Terraform and CI templates.

grep -rEn "apiVersion: (extensions/v1beta1|policy/v1beta1|batch/v1beta1|autoscaling/v2beta1|autoscaling/v2beta2)" .
helm template my-release ./chart | grep -E "^apiVersion:" | sort | uniq -c

Also check what Helm has recorded: a release whose stored manifest uses a removed API can fail to upgrade even when the live objects are fine. The helm mapkubeapis plugin exists for that.

Finally, check admission webhooks and CRD conversion webhooks. A webhook registered with a removed admissionregistration version, or a CRD that declares a conversion webhook whose Service has no ready endpoints, will cause API errors that look unrelated to the upgrade.

2. PDBs that will stall node-pool upgrades

The control plane upgrade is quick. The node-pool upgrade is where clusters get stuck. Every provider's node upgrade drains each node, and drain respects PodDisruptionBudgets. A budget with zero allowed disruptions pins its pod, and the node waits.

kubectl get pdb -A
kubectl get pdb -A -o json | jq -r '
  .items[] | select(.status.disruptionsAllowed == 0)
  | "\(.metadata.namespace)/\(.metadata.name)"'

Three causes cover most cases:

• A Deployment with one replica and minAvailable: 1 (or maxUnavailable: 0). Allowed disruptions is always zero.
• minAvailable equal to the replica count.
• A workload that is already unhealthy, so the budget has nothing to give.

A roll with surge nodes does not fix this. Surge adds capacity so pods have somewhere to go; it does not change the budget. The pod still cannot be evicted. Providers handle the stall differently: some time out and fail the upgrade, some force the eviction after a window. See section 5, and read your provider's current documentation for the exact timeout.

Beyond budgets, a node roll also trips on:

• pods with no controller, which kubectl drain refuses without --force;
• pods with emptyDir data;
• pods that cannot reschedule: node selectors, required anti-affinity, taints, zonal volumes, or no spare capacity once a node is gone.

The same checks apply as for a single drain; why kubectl drain hangs covers each. The difference for an upgrade is that you need the answer for every node in the pool, not one.

Also confirm the pool can absorb a surge node: VM quota for the family and free IP addresses in the subnet. A surge that cannot get either fails for reasons unrelated to Kubernetes.

3. Version skew

The upstream version skew policy sets the limits. Verify the current numbers against it, since they have changed over time:

• The kubelet may not be newer than the API server. It may be older, by up to three minor versions on recent releases (it was two on older ones).
• kubectl is supported within one minor version of the API server, in either direction.
• Controller-manager, scheduler and cloud-controller-manager must not be newer than the API server.

On managed services the control plane moves first, so the practical rule is: after the control plane upgrade, the nodes will be behind, and you want that gap to stay inside the policy while you work through the pools. If your nodes are already two minors behind, upgrading the control plane another minor puts them at three, and then you have no room. Upgrade node pools soon after the control plane rather than letting them lag.

Check the gap:

kubectl get nodes -o custom-columns='NODE:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion'
kubectl version

Clients matter too: CI runners with an old kubectl, and operators built against client libraries that only speak a removed API version, break at the API server upgrade, not the node upgrade.

4. Add-ons

Add-ons are what actually break. Each one has its own compatibility matrix with Kubernetes versions, and most are not upgraded automatically.

Make a list:

kubectl get deploy,ds -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image'

Then, for each one, look up whether the target Kubernetes version is supported by the add-on version you run. Typical entries:

• CNI plugin and CSI drivers (storage drivers are the ones that hurt most when they are wrong).
• CoreDNS and kube-proxy, where the provider manages versions.
• Cluster autoscaler (its minor version should match the cluster's) and ingress controllers.
• cert-manager, service meshes, policy engines and any operator that ships CRDs.
• Anything with a webhook. A webhook with failurePolicy: Fail and no backing pod can block all writes of the resource it covers.

5. What to check per provider

These are the managed-service specifics that have been stable for a while. Confirm details against the current docs, because defaults and limits get changed.

EKS

• Upgrade insights. EKS runs checks for deprecated API usage and other issues and reports them per cluster. In the console or with aws eks list-insights --cluster-name prod-eu-1.
• Add-ons. Managed add-ons (VPC CNI, CoreDNS, kube-proxy, EBS CSI) have versions per Kubernetes release: aws eks describe-addon-versions --kubernetes-version <target> --addon-name <name>. Self-managed ones are your job.
• Order. Control plane first, then add-ons, then node groups. A control plane upgrade cannot be rolled back.
• Managed node groups. The update config has maxUnavailable (default one node at a time) or a percentage; raising it speeds up the roll but concentrates the disruption.

AKS

• Available versions. az aks get-upgrades -g <rg> -n prod-eu-1 -o table. AKS does not let you skip minor versions.
• Node surge. Each node pool has a max surge setting, with a default of a percentage of the pool; az aks nodepool update --max-surge 33% changes it. Surge nodes count against your VM quota and subnet addresses.
• Drain behaviour. AKS drains nodes during upgrade and honours PDBs; a drain that cannot finish within the node drain timeout fails the upgrade. Check the current timeout in the docs and set it deliberately.

GKE

• Release channels and maintenance windows. On a release channel the control plane upgrades on Google's schedule within your window; set exclusions for periods you cannot afford a change.
• Deprecation insights. GKE reports usage of deprecated APIs for upcoming versions in the console, and can pause automatic upgrades when it sees usage.
• Surge upgrades. Node pools use maxSurge and maxUnavailable; the defaults favour one surge node and no unavailable nodes. Blue-green is an alternative for pools that need to roll back quickly.
• PDBs. GKE honours PodDisruptionBudgets during node upgrades for a bounded time, after which it proceeds. Do not count on that: a budget that waits an hour then gets bypassed is not a design.

6. The checklist

1. Target version chosen, one minor step, release notes read.
2. Removed APIs: metric checked, discovery checked, manifests and Helm releases grepped, webhooks checked.
3. PDBs: none with zero allowed disruptions on a workload you are about to roll.
4. Drain blockers: unmanaged pods, emptyDir, unschedulable replacements, per node.
5. Quota and IP headroom for surge.
6. Skew: kubelet gap, kubectl, CI clients.
7. Add-on compatibility for the target version, with an upgrade order.
8. Backups and a rollback plan: the control plane usually cannot be downgraded, so the plan is "fix forward" or "restore a new cluster".
9. Staging cluster upgraded first, on the same version path.

A tool for part of it

Shoalcheck is a read-only Kubernetes audit tool, a CLI and a desktop app for macOS, Windows and Linux, and it installs nothing in the cluster. shoalcheck preflight upgrade <version> answers one question: what stops the way to that version. It compares the group/versions the cluster still serves with a removal schedule taken from the upstream deprecation guide, and it reports the node pools that cannot roll because a drain stops on them. That covers sections 1 and 2 above, not skew or add-ons. Preflight is free on every plan: shoalcheck.com, downloads on GitHub releases.

Closing

Most failed managed upgrades are not Kubernetes bugs. They are a removed API that nobody grepped for, a PDB that can never be satisfied, a node pool that cannot get surge capacity, or an add-on three versions behind. Each of those is visible in the cluster before the upgrade starts.