DEV Community: Illia Bilous

Kubernetes upgrade preflight checklist for AKS, EKS and GKE

Illia Bilous — Mon, 26 Oct 2026 21:48:16 +0000

Managed Kubernetes makes the control-plane upgrade a button. The button does not tell you what will break in the workloads, and it does not tell you that the node-pool roll will stall at the first node with a bad PodDisruptionBudget. This is the checklist to run on a cluster (call it prod-eu-1) before you press it. The first four sections apply everywhere; the last covers what differs on AKS, EKS and GKE.

0. Read the release notes and pick a target

Upgrade one minor version at a time. Kubernetes does not support skipping minors on the control plane, and the managed providers enforce this in different ways. Read the upstream changelog for the target version, specifically the "Urgent upgrade notes" and "Deprecation" sections, and your provider's own release notes for that version.

1. APIs removed in the target version

Kubernetes removes beta API versions on a published schedule. The authoritative list is the upstream deprecated API migration guide at kubernetes.io. Some well-known removals:

1.22: extensions/v1beta1 and networking.k8s.io/v1beta1 Ingress, apiextensions.k8s.io/v1beta1 CRDs, admissionregistration.k8s.io/v1beta1 webhooks.
1.25: batch/v1beta1 CronJob, policy/v1beta1 PodDisruptionBudget and PodSecurityPolicy, autoscaling/v2beta1 HorizontalPodAutoscaler, discovery.k8s.io/v1beta1 EndpointSlice.
1.26: autoscaling/v2beta2 HorizontalPodAutoscaler, flowcontrol.apiserver.k8s.io/v1beta1.
1.27: storage.k8s.io/v1beta1 CSIStorageCapacity.
1.29: flowcontrol.apiserver.k8s.io/v1beta2.
1.32: flowcontrol.apiserver.k8s.io/v1beta3.

The list is illustrative; check the guide for your exact target.

Find what still uses a removed API

There are three places a removed API can hide, and none of them alone is sufficient.

Live requests. The API server counts calls to deprecated APIs. This is the best signal that something is actively using one:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

Each series carries group, version, resource and removed_release labels. A non-zero series with removed_release="1.32" means a client called that API recently. Metrics reset on API server restart and on managed services you may not be able to scrape /metrics at all, so treat an empty result as "not seen", not "not used".

What the cluster still serves. Discovery lists every group/version the server answers:

kubectl api-versions | sort
kubectl api-resources --verbs=list -o name | sort

A beta group/version in this list that your target removes is the thing to chase.

What is stored and what is applied. Objects are stored in one version and served in others, so kubectl get returns the preferred version regardless of how it was written. Manifests are where the old version lives: grep Helm charts, Kustomize overlays, Terraform and CI templates.

grep -rEn "apiVersion: (extensions/v1beta1|policy/v1beta1|batch/v1beta1|autoscaling/v2beta1|autoscaling/v2beta2)" .
helm template my-release ./chart | grep -E "^apiVersion:" | sort | uniq -c

Also check what Helm has recorded: a release whose stored manifest uses a removed API can fail to upgrade even when the live objects are fine. The helm mapkubeapis plugin exists for that.

Finally, check admission webhooks and CRD conversion webhooks. A webhook registered with a removed admissionregistration version, or a CRD that declares a conversion webhook whose Service has no ready endpoints, will cause API errors that look unrelated to the upgrade.

2. PDBs that will stall node-pool upgrades

The control plane upgrade is quick. The node-pool upgrade is where clusters get stuck. Every provider's node upgrade drains each node, and drain respects PodDisruptionBudgets. A budget with zero allowed disruptions pins its pod, and the node waits.

kubectl get pdb -A
kubectl get pdb -A -o json | jq -r '
  .items[] | select(.status.disruptionsAllowed == 0)
  | "\(.metadata.namespace)/\(.metadata.name)"'

Three causes cover most cases:

A Deployment with one replica and minAvailable: 1 (or maxUnavailable: 0). Allowed disruptions is always zero.
minAvailable equal to the replica count.
A workload that is already unhealthy, so the budget has nothing to give.

A roll with surge nodes does not fix this. Surge adds capacity so pods have somewhere to go; it does not change the budget. The pod still cannot be evicted. Providers handle the stall differently: some time out and fail the upgrade, some force the eviction after a window. See section 5, and read your provider's current documentation for the exact timeout.

Beyond budgets, a node roll also trips on:

pods with no controller, which kubectl drain refuses without --force;
pods with emptyDir data;
pods that cannot reschedule: node selectors, required anti-affinity, taints, zonal volumes, or no spare capacity once a node is gone.

The same checks apply as for a single drain; why kubectl drain hangs covers each. The difference for an upgrade is that you need the answer for every node in the pool, not one.

Also confirm the pool can absorb a surge node: VM quota for the family and free IP addresses in the subnet. A surge that cannot get either fails for reasons unrelated to Kubernetes.

3. Version skew

The upstream version skew policy sets the limits. Verify the current numbers against it, since they have changed over time:

The kubelet may not be newer than the API server. It may be older, by up to three minor versions on recent releases (it was two on older ones).
kubectl is supported within one minor version of the API server, in either direction.
Controller-manager, scheduler and cloud-controller-manager must not be newer than the API server.

On managed services the control plane moves first, so the practical rule is: after the control plane upgrade, the nodes will be behind, and you want that gap to stay inside the policy while you work through the pools. If your nodes are already two minors behind, upgrading the control plane another minor puts them at three, and then you have no room. Upgrade node pools soon after the control plane rather than letting them lag.

Check the gap:

kubectl get nodes -o custom-columns='NODE:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion'
kubectl version

Clients matter too: CI runners with an old kubectl, and operators built against client libraries that only speak a removed API version, break at the API server upgrade, not the node upgrade.

4. Add-ons

Add-ons are what actually break. Each one has its own compatibility matrix with Kubernetes versions, and most are not upgraded automatically.

Make a list:

kubectl get deploy,ds -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image'

Then, for each one, look up whether the target Kubernetes version is supported by the add-on version you run. Typical entries:

CNI plugin and CSI drivers (storage drivers are the ones that hurt most when they are wrong).
CoreDNS and kube-proxy, where the provider manages versions.
Cluster autoscaler (its minor version should match the cluster's) and ingress controllers.
cert-manager, service meshes, policy engines and any operator that ships CRDs.
Anything with a webhook. A webhook with failurePolicy: Fail and no backing pod can block all writes of the resource it covers.

5. What to check per provider

These are the managed-service specifics that have been stable for a while. Confirm details against the current docs, because defaults and limits get changed.

EKS

Upgrade insights. EKS runs checks for deprecated API usage and other issues and reports them per cluster. In the console or with aws eks list-insights --cluster-name prod-eu-1.
Add-ons. Managed add-ons (VPC CNI, CoreDNS, kube-proxy, EBS CSI) have versions per Kubernetes release: aws eks describe-addon-versions --kubernetes-version <target> --addon-name <name>. Self-managed ones are your job.
Order. Control plane first, then add-ons, then node groups. A control plane upgrade cannot be rolled back.
Managed node groups. The update config has maxUnavailable (default one node at a time) or a percentage; raising it speeds up the roll but concentrates the disruption.

AKS

Available versions. az aks get-upgrades -g <rg> -n prod-eu-1 -o table. AKS does not let you skip minor versions.
Node surge. Each node pool has a max surge setting, with a default of a percentage of the pool; az aks nodepool update --max-surge 33% changes it. Surge nodes count against your VM quota and subnet addresses.
Drain behaviour. AKS drains nodes during upgrade and honours PDBs; a drain that cannot finish within the node drain timeout fails the upgrade. Check the current timeout in the docs and set it deliberately.

GKE

Release channels and maintenance windows. On a release channel the control plane upgrades on Google's schedule within your window; set exclusions for periods you cannot afford a change.
Deprecation insights. GKE reports usage of deprecated APIs for upcoming versions in the console, and can pause automatic upgrades when it sees usage.
Surge upgrades. Node pools use maxSurge and maxUnavailable; the defaults favour one surge node and no unavailable nodes. Blue-green is an alternative for pools that need to roll back quickly.
PDBs. GKE honours PodDisruptionBudgets during node upgrades for a bounded time, after which it proceeds. Do not count on that: a budget that waits an hour then gets bypassed is not a design.

6. The checklist

Target version chosen, one minor step, release notes read.
Removed APIs: metric checked, discovery checked, manifests and Helm releases grepped, webhooks checked.
PDBs: none with zero allowed disruptions on a workload you are about to roll.
Drain blockers: unmanaged pods, emptyDir, unschedulable replacements, per node.
Quota and IP headroom for surge.
Skew: kubelet gap, kubectl, CI clients.
Add-on compatibility for the target version, with an upgrade order.
Backups and a rollback plan: the control plane usually cannot be downgraded, so the plan is "fix forward" or "restore a new cluster".
Staging cluster upgraded first, on the same version path.

A tool for part of it

Shoalcheck is a read-only Kubernetes audit tool, a CLI and a desktop app for macOS, Windows and Linux, and it installs nothing in the cluster. shoalcheck preflight upgrade <version> answers one question: what stops the way to that version. It compares the group/versions the cluster still serves with a removal schedule taken from the upstream deprecation guide, and it reports the node pools that cannot roll because a drain stops on them. That covers sections 1 and 2 above, not skew or add-ons. Preflight is free on every plan: shoalcheck.com, downloads on GitHub releases.

Closing

Most failed managed upgrades are not Kubernetes bugs. They are a removed API that nobody grepped for, a PDB that can never be satisfied, a node pool that cannot get surge capacity, or an add-on three versions behind. Each of those is visible in the cluster before the upgrade starts.

kubectl drain stuck? Why drains hang on PodDisruptionBudgets, and how to check before you start

Illia Bilous — Mon, 26 Oct 2026 21:47:29 +0000

You cordon a node, run kubectl drain, and the terminal prints the same line every five seconds:

error when evicting pods/"api-7d9c6b8f5-x2k4q" -n payments (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

Nothing is broken. Drain is doing what it was told: asking the API server to evict pods one at a time and waiting. A drain hangs for a small number of reasons, and almost all of them can be read from the cluster before you start. This post goes through them with commands you can run against any cluster.

What drain actually does

kubectl drain <node> first cordons the node (marks it unschedulable), then evicts each pod through the Eviction API. Eviction is not delete. The API server checks the pod's PodDisruptionBudget (PDB) and refuses with HTTP 429 if the eviction would take the workload below its budget. kubectl keeps retrying, which is the "hang".

Drain also refuses to start on some pods, and it can succeed on eviction while the pod then has nowhere to run. Those are three different failure classes:

Eviction is refused (PDB).
Drain refuses the pod up front (unmanaged pods, local storage, DaemonSets).
Eviction works, but the replacement pod stays Pending.

1. PDB math

List every budget and look at one column:

kubectl get pdb -A

NAMESPACE   NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
payments    api    1               N/A               0                     41d
payments    web    N/A             1                 1                     41d

ALLOWED DISRUPTIONS is the number of pods the Eviction API will let go right now. If it is 0 for a workload that has a pod on your node, that pod cannot be evicted and the drain waits.

To get just the blocking budgets:

kubectl get pdb -A -o json | jq -r '
  .items[] | select(.status.disruptionsAllowed == 0)
  | "\(.metadata.namespace)/\(.metadata.name)  healthy=\(.status.currentHealthy)/\(.status.desiredHealthy) expected=\(.status.expectedPods)"'

minAvailable vs maxUnavailable

Both express the same idea from opposite sides.

minAvailable: N means at least N pods must stay healthy. Allowed disruptions = healthy pods minus N.
maxUnavailable: N means at most N pods may be unavailable. Allowed disruptions = N minus pods already unavailable.

Percentages are rounded up. With 10 replicas, minAvailable: 80% allows 2 disruptions; maxUnavailable: 25% allows 3 (2.5 rounds up).

The two behave differently when replicas change. minAvailable: 3 on a 3-replica Deployment allows zero disruptions forever; maxUnavailable: 1 keeps allowing one disruption at any size. If you want a budget that scales with the workload, prefer maxUnavailable.

A budget also counts only healthy pods. If one of three replicas is already crash-looping and the budget is minAvailable: 2, allowed disruptions is 0 even though no one touched anything. A PDB can set unhealthyPodEvictionPolicy: AlwaysAllow (beta since 1.27, GA in 1.31) so that pods which are not ready can be evicted regardless of the budget; check your cluster version before relying on it.

The single-replica Deployment with a PDB

This is the most common self-inflicted drain block:

apiVersion: apps/v1
kind: Deployment
metadata: { name: report-worker, namespace: payments }
spec:
  replicas: 1
  selector: { matchLabels: { app: report-worker } }
  template:
    metadata: { labels: { app: report-worker } }
    spec:
      containers: [{ name: app, image: registry.example/report-worker:1.4 }]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: report-worker, namespace: payments }
spec:
  minAvailable: 1
  selector: { matchLabels: { app: report-worker } }

One replica, one required. Allowed disruptions is 0, always. maxUnavailable: 0 has the same effect. Every drain that touches the node hosting this pod will hang until someone deletes the budget or scales up.

The honest options are: run at least two replicas, set maxUnavailable: 1 and accept a brief gap, or drop the PDB for a workload that cannot meet it anyway. A PDB that can never be satisfied protects nothing and blocks every node upgrade.

Also check the selector. A PDB whose selector matches no pods shows expectedPods: 0, and a selector that matches pods from several controllers produces unspecified results; the API documentation says to keep one PDB per workload.

2. Pods that drain refuses to evict

Before any eviction, kubectl drain checks each pod and aborts with a message if it finds one of these. These are flags you have to choose deliberately.

Pods without a controller

A bare pod (no ReplicaSet, StatefulSet, Job or similar owner) will not be recreated anywhere. Drain refuses unless you pass --force, and then the pod is simply gone.

Find them:

kubectl get pods -A -o json | jq -r '
  .items[] | select((.metadata.ownerReferences // []) | length == 0)
  | "\(.spec.nodeName)\t\(.metadata.namespace)/\(.metadata.name)"'

Static (mirror) pods show up here too; those are owned by the node and drain ignores them. Anything else on the list is a pod you need to decide about: recreate it as a Deployment, or accept losing it.

emptyDir and local storage

Pods with an emptyDir volume lose that data on eviction. Drain refuses unless you pass --delete-emptydir-data (older versions called it --delete-local-data). That flag is a decision, not a fix. If the emptyDir holds a cache, fine. If it holds the only copy of something, it is not.

kubectl get pods -A -o json | jq -r '
  .items[] | select(any(.spec.volumes[]?; has("emptyDir")))
  | "\(.spec.nodeName)\t\(.metadata.namespace)/\(.metadata.name)"'

A different local-storage trap: a PersistentVolume with node affinity (local PVs, or zonal disks) pins the pod to a node or zone. The pod can be evicted, but the replacement will not schedule anywhere else. That belongs in section 3.

DaemonSets

DaemonSet pods are recreated on the same node immediately, so evicting them is pointless. Drain refuses until you pass --ignore-daemonsets, which skips them. This is the one flag nearly everyone passes:

kubectl drain prod-eu-1-node-a --ignore-daemonsets --delete-emptydir-data --timeout=10m

Put --timeout on every drain you script. Without it, a PDB block waits forever.

3. Pods that cannot reschedule

Eviction succeeds, the drain moves on, and the replacement pod sits in Pending. For a single-replica workload that is an outage; for a multi-replica one it is lost capacity. Check this before you drain, because the scheduler will not tell you until it is too late.

Look at what is already pending:

kubectl get pods -A --field-selector=status.phase=Pending
kubectl describe pod -n payments <pod> | sed -n '/Events:/,$p'

The event reads like 0/6 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 3 Insufficient cpu, 1 node(s) had untolerated taint. Each clause is a reason, and you can test for each one ahead of time.

Node selectors and affinity. List what the pods on your node require:

kubectl get pods -A --field-selector spec.nodeName=prod-eu-1-node-a -o json | jq -r '
  .items[] | "\(.metadata.namespace)/\(.metadata.name)\t\(.spec.nodeSelector // {})\t\(.spec.affinity != null)"'

If a pod selects disktype=ssd and this is the only node with that label, there is nowhere for it to go. The same applies to required pod anti-affinity: a pod that must not share a node with its siblings needs one free node per replica.

Taints. Compare node taints to the tolerations of the pods you are about to move:

kubectl get nodes -o custom-columns='NODE:.metadata.name,TAINTS:.spec.taints[*].key'

Capacity. Look at what the remaining nodes can absorb:

kubectl describe nodes | grep -A6 "Allocated resources"
kubectl top nodes

Requests, not usage, decide scheduling. A node that looks idle in top can be full in Allocated resources.

Zones and volumes. A pod bound to a zonal volume can only restart in that zone. If the zone has no spare capacity, it stays Pending.

None of this replaces a real drain in a staging cluster, and the scheduler has more inputs than any manual check covers (topology spread constraints, priorities, extended resources). But the checks above catch the usual causes.

A pre-drain checklist

Run these in order:

kubectl get pdb -A and look for ALLOWED DISRUPTIONS of 0 on workloads with pods on the node.
List unmanaged pods and emptyDir pods on the node, and decide on --force and --delete-emptydir-data knowingly.
For each workload on the node, confirm it has a place to go: labels, taints, free requests, zone.
Cordon first (kubectl cordon), then drain with --ignore-daemonsets --timeout.
If it hangs, read the retry message. It names the pod and the namespace.

Doing this by hand for one node is fine. Doing it for every node in a pool before an upgrade is tedious and easy to get wrong.

Doing the same check with a tool

Shoalcheck is a read-only Kubernetes audit tool (CLI and desktop app) that installs nothing in the cluster. shoalcheck preflight drain <node> answers this one question: whether that node can be evacuated, and if not, which object stops it (a budget at zero, a pod nothing recreates, a volume pinned to the node, work with nowhere to go). It exits 0 when nothing stops the drain, 2 when something does, and 1 when RBAC refused some reads and the answer is partial. Preflight is free on every plan: shoalcheck.com, downloads on GitHub releases. Its model checks less than the scheduler does, so treat a clear result as "nothing found", not a guarantee.

Summary

A drain hangs because an eviction is refused or a replacement cannot start. A PDB with ALLOWED DISRUPTIONS of 0 is the usual cause, and the single-replica Deployment with minAvailable: 1 is the usual way to create one. Bare pods, emptyDir data and DaemonSets are things drain makes you acknowledge with flags. Unschedulable replacements are the quietest failure, so look at selectors, taints, capacity and zones before you cordon, not after.