Plan Kubernetes upgrades for AI infrastructure as a sequence: check deprecated APIs, upgrade the control plane, move node pools with surge or blue-green capacity, then move the GPU software stack together. PodDisruptionBudgets and startup and readiness probes that wait for model load keep inference serving, and checkpoints let training jobs stop cleanly. Keep change records, scan results, and audit logs as evidence. Platforms such as Control Plane run the Kubernetes underneath for you, so workload owners do not run cluster upgrades on its managed regions.
Disclosure: Control Plane publishes this guide.
TL;DR
Upgrade the control plane one minor version at a time, before the nodes.
- Scan manifests, Helm charts, and live objects for deprecated APIs with tools like pluto or kubent before you start.
- Use surge or blue-green node pools so new capacity is serving before old nodes drain.
- PodDisruptionBudgets plus startup and readiness probes protect inference from early eviction and early traffic.
- Cordon old GPU pools and checkpoint training jobs so they stop at a checkpoint.
- The NVIDIA driver, CUDA version, GPU Operator, device plugin, and container toolkit move as one tested set.
- Compliance comes from policy as code plus evidence captured at each step: change records, vulnerability scans, audit logs, and sign-offs.
- Control Plane runs the Kubernetes underneath its managed regions, so workload owners do not run cluster upgrades there, and Control Plane’s Managed Kubernetes (MK8s) upgrades the clusters it provisions to the version you select.
How do I plan zero downtime Kubernetes upgrades and migrations while keeping AI infrastructure compliant and stable?
Treat the upgrade as a fixed sequence with workload protections and evidence at every step.
Kubernetes releases about three minor versions per year, and each receives roughly 14 months of patch support from the upstream project. Falling two versions behind compresses your upgrade runway and forces you to chain upgrades under pressure. A lower-risk pattern is to stay within one minor version of the latest release and upgrade on a regular quarterly or semi-annual cadence.
Every upgrade follows the same high-level sequence. First, scan manifests, Helm charts, and live cluster objects for APIs that are deprecated or removed in the target version, and confirm that add-ons such as the GPU Operator, CNI, CSI driver, and ingress controller support the target version. Tools such as pluto and kubent automate the API scan and produce a report you can attach to your change record. Second, rehearse the full upgrade in a staging cluster with production-like workloads, including GPU jobs. Third, upgrade the control plane. EKS and regional GKE clusters keep the API available during this step, while a zonal GKE cluster’s API is unavailable until the upgrade finishes. Minor-version control plane upgrades generally cannot be reversed on EKS, GKE, or AKS, so the rollback plan covers nodes and workloads. Fourth, move worker nodes with surge capacity or a blue-green node pool so the cluster stays above the replica counts your PodDisruptionBudgets require. Fifth, upgrade the GPU software stack as one tested set. Sixth, validate inference latency, training throughput, and autoscaling on the new nodes before removing the old pool.
AI workloads add two complications. Large model servers can take minutes to load weights into GPU memory, so a probe that reports healthy before the model is loaded sends requests to a pod that cannot answer them. Long-running training jobs cannot simply be evicted. They need to checkpoint, so you cordon old GPU nodes and let running jobs reach a checkpoint before draining.
Compliance runs through every step. A change record opens before the upgrade begins and closes with a sign-off after the post-upgrade review. Policy engines such as Kyverno or OPA Gatekeeper block pods that use deprecated APIs, unsigned images, or missing disruption budgets. Audit logs shipped outside the cluster record who did what and when.
How do the managed providers handle upgrades?
Each managed Kubernetes provider has its own support windows, upgrade mechanics, and automatic-upgrade behavior.
| Provider | Standard support per minor version | Extended support | Control plane upgrade | Node upgrade options | Automatic upgrades |
|---|---|---|---|---|---|
| Amazon EKS | About 14 months from release | 12 additional months of extended support, billed per cluster-hour | AWS upgrades the control plane in place with the API available; you start the upgrade | Managed node groups launch replacement nodes before draining, controlled by maxUnavailable (minimum 1); self-managed node groups need manual drain and replace; Karpenter replaces nodes when a new AMI is selected | The cluster upgrade policy (EXTENDED by default, or STANDARD) sets when AWS auto-upgrades the control plane: at the end of extended support, or at the end of standard support if extended support is off; nodes are not auto-upgraded except in EKS Auto Mode |
| Google Kubernetes Engine (GKE) | About 14 months per minor version; the release channel (Rapid, Regular, Stable, Extended) sets when upgrades arrive | Available through the Extended channel, which keeps a minor version supported for up to 24 months, with an additional charge during the extended period | Automatic within the release channel or started manually; regional clusters keep the API available, zonal clusters do not during the upgrade | Surge upgrades (maxSurge and maxUnavailable per node pool) and blue-green node pool upgrades are both built in | On by default in release channels; maintenance windows and exclusions control timing |
| Azure Kubernetes Service (AKS) | About 12 months per minor version | Long-term support (LTS) on the Premium tier adds a second year, for about 24 months in total; all supported versions are eligible | Azure upgrades the control plane in place; you start or schedule the upgrade | Max surge and max unavailable settings control capacity during rolling upgrades; a native blue-green node pool upgrade strategy is available in preview | Planned maintenance windows, cluster auto-upgrade channels (none, patch, stable, rapid), and a separate node OS auto-upgrade channel |
| Self-managed (kubeadm) | About 14 months of upstream support per minor version | None; you upgrade or run unpatched | You take an etcd snapshot, then run kubeadm upgrade on each control-plane node in turn | Manual cordon, drain, and upgrade of each worker node; surge capacity is your responsibility | None |
GKE offers blue-green node pool upgrades as a built-in strategy, and AKS offers one in preview. On EKS, the same outcome comes from creating a new node group and migrating workloads. Self-managed clusters need the most manual work and carry the most operational risk.
Which parts of the GPU stack have to move together?
Upgrading one GPU component without the others can cause failures ranging from pod startup errors to degraded or failed inference.
| Component | What it does | Tied to | Upgrade risk | How to test before production |
|---|---|---|---|---|
| NVIDIA driver | Provides the kernel module that lets the OS talk to the GPU hardware | Kernel version on the node OS; must support the CUDA version used in containers | A driver too old for the container’s CUDA runtime causes workload crashes | Run a short GPU burn-in job and a known-good inference request on a canary node |
| CUDA version in the container image | Supplies the GPU computing libraries that frameworks call | Must be supported by the host NVIDIA driver (forward compatibility has limits) | A mismatch causes “CUDA driver version is insufficient” errors at startup | Build a test image with the target CUDA version and run unit tests on a canary node |
| NVIDIA GPU Operator | Automates driver, toolkit, and device plugin lifecycle on Kubernetes nodes | Kubernetes version, driver version, and container toolkit version | A new Operator release may change default driver versions or resource naming | Deploy the new Operator version in staging and confirm GPU resource requests still schedule |
| NVIDIA device plugin | Registers GPU resources with the kubelet so pods can request them | GPU Operator version and the Kubernetes device plugin API | A mismatch makes GPUs disappear from node capacity | Check node allocatable resources after the plugin restarts |
| NVIDIA Container Toolkit | Hooks into the container runtime to expose GPU devices inside containers | Driver version and container runtime (containerd, CRI-O) | Toolkit and runtime mismatches start containers without GPU access | Run nvidia-smi inside a test pod and confirm every expected GPU is listed |
| Node OS image and kernel | Provides the kernel that loads the NVIDIA driver module | Driver version (drivers are built against a specific kernel ABI) | A kernel update can break an existing driver module | Boot a node with the target OS image and confirm the driver loads cleanly |
| Inference server (for example vLLM or Triton) | Serves model predictions over HTTP or gRPC | CUDA version, framework version, and model format | A new server version may change default batch sizes, quantization support, or API surface | Run a latency and throughput benchmark with production models |
| Training framework (for example PyTorch) | Executes model training and fine-tuning | CUDA version, NCCL version, and driver capabilities | Framework updates can change numerical behavior or distributed training protocols | Run a short training loop and compare loss curves against a known baseline |
Pin a tested combination of driver, CUDA, Operator, and framework versions in staging, promote that exact combination to production, and block uncontrolled auto-upgrades of any single component.
What is the step-by-step upgrade plan?
Each step produces specific evidence that feeds your compliance record and post-upgrade review.
| Step | What to do | Stop condition | Evidence to keep |
|---|---|---|---|
| 1. Inventory and deprecated API scan | Document the current Kubernetes version, GPU stack versions, replica counts, and PDB settings. Run pluto or kubent against manifests, Helm releases, and live objects. Confirm the GPU Operator, CNI, CSI, and ingress versions support the target version, and upgrade them first if they do not. | Any deprecated or removed API in a production manifest is remediated before proceeding. | Scan report, remediation commits, and add-on compatibility notes in version control. |
| 2. Staging rehearsal | Run the full upgrade sequence in a staging cluster with representative GPU workloads, inference traffic, and training jobs. | The rehearsal completes without workload errors, GPU scheduling failures, or PDB-blocked drains. | Rehearsal runbook, test results, dashboard snapshots, and sign-off from platform and application owners. |
| 3. Control plane upgrade | On self-managed clusters, take an etcd snapshot and store it off the node first. Upgrade the control plane one minor version using the provider’s process or kubeadm, then upgrade add-ons such as CoreDNS, kube-proxy, and the CNI to matching versions. | API server health checks pass, system pods run, and no new deprecation warnings appear. | Provider upgrade event log, etcd snapshot record where applicable, health check output, and change record timestamp. |
| 4. Node pools with surge or blue-green | Create a new node pool on the target version (blue-green), or upgrade the existing pool in place with surge settings that add a node before removing one: maxSurge 1 and maxUnavailable 0 on GKE, a max surge value on AKS. EKS managed node groups add replacement nodes automatically and require maxUnavailable of at least 1. On GPU pools, surge needs spare GPU quota. | New nodes join the cluster, pass readiness checks, and accept scheduled pods. | Node pool event log, node readiness status, and capacity verification. |
| 5. GPU stack upgrade | Roll the target driver and container toolkit to the new node pool through the GPU Operator (per-pool driver versions use the NVIDIADriver resource) or the provider’s GPU driver setting. Validate that GPUs appear in node allocatable capacity. | nvidia-smi runs inside a test pod on every new GPU node, lists every expected GPU, and reports the expected driver version, and a CUDA sample or framework check in the same image confirms the runtime works. | Operator or driver rollout log, nvidia-smi output, and device plugin resource count. |
| 6. Inference cutover with readiness probes | Move inference workloads to the new node pool. Readiness probes pass only after model weights are loaded. | Latency and error rates on the new nodes match or beat the old nodes for a defined soak period. | Latency and error-rate dashboards, probe configuration, and PDB status. |
| 7. Training jobs stop at a checkpoint | Cordon old GPU nodes, or taint them NoSchedule (NoExecute would evict running pods), so no new training jobs land there. Let running jobs write a checkpoint, then drain. | Every training job on the old nodes has written a checkpoint and been rescheduled or completed. | Checkpoint timestamps, job logs, and cordon or taint events. |
| 8. Policy and security scan on new nodes | Run image vulnerability scans, policy engine validation (Kyverno or OPA Gatekeeper), and Pod Security Standards checks for every namespace with workloads on the new nodes. | No critical vulnerabilities, no policy violations, and pods in each namespace meet the enforced Pod Security Standard. | Scan reports, policy audit results, and namespace Pod Security labels. |
| 9. Old pool removal | Delete the old node pool after confirming no workloads remain and no local volumes hold unmigrated data. Network volumes reattach on new nodes in the same zone. | The old pool is empty and no data depends on its local disks. | Node pool deletion event and capacity reconciliation. |
| 10. Post-upgrade review | Review the upgrade with platform and application owners, record lessons learned, update the runbook, and close the change record. | All owners sign off and the change record is closed. | Meeting notes, updated runbook, final sign-off, and closed change record. |
How do you keep inference serving and training safe during the upgrade?
Protecting live AI workloads during an upgrade uses Kubernetes disruption controls plus AI-specific steps for model loading and checkpoints.
A PodDisruptionBudget (PDB) tells the eviction API how many pods in a set may be unavailable at once. For inference, set minAvailable to the replica count that handles peak traffic. A PDB that allows zero disruptions on a single-replica deployment blocks kubectl drain indefinitely, and managed node upgrades either fail (EKS, AKS) or force eviction after one hour (GKE). Run every protected workload with at least two replicas, spread across nodes with topology spread constraints or pod anti-affinity.
Deployment surge settings apply to rollouts. During a drain, the pod is evicted first and replaced afterwards, so the PDB is what limits disruption. On EKS managed node groups, GKE, and AKS, node upgrades add a new node before an old one drains, and on GPU pools that needs spare GPU quota.
A preStop sleep of a few seconds gives load balancers and ingress controllers time to stop routing to the pod after Kubernetes removes it from the Service endpoints. Kubernetes 1.30 and later support a native preStop sleep action for images that lack a sleep binary. Set terminationGracePeriodSeconds to the preStop sleep plus the longest in-flight request, because the grace period starts before the preStop hook runs. Long streaming LLM responses can take minutes.
A health check that returns 200 as soon as the web server starts sends traffic to a pod that has not loaded its model. The readiness probe should report ready only after model weights are loaded into GPU memory, using an endpoint such as Triton’s /v2/health/ready. Add a startupProbe with a failureThreshold long enough to cover model load, so the liveness probe does not restart the pod mid-load. Run a full test inference once at startup instead of on every probe.
Pre-pull large AI container images, which can be tens of gigabytes. If the kubelet pulls the image only after the pod is scheduled, the pod sits in ContainerCreating for minutes and the readiness probe cannot pass until the pull and model load finish. Pre-pull with a DaemonSet on the new pool, bake the image into the node image, or use provider image streaming.
Training jobs need a different approach. Cordoning old GPU nodes, or tainting them NoSchedule, keeps new training jobs off nodes that are about to drain. PyTorch provides checkpoint APIs such as torch.save and torch.distributed.checkpoint, and the training loop sets the interval, so choose one that keeps lost work acceptable. A stronger pattern is to have the training code write a checkpoint when it receives SIGTERM and to set terminationGracePeriodSeconds long enough to finish that write. In most distributed jobs, evicting one worker stops the whole job.
For major migrations that jump several Kubernetes versions or change cloud providers, blue-green at the cluster level keeps the old cluster intact as a rollback target. Stand up the new cluster, validate it end to end, lower DNS TTLs in advance, replicate stateful data, shift traffic gradually with DNS or a global load balancer, and remove the old cluster only after a soak period with no issues. The broader cluster design behind these steps is covered in the Kubernetes production reference architecture, and the GPU-specific friction in why Kubernetes is hard for AI infrastructure teams. If an upgraded component misbehaves, roll back the change before continuing the rotation.
How do you stay compliant through an upgrade?
Compliance during a Kubernetes upgrade comes from controls and evidence collection built into every step.
Every upgrade should begin with a formal change request that includes the scope, risk assessment, rollback plan, and approvals. The change record stays open through the upgrade and closes only after the post-upgrade review and sign-off. Under SOC 2 or HIPAA programs, this record supports change management controls and gives auditors a timeline of who approved what and when.
Policy as code enforces guardrails automatically. Kyverno or OPA Gatekeeper can block deployments that reference deprecated APIs, use unsigned container images, lack PodDisruptionBudgets, or violate Pod Security Standards. Run the same policies in staging and production, save the policy audit results as evidence, and keep the policies in version control with pull-request approvals so changes to the guardrails are as auditable as changes to the workloads.
Scan every container image deployed to the new node pool and keep the results with the upgrade evidence. Pod Security Standards, enforced per namespace, ensure that pods in each namespace meet the baseline or restricted profile.
Enable API server audit logging before the upgrade. EKS and AKS export it only after you turn it on, and self-managed clusters need an audit policy file. Each entry records the user, verb, resource, and response code. Ship the logs to a store outside the cluster so they survive if the cluster is replaced, and use an append-only or tamper-evident store in regulated environments.
Change records support change management controls, scan results support vulnerability management, audit logs support monitoring and logging, and policy engine results support configuration management. Collect this evidence during the upgrade, when it is easy to capture. The same controls apply to running regulated apps in the cloud under SOC 2, HIPAA, and PCI DSS.
How does Control Plane handle Kubernetes upgrades?
Control Plane handles Day 2 operations for you, including autoscaling, built-in observability, and security.
On Control Plane’s managed regions on AWS, GCP, Azure, and Oracle Cloud Infrastructure, serverless, standard, stateful, and cron workloads run with no Kubernetes cluster for you to build or operate, so workload owners do not plan or run cluster upgrades there. For clusters you need to run yourself, Control Plane’s Managed Kubernetes (MK8s) provisions Kubernetes on many providers and on your own servers and handles upgrades to the Kubernetes version you select, along with autoscaling.
Standard and stateful workloads on Control Plane roll out as rolling updates, and the readiness probe you configure decides when a new replica receives traffic, so the model-load probe described earlier applies directly. A workload runs in every location of its Control Plane Global Virtual Cloud (GVC), the set of cloud locations Control Plane deploys it to, and its canonical endpoint routes clients to the nearest location where it passes its readiness probes, so a workload deployed to several locations keeps serving when one location fails.
Control Plane keeps a tamper-proof audit trail of platform actions, recording who did what and when. Workloads can attach NVIDIA GPUs, and the pricing page lists models from T4 to B200. Training jobs still need their own checkpointing, and images must still be built for the CUDA and driver versions of the GPUs they run on. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant. Day-to-day changes to model serving still follow the readiness and capacity rules covered in Day 2 operations for AI infrastructure.
FAQ
Common questions about planning zero-downtime Kubernetes upgrades for AI infrastructure.
How do I plan zero downtime Kubernetes upgrades and migrations while keeping AI infrastructure compliant and stable?
Upgrade the control plane one minor version at a time, then move node pools on surge or blue-green capacity with the GPU stack pinned as one tested set. PodDisruptionBudgets, startup and readiness probes, and checkpoints protect workloads, and change records, scans, and audit logs are the evidence. On Control Plane, the platform runs the Kubernetes underneath, so cluster upgrades come off the workload owner’s list.
Can you skip Kubernetes minor versions when upgrading?
Not for the control plane. Kubernetes upgrades the control plane one minor version at a time, so you chain upgrades with testing at each step. Upstream Kubernetes allows worker nodes to run up to three minor versions behind the control plane but never ahead of it, so node pools can move in one step after the control plane reaches the target, if your provider’s skew limit allows it.
Should I upgrade in place or use blue-green node pools?
In-place rolling upgrades with surge capacity suit routine single-version upgrades on mostly stateless workloads. Blue-green node pools or clusters are safer for catching up several versions, structural changes, or when you need a clean rollback target and compliance evidence.
How do I upgrade GPU nodes without killing training jobs?
Cordon old GPU nodes or taint them NoSchedule, let running jobs write a checkpoint, ideally on SIGTERM with a long enough grace period, then drain and restart the jobs from the checkpoint on new nodes.
How do I keep LLM inference online during node upgrades?
Run at least two replicas spread across nodes, set PodDisruptionBudgets, use a startupProbe plus a readiness probe that passes only after model weights load, pre-pull large images to new nodes, and size the termination grace period to cover the preStop sleep and the longest in-flight request.
How do I find deprecated Kubernetes APIs before upgrading?
Run pluto or kubent against your manifests, Helm releases, and live cluster objects. Both compare the API versions in use against the deprecation and removal schedule for your target version and list the resources that need updates.
What happens when an EKS, GKE, or AKS version reaches end of support?
With the default EXTENDED upgrade policy, EKS moves a cluster into paid extended support for 12 months, then automatically upgrades the control plane; with the STANDARD policy, it upgrades at the end of standard support. GKE automatically upgrades clusters according to their release channel. AKS performs a platform-initiated upgrade 60 days after a version leaves support, regardless of the auto-upgrade channel. Running an unsupported version means no security patches and growing compatibility risk.
Does Control Plane upgrade Kubernetes for me?
Yes, in both setups. On Control Plane’s managed regions, there is no Kubernetes cluster for you to operate, so cluster upgrades are not your task. For clusters provisioned through Control Plane’s Managed Kubernetes (MK8s), Control Plane handles upgrades to the Kubernetes version you select, along with autoscaling.
Talk to the Control Plane team about running AI workloads without operating cluster upgrades, or sign up free.
Originally published on the Control Plane blog.
Top comments (0)