DEV Community

Madialagan Ajith kumara
Madialagan Ajith kumara

Posted on

How I Upgrade Amazon EKS in Production Without Downtime (Step-by-Step Runbook)

I run production workloads on Amazon EKS. EKS versions reach end of standard support roughly every 14 months, so upgrades aren't optional. Here's the exact runbook I follow.

Step 1: Check upgrade readiness

aws eks describe-cluster --name my-cluster --query "cluster.version"
aws eks list-insights --cluster-name my-cluster
Enter fullscreen mode Exit fullscreen mode

EKS Upgrade Insights flags deprecated APIs your workloads still use. Fix every "ERROR" before you move on.

Step 2: Scan your manifests for deprecated APIs

pluto detect-helm -o wide
pluto detect-files -d ./k8s-manifests
Enter fullscreen mode Exit fullscreen mode

Step 3: Protect your workloads

Make sure critical apps have at least 2 replicas and a PodDisruptionBudget:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-app
Enter fullscreen mode Exit fullscreen mode

Step 4: Upgrade the control plane (one minor version at a time)

eksctl upgrade cluster --name my-cluster --version 1.XX --approve
Enter fullscreen mode Exit fullscreen mode

Step 5: Upgrade managed add-ons

aws eks describe-addon-versions --addon-name vpc-cni --kubernetes-version 1.XX
aws eks update-addon --cluster-name my-cluster --addon-name vpc-cni --addon-version <version>
Enter fullscreen mode Exit fullscreen mode

Repeat for coredns, kube-proxy and aws-ebs-csi-driver.

Step 6: Upgrade node groups

eksctl upgrade nodegroup --name ng-1 --cluster my-cluster --kubernetes-version 1.XX
Enter fullscreen mode Exit fullscreen mode

Managed node groups drain and replace nodes one by one, respecting your PDBs.

Step 7: Verify

kubectl get nodes
kubectl get pods -A | grep -v Running
Enter fullscreen mode Exit fullscreen mode

Lessons learned

  1. Never skip Upgrade Insights
  2. PDBs + 2 replicas = zero downtime
  3. Upgrade add-ons right after the control plane
  4. Do it in a non-prod cluster first

What's your biggest EKS upgrade pain? Let me know in the comments.

Top comments (0)