DEV Community

Evgeny Anikiev
Evgeny Anikiev

Posted on Originally published at Medium

EKS 1.37 Upgrade Checklist: What Breaks and How to Test It

#kubernetes #aws #devops #sre

Seven changes that break Amazon EKS clusters going from 1.36 to 1.37, with a test command for each.

By Evgeny Anikiev, AWS-certified DevOps engineer with 10+ years running build automation, CI/CD, and Kubernetes in production. Last verified: October 22, 2026, against the Amazon EKS 1.37 documentation.

TL;DR: Amazon EKS has supported Kubernetes 1.37 since October 2, 2026. Most of the release is additive, but seven changes can break running workloads. Test each one with the command in its section, rehearse on a staging cluster, then upgrade the control plane, add-ons, and nodes in that order.

The seven changes:

  1. SELinux mount labeling is on by default.
  2. Static pods can no longer reference Secrets or ConfigMaps.
  3. The kubelet refuses to start with deprecated cAdvisor flags.
  4. eventRecordQPS: 0 now means "no limit".
  5. EKS Auto Mode defaults new NodePools to Balanced consolidation.
  6. HPA scale-to-zero is enabled by default.
  7. kube-proxy IPVS mode is deprecated.

What is new in Kubernetes 1.37 on EKS?

Kubernetes v1.37 ("Garhwal") shipped upstream on August 26, 2026, with 67 enhancements: 16 graduated to Stable, 23 to Beta, 27 new Alpha features, and 1 deprecation/removal. Amazon EKS and EKS Distro followed on October 2, 2026, in all commercial and GovCloud (US) regions.

Headline items for EKS users:

  • Metrics API GA: metrics.k8s.io/v1.
  • DRA device taints and tolerations GA. EKS does not install a DRA driver for you.
  • HPA configurable tolerance GA: per-HPA tolerance for scale-up and scale-down.
  • HPA scale-to-zero Beta, enabled by default (HPAScaleToZero).
  • Pod Certificates and ClusterTrustBundles GA.
  • StatefulSet maxUnavailable re-enabled by default after a bug held it back in 1.36.

None of those break you on their own. The list below is where upgrades actually fail.

The EKS 1.37 breaking-change checklist

1. SELinux mount labeling is on by default (SELinuxMount)

What changed: When a Pod sets seLinuxOptions and its CSI driver advertises seLinuxMount: true, the kubelet now mounts the volume with -o context=<label> instead of recursively relabeling every file. It is faster, but if two Pods share a volume with different SELinux labels, change policies, or privilege levels, one of them can sit in ContainerCreating indefinitely.

Who is affected: Mostly Bottlerocket nodes (SELinux enforcing) and anyone setting seLinuxOptions explicitly.

How to test:

# Which CSI drivers opt into SELinux mount labeling?
kubectl get csidriver -o custom-columns=NAME:.metadata.name,SELINUXMOUNT:.spec.seLinuxMount

# Which Pods set SELinux options?
kubectl get pods -A -o json | jq -r '
  .items[] | select(.spec.securityContext.seLinuxOptions != null
    or any(.spec.containers[]; .securityContext.seLinuxOptions != null))
  | "\(.metadata.namespace)/\(.metadata.name)"'
Enter fullscreen mode Exit fullscreen mode

Fix: For workloads that share volumes across differently labeled Pods, opt out per Pod:

spec:
  securityContext:
    seLinuxChangePolicy: Recursive
Enter fullscreen mode Exit fullscreen mode

2. Static pods can no longer reference Secrets or ConfigMaps

What changed: Static pods referencing API objects (secretRef, configMapRef, and similar) are now strictly prohibited, and the PreventStaticPodAPIReferences feature gate that let you opt out has been removed.

Who is affected: Self-managed node groups with custom static pod manifests in /etc/kubernetes/manifests (log shippers, security agents, node-local proxies). Managed node groups and Auto Mode rarely use static pods.

How to test:

# On a node (SSM session), look for API references in static pod manifests
sudo grep -rnE 'secretRef|configMapRef|secretKeyRef|configMapKeyRef|projected' \
  /etc/kubernetes/manifests/ 2>/dev/null
Enter fullscreen mode Exit fullscreen mode

Fix: Convert those static pods to DaemonSets, or render the values into files on the host at bootstrap time.

3. Kubelet refuses to start with deprecated cAdvisor flags

What changed: The kubelet no longer starts if a deprecated cAdvisor flag is set (except --housekeeping-interval). It also drops three metrics: container_cpu_load_average_10s, container_cpu_load_d_average_10s, and container_tasks_state.

Who is affected: Custom AMIs, custom launch templates with --kubelet-extra-args, or Karpenter kubelet blocks that inherited old flags. EKS-optimized AMIs do not set them.

How to test:

# Inspect the live kubelet command line and config on a node
ps -o args= -C kubelet | tr ' ' '\n' | grep -E '^--'

# Grep dashboards/alerts for the removed metrics
grep -rnE 'container_cpu_load_average_10s|container_cpu_load_d_average_10s|container_tasks_state' \
  ./dashboards ./alerts ./prometheus-rules
Enter fullscreen mode Exit fullscreen mode

Fix: Remove deprecated flags from user data and launch templates. Rewrite dashboards and alerts that use the dropped metrics.

4. eventRecordQPS: 0 now means "no limit"

What changed: Before 1.37, eventRecordQPS: 0 meant the default of 5 events/second. Now it means unlimited, so a noisy CrashLoopBackOff node can flood the API server with events.

How to test:

kubectl get --raw "/api/v1/nodes/<node-name>/proxy/configz" | jq '.kubeletconfig.eventRecordQPS'
Enter fullscreen mode Exit fullscreen mode

Fix: If you set 0 explicitly, change it to 5 to keep the old behavior. EKS-optimized AMIs do not set this field.

5. EKS Auto Mode: default consolidation policy is now Balanced

What changed: New Auto Mode NodePools that don't set consolidationPolicy default to Balanced instead of WhenEmptyOrUnderutilized. The built-in general-purpose and system NodePools also use Balanced, and you cannot change that on them.

Why it matters: Bin-packing behavior and node churn change. Cost dashboards may move; PodDisruptionBudget pressure patterns change.

How to test:

kubectl get nodepools -o custom-columns=NAME:.metadata.name,POLICY:.spec.disruption.consolidationPolicy
Enter fullscreen mode Exit fullscreen mode

Any row showing <none> will pick up the new default.

Fix: Pin the old behavior explicitly on your own NodePools and target them from workloads that need it:

spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
Enter fullscreen mode Exit fullscreen mode

6. HPA scale-to-zero is enabled by default

What changed: HPAScaleToZero is Beta and on by default. An HPA with minReplicas: 0 on Object or External metrics can now drive a Deployment to zero replicas.

Who is affected: Teams that templated minReplicas: 0 by accident (Helm defaults, copy-paste) where it was previously rejected or ignored. EKS does not install a metrics adapter, so this only applies if you run one (KEDA, Prometheus Adapter, CloudWatch adapter).

How to test:

kubectl get hpa -A -o json | jq -r '
  .items[] | select(.spec.minReplicas == 0)
  | "\(.metadata.namespace)/\(.metadata.name)"'
Enter fullscreen mode Exit fullscreen mode

Fix: Set minReplicas: 1 anywhere zero is not intentional.

7. kube-proxy IPVS mode is deprecated

What changed: kube-proxy in ipvs mode now logs a deprecation warning on startup. Upstream plans to disable it by default around v1.40 and remove it around v1.43. Nothing breaks in 1.37, but this is the release to plan your move to nftables (or iptables), which now talks to the kernel through netlink instead of the nft CLI.

How to test:

kubectl -n kube-system get cm kube-proxy-config -o yaml | grep -i mode
Enter fullscreen mode Exit fullscreen mode

Also on the radar

  • cgroup v1: failCgroupV1 has defaulted to true since 1.35. The temporary override still works in 1.37, but removal is coming. Move any remaining cgroup v1 nodes (old AL2 custom AMIs) to AL2023 or Bottlerocket.
  • kube-dns: No new packages expected after v1.40. EKS uses CoreDNS by default, so check that you are not running a legacy kube-dns.
  • kubectl run -f: The --filename/-f flag is deprecated. Fix scripts that use it.
  • Kubelet logs its full effective config at startup. Useful for verifying the items above, but check that your log pipeline does not alert on it.

How to test an EKS 1.37 upgrade safely

Step 1: Run EKS Upgrade Insights 

aws eks list-insights --cluster-name my-cluster \
  --filter '{"categories":["UPGRADE_READINESS"]}'

aws eks describe-insight --cluster-name my-cluster --id <insight-id>
Enter fullscreen mode Exit fullscreen mode

Insights flag deprecated API usage seen in audit logs and add-on/version-skew problems. Treat any ERROR as a blocker.

Step 2: Scan manifests for deprecated APIs 

# Live cluster
kubent

# Helm charts and GitOps repos
pluto detect-files -d ./k8s --target-versions k8s=v1.37.0
pluto detect-helm -o wide --target-versions k8s=v1.37.0
Enter fullscreen mode Exit fullscreen mode

Step 3: Check add-on compatibility 

aws eks describe-addon-versions --kubernetes-version 1.37 \
  --query 'addons[].{name:addonName,default:addonVersions[?compatibilities[?defaultVersion==`true`]]|[0].addonVersion}' \
  --output table
Enter fullscreen mode Exit fullscreen mode

Also check self-managed controllers: AWS Load Balancer Controller, Karpenter, cert-manager, external-dns, Istio/Cilium, the EBS/EFS CSI drivers, and your service mesh. Each publishes a supported-Kubernetes matrix; confirm 1.37 is on it.

Step 4: Rehearse on a staging clone

Create a staging cluster from the same IaC at 1.36, deploy production manifests, then upgrade it:

eksctl upgrade cluster --name staging --version 1.37 --approve
# or Terraform: bump cluster_version = "1.37" and apply
Enter fullscreen mode Exit fullscreen mode

Then run the seven checks above, your smoke tests, and a synthetic load test. Watch for Pods stuck in ContainerCreating (SELinux), nodes failing to join (kubelet flags), and event volume spikes (eventRecordQPS).

Step 5: Upgrade production in the right order

  1. Control plane: one minor version at a time (1.36 → 1.37 only).
  2. EKS add-ons: VPC CNI, CoreDNS, kube-proxy, EBS CSI, Pod Identity Agent.
  3. Data plane: managed node groups (rolling), Karpenter (drift), or Auto Mode (automatic).
  4. Self-managed controllers.

Step 6: Post-upgrade verification 

kubectl get nodes -o wide                        # all nodes Ready on v1.37
kubectl get pods -A | grep -vE 'Running|Completed'
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get apiservices | grep -v True          # unhealthy aggregated APIs
Enter fullscreen mode Exit fullscreen mode

EKS 1.37 upgrade checklist (copy-paste)

  • [ ] EKS Upgrade Insights: zero ERROR findings
  • [ ] kubent / pluto clean against v1.37
  • [ ] All EKS add-ons on a 1.37-compatible version
  • [ ] Third-party controllers confirm 1.37 support
  • [ ] SELinux: CSI drivers and seLinuxOptions Pods audited; seLinuxChangePolicy: Recursive where needed
  • [ ] Static pods: no Secret/ConfigMap references
  • [ ] Kubelet: no deprecated cAdvisor flags; dashboards don't use dropped metrics
  • [ ] eventRecordQPS: 0 replaced with 5 where set
  • [ ] Auto Mode NodePools: consolidationPolicy set explicitly
  • [ ] HPAs: no accidental minReplicas: 0
  • [ ] kube-proxy: IPVS migration plan logged
  • [ ] No cgroup v1 nodes left
  • [ ] Staging upgrade rehearsed, smoke + load tests passed
  • [ ] Rollback plan: blue/green node groups ready (control plane cannot be downgraded)

FAQ

When did Amazon EKS release Kubernetes 1.37?

EKS and EKS Distro added Kubernetes 1.37 on October 2, 2026, about five weeks after the upstream release on August 26, 2026.

Can I downgrade an EKS cluster from 1.37 to 1.36?

No. EKS control plane upgrades are one-way. Your rollback options are at the node level (keep a 1.36 node group until 1.37 nodes are proven) or a blue/green cluster migration.

Can I skip from EKS 1.35 straight to 1.37?

No. The control plane upgrades one minor version at a time: 1.35 → 1.36 → 1.37. Nodes may lag the control plane within the supported skew, but plan for both hops.

What is the biggest risk when upgrading to EKS 1.37?

For most clusters it is SELinux mount labeling on Bottlerocket nodes, followed by custom kubelet configuration (deprecated cAdvisor flags and eventRecordQPS: 0). Auto Mode users should check the Balanced consolidation default.

Does EKS 1.37 remove any APIs?

The upstream release lists one deprecation/removal. The practical removals are behavioral: the PreventStaticPodAPIReferences gate is gone and deprecated kubelet cAdvisor flags now stop the kubelet from starting. Still run kubent or pluto to catch older deprecated APIs you never migrated.

How long is EKS 1.37 supported?

EKS gives each version 14 months of standard support followed by 12 months of extended support. Check the official EKS Kubernetes version lifecycle page for exact dates.

Related reading

Sources

Planning an EKS upgrade and want a second pair of eyes? I help teams run zero-downtime Kubernetes upgrades on AWS at DevOps Cloud Consult.

Top comments (1)

Collapse
 
kashif_manzer profile image
Kashif Manzer •

The connection between item 3 and the startup-config logging is worth calling out: since 1.37 logs the kubelet's full effective config at startup, you can verify deprecated cAdvisor flags were actually removed by grepping node logs before rolling the rest of the node groups. And for item 6, one gotcha: pluto or kubent won't catch accidental minReplicas 0 on HPAs, so a quick jq across the cluster is the real check there. Good checklist, the test command for each item is the part that makes this useful.