DEV Community

Pradeep Kandepaneni
Pradeep Kandepaneni

Posted on

What Actually Breaks During an Amazon EKS Upgrade: A Failure-First Upgrade Strategy

#kubernetes #debugging #eks #webpack

Initial published - https://builder.aws.com/content/3JAGxLt39zk79cnPoXNrhQkXxUU/what-actually-breaks-during-an-amazon-eks-upgrade-a-failure-first-upgrade-strategy

An Amazon EKS upgrade is not merely a control-plane version change.
It is a compatibility migration across Kubernetes APIs, admission webhooks, controllers, networking components, compute capacity, and applications.
The compatibility graph
Kubernetes API
|-- Admission webhooks and CRDs
|-- Controllers and operators
|-- CoreDNS, kube-proxy, VPC CNI
|-- EC2 nodes and Fargate workloads
`-- Application delivery and observability

A healthy API server does not prove that the entire graph is compatible.

  1. Deprecated APIs can fail on the next deployment
    Existing workloads may remain running even when a deployment manifest uses an API version that is no longer served. The failure appears when GitOps or CI/CD next tries to create or update the object. Inventory Helm charts, rendered manifests, operators, CRDs, and disaster-recovery scripts. Use EKS upgrade insights, but also scan repositories and exercise deployment workflows against the target version.

  2. Admission webhooks are production dependencies
    A webhook that cannot process requests from the upgraded API server can block Pod creation or resource updates. Inventory ValidatingWebhookConfiguration and MutatingWebhookConfiguration, their supported versions, service endpoints, and failurePolicy. Do not switch security-sensitive webhooks to Ignore merely to make an upgrade pass.

  3. Version skew is temporary, not a target state
    Upgrade the control plane and then bring nodes, kube-proxy, add-ons, and controllers to compatible versions. Document the supported skew for the specific source and target versions. Set a deadline for closing the skew window and verify it automatically.

  4. Node drains test application architecture
    A node upgrade exercises PodDisruptionBudgets, replicas, topology spread, persistent volume attachment, termination grace periods, and startup behavior. Rehearse voluntary disruption in a nonproduction environment. A stalled drain can indicate a correct availability safeguard rather than a faulty upgrade.

  5. Replacement capacity must be provisionable
    CPU utilization is not a reliable measure of upgrade headroom. Validate EC2 quotas, subnet IP addresses, instance availability, node-group limits, Pod density, and placement constraints. Replacement nodes and rolling-deployment surge Pods may need capacity before old infrastructure can be removed.

  6. Add-ons can fail independently
    Review compatibility and rollout order for Amazon VPC CNI, CoreDNS, kube-proxy, AWS Load Balancer Controller, CSI drivers, KEDA, cert-manager, and other operators. Validate real network paths: Pod-to-Pod, Service routing, DNS, ingress, and outbound dependency access. An upgraded cluster version alone is not evidence of network health.

  7. Observability must survive the maintenance event
    Node replacement can disrupt log agents, metrics exporters, and OpenTelemetry collectors. Preserve critical signals during the change and correlate node churn with application error rate, latency, Pending Pods, DNS failures, admission latency, and controller reconciliation errors.

  8. Autoscaling can amplify transient pressure
    Draining nodes moves Pods, creates image pulls, and increases startup demand. HPA may request more replicas while node autoscaling is still acquiring capacity. Review maxSurge, HPA limits, Karpenter constraints, consolidation policy, and quota headroom together rather than independently.

  9. Account for every compute model
    Managed node groups, self-managed nodes, Karpenter-managed nodes, and Fargate have different lifecycle procedures. Inventory all compute types and verify the current AWS guidance for refreshing each after the control-plane change.

  10. Rollback is not a cluster snapshot
    Even where EKS control-plane rollback is supported, it does not automatically restore application state, database migrations, controllers, or every add-on version. Confirm eligibility and time limits from current AWS documentation before relying on rollback. Prepare add-on compatibility and recovery procedures independently.

A failure-first execution sequence

  1. Inventory APIs, nodes, webhooks, add-ons, operators, and compute types.
  2. Validate compatibility against the target release and rehearse deployment operations.
  3. Confirm replacement capacity and disruption tolerance.
  4. Define stop conditions from application SLIs and platform metrics.
  5. Upgrade a lower-risk failure domain first.
  6. Verify scheduling, networking, controller reconciliation, and real transactions.
  7. Expand to the next failure domain only after evidence supports continuation.
  8. Close version skew and observe the system before declaring completion.

Stop conditions matter more than a green upgrade command

Stop expansion if critical synthetic transactions fail, sustained error rates exceed the change threshold, DNS or CNI errors increase, webhook requests fail, or critical workloads remain unschedulable. Establish thresholds before the maintenance window, not during the incident.

Final lesson

An EKS upgrade is a controlled compatibility migration across a distributed platform. The right question is not whether the API server can reach the next minor version. It is whether every dependent controller, workload, network path, and operational procedure continues to work after the change. Upgrade in bounded failure domains and require evidence before increasing blast radius.

References

  • Amazon EKS: Update an existing cluster
  • Amazon EKS: Cluster insights
  • Kubernetes version skew policy
  • Kubernetes API deprecation guide
  • Kubernetes disruptions

Top comments (0)