DEV Community

Cover image for EKS cluster upgrade fail with Kubelet version of Fargate pods must be updated to match cluster version!!
Nowsath for AWS Community Builders

Posted on

3

EKS cluster upgrade fail with Kubelet version of Fargate pods must be updated to match cluster version!!

EKS Cluster - Could not update cluster version

You might encounter the following error message within the AWS EKS console when attempting to upgrade the Kubernetes version of your cluster.

Kubelet version of Fargate pods must be updated to match cluster version 1.25 before updating cluster version; Please recycle all offending pod replicas.

For instance, if you are in the process of incrementally upgrading the Kubernetes version from 1.24 to 1.27.

The upgrade from version 1.24 to 1.25 will proceed without encountering errors. However, please note that it won’t automatically update the kubelet version of the Fargate nodes/Pods until a restart is performed.

AWS EKS Console (Kubernetes version 1.25)

But when you check Fargate nodes versions,
Fargate Node details

Or when you check from terminal,



# To list all nodes
kubectl get nodes


Enter fullscreen mode Exit fullscreen mode

Fargate Nodes details — Terminal

Solutions..

If the Fargate pods’ versions are not compatible with the upgraded EKS cluster, you might need to recreate the pods using a compatible version.

  • You can do this by updating the deployment or stateful set YAML files with the correct version and applying the changes to the cluster. Kubernetes will automatically create new pods with the updated version.

  • You can get all deployment, statefulset from all namespaces and restart it as shown below.



# To list all deployments
kubectl get deployment -A

# To list all deployments
kubectl get statefulset -A

# To restart deployments
kubectl rollout restart deployment coredns -n kube-system
kubectl rollout restart deployment metrics-server -n kube-system
kubectl rollout restart deployment aws-load-balancer-controller -n kube-system
kubectl rollout restart deployment ebs-csi-controller -n kube-system

# To restart statefulset
kubectl rollout restart statefulset adot-collector -n fargate-container-insights


Enter fullscreen mode Exit fullscreen mode

Please note that this solution will work exclusively with Fargate nodes. If you are using other cluster modes, please refer to the links provided at the end of this article for alternative solutions.


Also don’t forget to update kubectl as well.



# To check kubectl version
kubectl version --short --client


Enter fullscreen mode Exit fullscreen mode

Link to update kubectl:
Installing or updating kubectl

For more details about Amazon EKS Kubernetes versions upgrade:
Amazon EKS Kubernetes versions
Updating an Amazon EKS cluster Kubernetes version


I’ll always strive to provide simple explanations. If you found this article useful, follow me for more similar articles in the future. Your support keeps me motivated to create content like this. 🤝😊

Image of AssemblyAI tool

Challenge Submission: SpeechCraft - AI-Powered Speech Analysis for Better Communication

SpeechCraft is an advanced real-time speech analytics platform that transforms spoken words into actionable insights. Using cutting-edge AI technology from AssemblyAI, it provides instant transcription while analyzing multiple dimensions of speech performance.

Read full post

Top comments (0)

Best Practices for Running  Container WordPress on AWS (ECS, EFS, RDS, ELB) using CDK cover image

Best Practices for Running Container WordPress on AWS (ECS, EFS, RDS, ELB) using CDK

This post discusses the process of migrating a growing WordPress eShop business to AWS using AWS CDK for an easily scalable, high availability architecture. The detailed structure encompasses several pillars: Compute, Storage, Database, Cache, CDN, DNS, Security, and Backup.

Read full post

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay