In the previous blog, I explained that EKS Auto mode is now supported by terraform-eks-module and illustrated how we can create new cluster with EKS Auto Mode.
In this blog, we’ll learn how to enable EKS Auto Mode on existing clusters and migrate workloads from EKS Managed Node Groups to EKS Auto nodes with ZERO DOWNTIME and continued application availability using my terraform code.
I have also added a BONUS section which explains how we can control our pod's deployments on EKS Auto Mode nodes or other compute types.
Motivation
- Terraform-provider-aws released a new version v5.8.1 which allows to enabled EKS Auto with built-in NodePools on existing cluster.
- Terraform-aws-eks release a new version v20.31.1 which allows to use custom NodeClass/NodePools when EKS Auto is enabled without built-in NodePools.
I want this blog to be really short, crisp and efficient so lets jump into actual steps!
Deploy Terraform cluster without EKS Auto Mode
We want to create the use case where we have an existing cluster WITHOUT EKS Auto Mode using EKS MNG.
Use this repository code to deploy EKS cluster with Managed node group.
Note: I am attaching policies to the node IAM role for EKS MNG - this is too permissive, better to use EKS Pod Identity (or IRSA, but EKS Pod Identity is preferred). Feel free to send a PR to the repo :)
Deploy workload or pods
- We will automate this as well using terraform's
kubectl_manifest
resource, we will deploy workload yaml code using terraform
Note: During cluster creation, test workload(pods) were not deployed because kubectl context was not set locally. So run the following command to set the kubectl context
and run terraform apply
again once cluster is created.
aws eks --region us-east-1 update-kubeconfig --name eks-existing-cluster-tf-test --profile <your-profile-name> ; terraform apply
Current state of EKS cluster before EKS Auto Mode
Let's verify the current state of eks cluster when EKS Auto mode is not enabled.
EKS Auto mode is disabled.
- EKS Auto Managed Node group created by me is running.
- Pods are running on EKS managed node group
Enable EKS Auto Mode on Existing Cluster
- Uncomment the following code to the eks.tf and
terraform apply
to enable EKS Auto Mode
bootstrap_self_managed_addons = true
cluster_compute_config = {
enabled = true
}
-
bootstrap_self_managed_addons = true
is very important otherwise you will face error where terraform tries to recreate the cluster again. I literally cried over this
Current state of EKS cluster after EKS Auto Mode
- As expected built-in NodePools are empty
Migrate workload(pods) from EKS MNG to EKS Auto Node
- There are couple of ways to smoothly migrate existing workloads from MNG to EKS Auto with minimal disruption while maintaining application’s availability throughout the migration.
Note: Copy the EKS MNG node group name.
Using eksctl tool
- The following command will cordon all nodes and all pods are evicted from a nodegroup and EKS will provision pods to node managed by EKS Auto.
eksctl drain nodegroup --cluster=<clusterName> --name=<copiedNodegroupName> --region us-east-1 --profile=<profile>
eksctl command evicts pod one at a time which I have tested so application availability is maintained.
But if you still want to be 100% sure, you can use the best practice of using pod Disruption budget. We will automate this using terraform so run
terraform apply
resource "kubectl_manifest" "test_pdb" {
yaml_body = <<YAML
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: test-pdb
labels:
environment: test
spec:
minAvailable: 1
selector:
matchLabels:
environment: test
YAML
}
- After migrating; If we want to allow scheduling pods to EKS MNG we need to uncordon the EKS MNG or you can delete the Node group
eksctl drain nodegroup --cluster=<clusterName> --name=<copiedNodegroupName> --region us-east-1 --profile=<profile> --undo
Using kubectl
- we can use the following command to drain the nodes using kubectl
kubectl drain --ignore-daemonsets <node name>
- Once it returns (without giving an error), you can delete the node or you want to tell Kubernetes that it can resume scheduling new pods onto the node
kubectl uncordon <node name>
[ BONUS ] How to schedule Pods always on EKS Auto Nodes?
- There are 2 options to achieve this :
Either delete The NodeGroup and let EKS Auto handle the scheduling on EKS Auto Nodes
Using labels and NodeAffinity
Control if a workload is deployed on EKS Auto Mode nodes
There is concept called mix-mode cluster where you’re running both EKS Auto Mode and other compute types, such as self-managed Karpenter provisioners or EKS Managed Node Groups.
In mix mode clusters by default deployment is deployed to EKS MNG nodes and not EKS Auto Nodes
In such case we can use labels and nodeAffinity.
Using NodeSelector label
- Use the label
eks.amazonaws.com/compute-type: auto
when you want a workload is deployed to EKS Auto Node. - This nodeSelector value is only relevant if you are running a cluster in a mixed mode, node types not managed by EKS Auto Mode
apiVersion: apps/v1
kind: Deployment
spec:
nodeSelector:
eks.amazonaws.com/compute-type: auto
- I have an added the above configuration in sample_app_on_eks_auto_nodes.tf file. We are automating using Terraform so uncomment and run `terraform apply.
Using nodeAffinity
- You can add this nodeAffinity to Deployments or other workloads to require Kubernetes to not schedule them onto EKS Auto Mode nodes
- I have added the workload with nodeAffinity in sample_app_not_on_eks_auto_nodes.tf . We are automating using Terraform so uncomment and run
terraform apply
.
From DevOps, IaC Perspective
We saw how we can enable EKS Auto mode for Existing clusters with built-in NodePools using terraform-eks-module
We also saw how we can migrate our existing workload from EKS Managed Group to EKS Auto Nodes without any down time as EKS Auto node respect PodDisruptionBudget.
We also saw how we can use nodeSelector Labels and nodeAffinity to control deployment of workload in case of mixed-mode EKS clusters.
Currently EKS Auto deploys EC2 of instance type c6a.large which can be also customized using nodeClass and NodePool which we will see in the next blog. Follow me on Linkedin or on dev.to so that you get timely updates of what I share.
Feel free to reach out to me on Linkedin, X if you face any error migrating your Existing workloads to EKS Auto Mode Nodes using terraform.
Top comments (8)
Thanks for the post Jatin, This is very helpful.
I have quick query on how do we create custom NodePool and Node Class for existing cluster as we have specific requirements from compute perspective.
THANK YOU @deepak_b. I am happy that it helped you.
Here is the documentation
docs.aws.amazon.com/eks/latest/use...
docs.aws.amazon.com/eks/latest/use...
Nodepool requires to have nodeclass in the cluster.
Let me know if you have any other questions would be happy to help and I am very curious to understand how you are using EKS auto for your needs and how it is helping you. Any challenges you are facing or you wish they have included some specific feature in EKS Auto? Please share your views.
Thanks Jatin.. was wondering on how can we add it to the terraform code for existing cluster so that when migrate our workload to Auto mode we make sure its using custom nodeclass and nodePool
@deepak_b Thank you for your question
There is one consideration before you can add custom nodeClass and nodePool
nodeClass and nodePool are the concepts of karpenter. Having said that it also means the EKS Auto when enabled on EKS installs karpenter and CRD to support nodeClass and nodePool
This means first you need to enable EKS Auto mode on your cluster then using terraform install nodeClass and nodePool
nodeClass and nodePool essentially are yaml manifest which can be applied using terraform using kubectl_manifest resource
Note: In order to use this resource you need to connect respective kubectl provider and configure it so that it can connect to eks cluster
Hope it helps.
Thanks Jatin. This helps :)
I am also curious how you are trying to use EKS Auto for what kind of use case and why did you guys decided to move to EKS Auto? Please share it if you have time
bootstrap_self_managed_addons = true
saved my day :D, Thanks!@harik8
I lost my few hairs for this :D Happy that it helped. Share it with Other EKS lovers or in your Linkedin Network if you think it can help others too