Exploring AWS EKS Auto Mode: A Simplified Kubernetes Experience

EKS Auto Mode extends AWS management of Kubernetes clusters beyond the cluster itself, to allow AWS to also set up and manage the infrastructure that enables the smooth operation of your workloads. You can delegate key infrastructure decisions and leverage the expertise of AWS for day-to-day operations. Cluster infrastructure managed by AWS includes many Kubernetes capabilities as core components, as opposed to add-ons, such as compute autoscaling, pod and service networking, application load balancing, cluster DNS, block storage, and GPU support.

Features

EKS Auto Mode provides the following high-level features:

Streamline Kubernetes Cluster Management: EKS Auto Mode streamlines EKS management by providing production-ready clusters with minimal operational overhead. With EKS Auto Mode, you can run demanding, dynamic workloads confidently, without requiring deep EKS expertise.

Application Availability: EKS Auto Mode dynamically adds or removes nodes in your EKS cluster based on the demands of your Kubernetes applications. This minimizes the need for manual capacity planning and ensures application availability.

Efficiency: EKS Auto Mode is designed to compute costs while adhering to the flexibility defined by your NodePool and workload requirements. It also terminates unused instances and consolidates workloads onto other nodes to improve cost efficiency.

Security: EKS Auto Mode uses AMIs that are treated as immutable for your nodes. These AMIs enforce locked-down software, enable SELinux mandatory access controls, and provide read-only root file systems. Additionally, nodes launched by EKS Auto Mode have a maximum lifetime of 21 days (which you can reduce), after which they are automatically replaced with new nodes. This approach enhances your security posture by regularly cycling nodes, aligning with best practices already adopted by many customers.

Automated Upgrades: EKS Auto Mode keeps your Kubernetes cluster, nodes, and related components up to date with the latest patches, while respecting your configured Pod Disruption Budgets (PDBs) and NodePool Disruption Budgets (NDBs). Up to the 21-day maximum lifetime, intervention might be required if blocking PDBs or other configurations prevent updates.

Managed Components: EKS Auto Mode includes Kubernetes and AWS cloud features as core components that would otherwise have to be managed as add-ons. This includes built-in support for Pod IP address assignments, Pod network policies, local DNS services, GPU plug-ins, health checkers, and EBS CSI storage.

Customisable NodePools and NodeClasses: If your workload requires changes to storage, compute, or networking configurations, you can create custom NodePools and NodeClasses using EKS Auto Mode. While default NodePools and NodeClasses can’t be edited, you can add new custom NodePools or NodeClasses alongside the default configurations to meet your specific requirements.

Before Auto Mode

After Auto Mode

Amazon EKS Auto Mode managed instances

Amazon EKS Auto Mode automates routine tasks for creating new EC2 Instances, and attaches them as nodes to your EKS cluster. EKS Auto Mode detects when a workload can’t fit onto existing nodes, and creates a new EC2 Instance.

Amazon EKS Auto Mode is responsible for creating, deleting, and patching EC2 Instances. You are responsible for the containers and pods deployed on the instance.

EC2 Instances created by EKS Auto Mode are different from other EC2 Instances, they are managed instances. These managed instances are owned by EKS and are more restricted. You can’t directly access or install software on instances managed by EKS Auto Mode.

AWS suggests running either EKS Auto Mode or self-managed Karpenter. You can install both during a migration or in an advanced configuration. If you have both installed, configure your node pools so that workloads are associated with either Karpenter or EKS Auto Mode.

Standard EC2 Instance

• You are responsible for patching and updating the instance.
• EKS is not responsible for the software on the instance.
• You can delete the EC2 Instance using the EC2 API.
• You can use SSH to access the EC2 Instance.
• You determine the operating system and image (AMI).
• You can deploy workloads that rely on Windows or Ubuntu functionality.
• You determine what instance type and family to launch.

EKS Auto Mode managed instance

• AWS automatically patches and updates the instance.
• EKS is responsible for certain software on the instance, such as kubelet, the container runtime, and the operating system.
• EKS determines the number of instances deployed in your account. If you delete a workload, EKS will reduce the number of instances in your account.
• You can deploy pods and containers to the managed instance.
• AWS determines the operating system and image.
• You can deploy containers based on Linux, but without specific OS dependencies.
• AWS determines what instance type and family to launch. You can use a Node Pool to limit the instance types EKS Auto Mode selects from.

Identity and access in EKS Auto Mode

In EKS Auto Mode, AWS IAM roles are automatically mapped to Kubernetes permissions through EKS access entries, removing the need for manual configuration of aws-auth ConfigMaps or custom bindings. When you create a new auto mode cluster, EKS automatically creates the corresponding Kubernetes permissions using Access entries, ensuring that AWS services and cluster components have the appropriate access levels within both the AWS and Kubernetes authorization systems. This automated integration reduces configuration complexity and helps prevent permission-related issues that commonly occur when managing EKS clusters.

Cluster IAM role
The Cluster IAM role is an AWS Identity and Access Management (IAM) role used by Amazon EKS to manage permissions for Kubernetes clusters. This role grants Amazon EKS the necessary permissions to interact with other AWS services on behalf of your cluster, and is automatically configured with Kubernetes permissions using EKS access entries.

Node IAM role
The Node IAM role is an AWS Identity and Access Management (IAM) role used by Amazon EKS to manage permissions for worker nodes in Kubernetes clusters. This role grants EC2 instances running as Kubernetes nodes the necessary permissions to interact with AWS services and resources, and is automatically configured with Kubernetes RBAC permissions using EKS access entries.

VPC networking and load balancing in EKS Auto Mode

When you use EKS Auto Mode, AWS manages the VPC Container Network Interface (CNI) configuration and load balancer provisioning for your cluster. You can influence networking behaviors by defining NodeClass objects and applying specific annotations to your Service and Ingress resources, while maintaining the automated operational model that EKS Auto Mode provides.

VPC CNI networking
With EKS Auto Mode, you do not directly configure the AWS VPC CNI. AWS manages node and pod networking. Instead, you create a NodeClass Kubernetes object.

Configure VPC CNI with NodeClass
The NodeClass resource in EKS Auto Mode allows you to customize certain aspects of the VPC Container Network Interface (CNI) configuration without directly managing the CNI plugin. Through NodeClass, you can specify security group selections, control node placement across VPC subnets, set SNAT policies, configure network policies, and enable network event logging. This approach maintains the automated operational model of EKS Auto Mode while providing flexibility for network customization.

You can use a NodeClass to:

• elect a Security Group for Nodes
• Control how nodes are placed on VPC Subnets
• Set the Node SNAT Policy to random or disabled
• Set the Network Policy to Default Deny or Default Allow
• Enable Network Event Logging to a file.

Considerations

• EKS Auto Mode automatically formats and configures NVMe local storage on supported instance types. For nodes with multiple NVMe drives, EKS sets up a RAID 0 array. This automation eliminates the need for manual formatting and RAID configuration of local NVMe storage in EKS clusters.
• Amazon EKS Auto Mode does not support AWS Fault Injection Service. For more information, see Managing Fault Injection Service experiments in the AWS Resilience Hub User Guide.
• You do not need to install the Neuron Device Plugin on EKS Auto Mode nodes.
• If you have other types of nodes in your cluster, you need to configure the Neuron Device plugin to not run on auto mode nodes. For more information, see Control if a workload is deployed on EKS Auto Mode nodes.

EKS Auto Mode supports:

• EKS Network Policies.
• The HostPort and HostNetwork options for Kubernetes Pods.
• Pods in public or private subnets.

EKS Auto Mode does not support:

• Security Groups per Pod (SGPP).
• Custom Networking. The IP Addresses of Pods and Nodes must be from the same CIDR Block.
• Warm IP, warm prefix, and warm ENI configurations.
• Minimum IP targets configuration.
• Enabling or disabling prefix delegation.
• Other configurations supported by the open-source AWS CNI.
• Network Policy configurations such as conntrack timer customization (default is 300s).
• Exporting network event logs to CloudWatch.

Sources

https://aws.amazon.com/blogs/containers/getting-started-with-amazon-eks-auto-mode

https://docs.aws.amazon.com/eks/latest/userguide/automode.html