AWS & SRE Field Manual (Part 5): Amazon EKS Architecture, Pod Networking & Next-Gen Scaling

1. TL;DR & Problem Statement

• Definition: A managed Kubernetes service that offloads the operational complexity of deploying, scaling, and maintaining the Kubernetes Control Plane while integrating natively with core AWS networking, identity, and storage backbones.
• Problem Solved: Eliminates single points of failure and maintenance overhead associated with self-managed etcd and master nodes (manual backup/restore, OS patching, control plane scaling) and bridges the gap between Kubernetes orchestration and cloud-native AWS primitives.
• Category: Containers & Compute

---

2. Core Architecture & Key Components

                     ┌──────────────────────────────────────────────────┐
                     │      AWS Managed Control Plane (Multi-AZ)        │
                     │   kube-apiserver  |  etcd  | controller-manager  │
                     └────────────────────────┬─────────────────────────┘
                                              │
                                  Single Endpoint (HTTPS)
                                              │
       ┌──────────────────────────────────────┼──────────────────────────────────────┐
       ▼                                      ▼                                      ▼
┌──────────────┐                      ┌──────────────┐                      ┌──────────────┐
│ Self-Managed │                      │ Managed Node │                      │ AWS Fargate  │
│  EC2 Nodes   │                      │ Groups (MNG) │                      │ (Serverless) │
└──────────────┘                      └──────────────┘                      └──────────────┘

2.1. Managed Control Plane

• AWS provisions and maintains kube-apiserver, etcd, kube-scheduler, and kube-controller-manager across at least 3 Availability Zones (AZs) behind redundant Network Load Balancers.
• Features automated etcd snapshot backups, control plane autoscaling, and version patch rollouts with an enterprise-grade availability SLA.

2.2. Data Plane Abstraction Models

• Self-Managed Nodes: Legacy pattern where instances are provisioned, bootstrapped, and maintained manually by the operator.
• Managed Node Groups (MNG): AWS automates EC2 provisioning, AMI upgrades, and graceful node draining (kubectl drain) during cluster lifecycle operations while retaining EC2 instance visibility.
• AWS Fargate: Serverless compute engine that removes the node abstraction entirely. Each pod runs inside its own isolated VM boundary with right-sized CPU/memory allocation and zero OS-level server maintenance.

2.3. Next-Gen Node Autoscaling (Karpenter vs. Cluster Autoscaler)

• Traditional Cluster Autoscaler (CA) depends on fixed Auto Scaling Groups (ASGs), causing slow scale-up times and instance over-provisioning.
• Karpenter: A declarative, group-less node autoscaler. It monitors pending unschedulable pods, calculates exact aggregate compute/memory requirements and topology spread constraints, and calls the AWS EC2 fleet API directly to provision right-sized Spot or On-Demand instances within seconds.

---

3. Deep Dive Integrations & Advanced Mechanisms

A. Networking: AWS VPC CNI & Prefix Delegation

• Standard Kubernetes overlays (e.g., Flannel) wrap packets inside VXLAN and apply SNAT. The AWS VPC CNI plugin assigns real, routable VPC subnet IPs directly to each Pod ENI.
• Prefix Delegation (ENABLE_PREFIX_DELEGATION=true): Standard EC2 instances have strict ENI/IP limits, restricting pod density per node. Prefix delegation assigns /28 IPv4 subnets (16 IPs per slot) instead of single IPs, multiplying maximum pod density per node without requiring larger instance types.

B. Workload Identity: IRSA vs. EKS Pod Identities

• IAM Roles for Service Accounts (IRSA): Uses OIDC federation. A Kubernetes ServiceAccount is annotated with an IAM Role ARN, and AWS STS injects short-lived credentials via projected volume tokens.
• EKS Pod Identities: The modern, streamlined alternative to IRSA. Eliminates manual OIDC provider setup and complex trust relationship JSONs. Permissions are mapped directly via the EKS API and enforced by the local EKS Pod Identity agent daemonset.

C. Storage: EBS & EFS CSI Drivers

• Bridges Kubernetes dynamic volume provisioning (PersistentVolumeClaim) with native AWS storage classes (gp3, io2, EFS).
• Handles dynamic volume creation, attachment across AZs, and automated disk resizing through standard Kubernetes manifests.

D. Endpoint Access Control (Private Clusters)

• Public and Private: API server is accessible from the internet; worker node traffic remains within the private VPC.
• Private Only (Production Standard): Disables public internet access to kube-apiserver completely. Cluster management requires direct VPC peering, AWS Client VPN, or an internal Bastion host.

---

4. Practical Notes & Configuration Snippets

EKS Pod Identity Association (Terraform)

resource "aws_eks_pod_identity_association" "s3_reader" {
  cluster_name    = "production-cluster"
  namespace       = "default"
  service_account = "s3-reader-sa"
  role_arn        = aws_iam_role.s3_reader_role.arn
}

Enabling Prefix Delegation on AWS VPC CNI

# Enable IPv4 prefix delegation for increased pod density
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

# Set warm prefix target to optimize IP allocation latency
kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1

Verify Cluster Endpoint and Worker Nodes

# Update local kubeconfig to point to private/public EKS cluster
aws eks update-kubeconfig --region us-east-1 --name production-cluster

# Check all node statuses and their instance types provisioned by Karpenter
kubectl get nodes -L node.kubernetes.io/instance-type,karpenter.sh/capacity-type

---

5. Gotchas & Common Pitfalls

• VPC Subnet IP Exhaustion: Because the VPC CNI assigns native subnet IPs to every single pod, running large clusters on narrow VPC CIDRs (e.g., /20 or /24) will quickly deplete available subnet IPs, preventing new EC2 instances and pods from launching. Always design secondary CIDR blocks (e.g., 100.64.0.0/16 CGNAT) for pod networking.
• EBS Cross-AZ Mounting Failure: Amazon EBS volumes are strictly zonal. A pod bound to an EBS volume in us-east-1a cannot mount that volume if rescheduled by Kubernetes onto a node in us-east-1b. Use EBS only with single-AZ stateful workloads or migrate to Amazon EFS / Amazon FSx for multi-AZ shared storage.
• CoreDNS Throttling: Rapid pod scale-ups can overwhelm the default 2-replica CoreDNS deployment, causing intermittent DNS query timeouts (5-second lookups). Deploy NodeLocal DNSCache and configure horizontal autoscaling for CoreDNS.

---

6. Production Best Practices

• Control Plane Logging to CloudWatch: Always enable all 5 control plane log types in production (api, audit, authenticator, controllerManager, scheduler) to maintain compliance and facilitate root cause analysis during incidents.
• AWS Load Balancer Controller over In-Tree Service Controller: Avoid using the deprecated built-in Kubernetes service load balancer. Use the AWS Load Balancer Controller to provision target-group-binding Application Load Balancers (ALBs) routing traffic directly to Pod IPs in IP-mode, bypassing kube-proxy NodePort latency.
• Strict AWS Security Group per Pod: For regulated workloads (PCI-DSS / HIPAA), use Security Groups for Pods to enforce native AWS stateful firewall rules directly at the Kubernetes pod ENI level instead of relying solely on Kubernetes NetworkPolicies.
• Karpenter Consolidation Policies: Enable consolidationPolicy: WhenUnderutilized in Karpenter NodePools to continuously repack workloads onto fewer or cheaper compute instances during low-traffic periods, reducing idle cluster costs by 30% to 50%.