EKS Production Hardening Guide: Security, Karpenter, Cost, and Upgrades for Real-World Kubernetes

Spinning up an EKS cluster takes 20 minutes. Running one in production without incidents takes a lot more. The gap between "it works in the demo" and "it survives Black Friday" is filled with security hardening, intelligent autoscaling, cost discipline, and upgrade strategy.

This guide covers the production-readiness checklist for EKS: the security controls, the Karpenter setup, the cost optimization, and the day-2 operations that keep clusters healthy at scale.

The EKS Shared Responsibility Model

┌─────────────────────────────────────────────────────────────────┐
│  AWS MANAGES (Control Plane)                                     │
│  • Kubernetes API server, etcd, scheduler, controller manager    │
│  • Control plane HA across 3 AZs                                 │
│  • Control plane patching and availability                       │
├─────────────────────────────────────────────────────────────────┤
│  YOU MANAGE (Data Plane + Config)                                │
│  • Worker nodes (or use EKS Auto Mode / Fargate)                 │
│  • Pod security, network policies, RBAC                          │
│  • Secrets management, image scanning                            │
│  • Add-ons, upgrades, cost optimization                          │
└─────────────────────────────────────────────────────────────────┘

---

Part 1: Security Hardening

1. RBAC and Least Privilege

Default deny → Grant specific permissions per role

├── Developers: read pods/logs in their namespace only
├── CI/CD: deploy to specific namespaces
├── Platform team: cluster-admin (limited members)
└── Applications: ServiceAccount with minimal permissions (IRSA)

IRSA (IAM Roles for Service Accounts): Map Kubernetes ServiceAccounts to IAM roles. Pods get AWS permissions without node-level credentials.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-reader
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/s3-read-role

EKS Pod Identity (newer alternative to IRSA): simpler association, no OIDC trust policy management.

2. Network Policies (Default Deny)

By default, all pods can talk to all pods. Lock it down:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Then explicitly allow required traffic. Use the VPC CNI network policy support or Cilium for enforcement.

3. Secrets Encryption

• Envelope encryption: Encrypt Kubernetes secrets in etcd using KMS
• External Secrets Operator: Sync from Secrets Manager / Parameter Store (don't store secrets in etcd at all)
• Never commit secrets to Git or bake into images

# EKS cluster with KMS envelope encryption
encryptionConfig:
  - resources: ["secrets"]
    provider:
      keyArn: arn:aws:kms:eu-west-1:123456789:key/xxx

4. Pod Security Standards

Enforce Pod Security Admission (replaces deprecated PodSecurityPolicy):

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted

restricted prevents: privileged containers, host namespace access, running as root, privilege escalation.

5. Image Security

• ECR image scanning: Scan on push, block deploy on critical CVEs
• Image signing: Cosign/Notation for supply chain integrity
• Admission control: Kyverno or OPA Gatekeeper to enforce policies (only signed images, no :latest, resource limits required)

6. Control Plane Logging

Enable all control plane log types → CloudWatch:

api | audit | authenticator | controllerManager | scheduler

The audit log is critical for security forensics — it records every API call to the cluster.

---

Part 2: Karpenter for Intelligent Compute

Karpenter replaced Cluster Autoscaler as the production standard. It provisions right-sized nodes in seconds based on actual pod requirements.

Why Karpenter Over Cluster Autoscaler

Feature	Cluster Autoscaler	Karpenter
Node selection	Fixed node groups	Dynamic, picks optimal instance type
Speed	Minutes	Seconds
Bin packing	Limited	Intelligent consolidation
Spot handling	Basic	Advanced (diversification, interruption)
Instance flexibility	Per node group	Any instance matching constraints

Karpenter NodePool (v1 API)

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["arm64", "amd64"]  # Graviton + x86
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        name: default
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s
  limits:
    cpu: 1000

Karpenter Production Best Practices

1. SQS interruption handling — configure Spot interruption queue so Karpenter drains nodes gracefully (2-min warning)
2. NodePool isolation — separate NodePools for system vs application workloads
3. Disruption budgets — limit how many nodes Karpenter can consolidate at once
4. AMI pinning — pin AMI versions for predictable upgrades (don't auto-update)
5. Consolidation — enable to bin-pack workloads and reduce node count
6. Diversification — allow multiple instance types to survive Spot interruptions

---

Part 3: Cost Optimization

EKS costs can spiral. Here's how to control them:

Compute Cost Levers

Lever	Savings
Karpenter consolidation	30-50% (bin-packing eliminates waste)
Spot for stateless workloads	Up to 90% vs On-Demand
Graviton (arm64) nodes	20-40% better price/performance
Savings Plans for baseline	Up to 72% for steady-state On-Demand
Right-size pod requests	Prevents over-provisioning nodes

The Baseline + Burst Pattern

Baseline (predictable load)  → On-Demand + Savings Plans (cost-committed)
Burst (variable load)        → Spot Instances (cheap, interruptible)
Critical system pods         → On-Demand (never interrupted)

Right-Sizing Pod Requests

resources:
  requests:
    cpu: 250m      # What the pod actually needs (from metrics)
    memory: 512Mi
  limits:
    memory: 512Mi   # Prevent OOM affecting neighbors
    # No CPU limit — let it burst (CPU is compressible)

Key insight: Over-requesting CPU/memory forces Karpenter to provision more/bigger nodes. Use VPA recommendations or metrics to right-size requests.

Eliminate NAT Gateway Costs

Route AWS API traffic through VPC endpoints (Gateway endpoints for S3/DynamoDB are free):

Without VPC endpoints: Pod → NAT Gateway ($0.045/GB) → S3
With VPC endpoints:    Pod → S3 Gateway Endpoint ($0) → S3

---

Part 4: Networking

VPC CNI Configuration

• Prefix delegation — assign /28 prefixes to ENIs (more pods per node)
• Custom networking — pods in separate subnets from nodes
• Security groups for pods — apply SGs at pod level (not just node)

Ingress and Load Balancing

Option	Use For
AWS Load Balancer Controller	ALB (L7) or NLB (L4) for ingress
Ingress (ALB)	HTTP/HTTPS routing, path-based
Gateway API	Modern replacement for Ingress (more expressive)
VPC Lattice	Cross-cluster service mesh with IAM auth

Service Mesh (When Needed)

Don't add a service mesh by default. Add Istio/Linkerd only when you need:

• mTLS between all services
• Advanced traffic management (retries, circuit breaking)
• Detailed L7 observability

For simpler needs, VPC Lattice or App Mesh may suffice.

---

Part 5: Observability

Metrics  → CloudWatch Container Insights + Prometheus (Managed Prometheus)
Logs     → Fluent Bit → CloudWatch Logs / OpenSearch
Traces   → ADOT (OpenTelemetry) → X-Ray
Dashboards → Managed Grafana

Essential Metrics to Monitor

• Node CPU/memory pressure
• Pod restart counts (crash loops)
• Pending pods (capacity issues)
• Karpenter provisioning latency
• Persistent volume usage
• API server latency

---

Part 6: Upgrades

EKS supports Kubernetes versions for ~14 months. Plan upgrades:

Upgrade Strategy

1. Read the changelog — check for deprecated APIs (use kubent / Pluto to find them)
2. Upgrade control plane first — AWS handles this (one minor version at a time)
3. Upgrade add-ons — VPC CNI, CoreDNS, kube-proxy must be compatible
4. Upgrade nodes — rolling replacement (Karpenter makes this easy — drain + provision new AMI)
5. Test in non-prod first — always

EKS Auto Mode (Simplify Everything)

EKS Auto Mode (GA) manages compute, scaling, and upgrades automatically:

• AWS manages node provisioning, patching, and Karpenter under the hood
• You focus on workloads, not infrastructure
• Trade-off: less control, slightly higher cost, but dramatically less ops

Consider Auto Mode when: small platform team, want minimal Kubernetes ops overhead.

---

Part 7: Resilience

Backup and DR

• Velero — backup cluster state and persistent volumes
• Multi-AZ — spread nodes across 3 AZs (Karpenter handles this)
• Pod Disruption Budgets — ensure minimum replicas during disruptions
• Cross-region DR — GitOps (ArgoCD) redeploys to DR cluster from Git

Pod Disruption Budget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

Ensures at least 2 API pods stay running during node drains/upgrades.

---

Production Readiness Checklist

SECURITY
☐ RBAC with least privilege
☐ IRSA / Pod Identity (no node-level AWS creds)
☐ Default-deny NetworkPolicy per namespace
☐ Secrets encrypted (KMS envelope or External Secrets)
☐ Pod Security Standards (restricted)
☐ ECR image scanning + admission control
☐ Control plane audit logging enabled

COMPUTE
☐ Karpenter with Spot + On-Demand + Graviton
☐ SQS interruption handling
☐ Consolidation enabled
☐ Right-sized pod requests

COST
☐ VPC endpoints (eliminate NAT charges)
☐ Savings Plans for baseline
☐ Spot for stateless workloads

OBSERVABILITY
☐ Container Insights + Prometheus
☐ Centralized logging (Fluent Bit)
☐ Distributed tracing (ADOT)

RESILIENCE
☐ Multi-AZ node distribution
☐ Pod Disruption Budgets
☐ Velero backups
☐ Tested upgrade path

GITOPS
☐ ArgoCD / Flux for declarative deployments
☐ All cluster config in Git

---

Common EKS Production Mistakes

Mistake	Impact	Fix
No resource requests/limits	Node overcommit, OOM kills	Set requests based on metrics
All traffic through NAT Gateway	High data processing costs	VPC endpoints for S3/DynamoDB
No NetworkPolicy	Lateral movement if breached	Default-deny + explicit allow
Node-level IAM credentials	Over-privileged pods	IRSA / Pod Identity
Cluster Autoscaler (legacy)	Slow, inefficient scaling	Migrate to Karpenter
No PDB	Outages during upgrades	Pod Disruption Budgets
Ignoring version support window	Forced emergency upgrades	Scheduled upgrade cadence
Secrets in etcd unencrypted	Exposure risk	KMS envelope encryption

---

Summary

Production EKS comes down to seven pillars:

1. Security — RBAC, IRSA, NetworkPolicies, Pod Security Standards, image scanning, audit logging
2. Compute — Karpenter with Spot + On-Demand + Graviton, intelligent consolidation
3. Cost — VPC endpoints, Savings Plans, Spot, right-sized requests
4. Networking — VPC CNI tuning, ALB Controller, Gateway API
5. Observability — Container Insights, Prometheus, ADOT tracing
6. Upgrades — scheduled cadence, add-on compatibility, or EKS Auto Mode
7. Resilience — multi-AZ, PDBs, Velero backups, GitOps

If your team is small and Kubernetes ops is a burden, seriously evaluate EKS Auto Mode — it handles most of the compute, scaling, and upgrade complexity so you can focus on workloads.

---

Alpesh Kumbhare is an AWS Architect at Atos, specializing in Kubernetes, container platforms, and AWS infrastructure automation. Connect on LinkedIn.