DEV Community: Prateek Srivastava

I Managed Kubernetes Across AWS, Azure, and GCP Simultaneously — Here's What Nobody Tells You

Prateek Srivastava — Fri, 21 Aug 2026 19:11:46 +0000

The day the same Helm chart behaved differently on three clouds in the same hour was the day I stopped trusting "cloud-agnostic" as a real thing.

Nobody plans to manage Kubernetes on three clouds at the same time. It happens gradually — an acquisition brings an Azure footprint, a new business unit commits to GCP, and you already run EKS. Then one morning a page fires and you're staring at three terminal windows, three different cluster behaviours, one very angry incident bridge.

I've spent the better part of two years in exactly that situation. What follows isn't a comparison chart you can find on any vendor blog. It's the stuff I learned the hard way — the silent differences that don't show up until something breaks in production at 2 AM.

📟 The Incident That Started It All

We deployed the same Helm chart — identical values, same image tag — to EKS, AKS, and GKE within the same 20-minute release window. EKS came up healthy. AKS came up healthy. GKE's pods started, ran for about 90 seconds, then began OOMKilling in a loop. Same YAML. Same limits. Three different outcomes. Four hours to find why. That story is lesson #3.

1. Networking: The Biggest Lie — "Pods Are Just Pods"

This is where most multi-cloud pain originates. Every Kubernetes tutorial tells you pods get their own IPs and can talk to each other. What it doesn't tell you is that where those IPs come from completely changes your blast radius, subnet planning, security posture, and the failure modes you'll face at 3 AM.

EKS: Your pods eat your VPC subnet IPs

EKS uses the AWS VPC CNI by default. Every pod gets a real, routable IP from your VPC subnet. This sounds fine until you have a /24 subnet and someone deploys a service with 200 replicas. The error isn't "out of IPs" — it's an ENI attachment timeout.

The harder gotcha: each EC2 instance type has a max ENI count × max IPs per ENI cap. A t3.medium can hold 3 ENIs × 6 IPs = 18 pod IPs max, minus 1 for the node. Pods sit Pending on nodes with plenty of CPU and RAM — the constraint is invisible to the scheduler.

# Check ENI and IP capacity on a node
kubectl describe node <node-name> | grep -A5 "Allocatable"

# Check aws-node daemonset for ENI attachment errors
kubectl logs -n kube-system -l k8s-app=aws-node --tail=50 | grep -i "err\|fail\|exhaust"

Fix: Enable VPC CNI prefix delegation — one ENI prefix = 16 IPs instead of 1.

AKS: Azure CNI pre-allocates IPs whether you like it or not

AKS with Azure CNI reserves IPs per node based on --max-pods (default 30) — before any pod runs. A 100-node cluster pre-consumes 3,000 VNet IPs. Use Azure CNI Overlay mode if IP-constrained.

GKE: Clean until you need to peer networks

GKE uses alias IP ranges, isolated from your main VPC. Clean story — until you peer your GKE VPC with on-prem and discover the pod CIDR overlaps your on-prem range, because you let GKE auto-assign ranges six months ago. Always explicitly define secondary ranges.

	EKS	AKS	GKE
Pod IP source	VPC subnet (real IPs)	VNet or overlay	Alias IP ranges
IP exhaustion risk	High — ENI limits	High — pre-allocated	Low — isolated
Common 3 AM failure	ENI attachment timeout	VNet IP exhaustion	CIDR overlap during peering

2. IAM + RBAC: Three Identity Models, One Misconfiguration Away from Disaster

EKS: The IRSA silent fallback trap

If your IRSA annotation has a typo, the assume-role call silently fails and the pod falls through to the node's instance profile — often over-permissive. You won't see an error. Always verify:

kubectl exec -it <pod-name> -- aws sts get-caller-identity

I once spent three hours debugging why a pod had too much S3 access. A typo in the annotation. Silent fallback to the node's IAM role. The pod happily worked — with the wrong permissions.

AKS: Workload Identity federation — fragile on setup

Three pieces must align exactly: Managed Identity, federated credential (exact OIDC issuer URL, including trailing slashes), and the Kubernetes service account annotation. One mismatch = silent 401.

GKE: Node rotation breaks implicit identity

Enabling Workload Identity on a node pool removes the node's default Google service account. Any workload relying on implicit node identity breaks on the next node rotation. Audit before enabling.

3. Memory + cgroups: Why the Same Container OOMKilled on GKE but Not on EKS

Here's the opening incident. Same image, same limits, EKS and AKS healthy, GKE OOMKilling every 90 seconds.

GKE had moved to containerd with cgroup v2 on newer node images. EKS and AKS were still on cgroup v1.

Our Java service used JVM ergonomics to auto-detect heap size:

cgroup v1: JVM reads from /sys/fs/cgroup/memory/memory.limit_in_bytes
cgroup v2: JVM reads from /sys/fs/cgroup/memory.max

Our old JVM didn't handle cgroup v2. It read host memory (64GB) instead of the container limit (2GB), allocated an 8GB heap into a 2GB container, and OOMKilled within 90 seconds of starting.

# Check which cgroup version the container sees
kubectl exec -it <pod-name> -- cat /proc/1/cgroup
# cgroup v1: "12:memory:/kubepods/..."
# cgroup v2: single line "0::/"

# Check what heap the JVM actually allocated
kubectl exec -it <pod-name> -- java -XshowSettings:all -version 2>&1 | grep -i heap

# See OOMKill events across all namespaces
kubectl get events --field-selector reason=OOMKilling -A --sort-by='.lastTimestamp'

Fix: JDK 15+ (native cgroup v2 support) or JDK 11 with -XX:+UseContainerSupport. Always explicitly set -Xmx and -Xms. This isn't just a Java problem — Go's GOMAXPROCS and Python's multiprocessing.cpu_count() have the same pattern.

	EKS	AKS	GKE
Default node OS	Amazon Linux 2023	Ubuntu 22.04 / Azure Linux	Container-Optimized OS
cgroup version	v2 (AL2023), v1 (AL2)	v2 (Ubuntu 22.04+)	v2 (COS since mid-2022)

4. Storage: The AZ-Pinning Trap

Block storage (EBS / Azure Disk / Persistent Disk) is AZ-specific on every cloud. Your PVC and the pod using it must be in the same AZ.

The scenario I've watched happen three times:

Stateful pod uses a PVC in AZ-A. Traffic grows. Cluster Autoscaler spins up nodes in AZ-B and AZ-C. New pods stay Pending — PVC is in AZ-A, the new nodes aren't. On-call engineer stares at CPU and memory graphs seeing nothing wrong.

# The event buried in pod describe that tips you off
kubectl describe pod <pending-pod> | grep "had volume node affinity conflict"
# "0/12 nodes are available: 8 node(s) had volume node affinity conflict"

Fix: Use volumeBindingMode: WaitForFirstConsumer on all storage classes. Delays PV creation until pod scheduling — volume always lands in the same AZ as the pod.

5. Load Balancers: Same Service Type, Three Different Outcomes

Each cloud's controller uses completely different annotation namespaces for the same intent:

annotations:
  # EKS — NLB instead of legacy Classic ELB
  service.beta.kubernetes.io/aws-load-balancer-type: "external"
  service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"

  # AKS — internal (private) load balancer
  service.beta.kubernetes.io/azure-load-balancer-internal: "true"

  # GKE — completely different namespace
  networking.gke.io/load-balancer-type: "Internal"

⚠️ We once deployed a chart to GKE with only AWS annotations. GKE ignored them and created a public Load Balancer. An internal API was internet-reachable for 40 minutes. SolarWinds monitoring caught it. It's the kind of miss that ends careers if the service is sensitive.

Rule: Validate LB visibility (internal vs public) on all three clouds in CI.

6. Autoscaling: Cluster Autoscaler Is Not the Same Everywhere

EKS scale-up path: CA detects unschedulable pods → calls ASG → ASG launches EC2 → joins cluster → kubelet registers → pods schedule. That's 4–8 minutes minimum.

GKE Node Auto Provisioning (NAP): Managed by Google, reacts faster, can create new node pools automatically. Gotcha: NAP creates pools with labels and taints you didn't ask for. Pods without matching tolerations won't schedule — even though the cluster "scaled up" in your dashboards.

What works across all three: Layer KEDA (application-level scaling, seconds) under Cluster Autoscaler (node-level, minutes). Scale pods first; nodes are the last resort.

What I'd Tell Myself Before Starting All Over

Multi-cloud Kubernetes isn't twice the work — it's about six times the work, because every subtle difference compounds.

Standardise observability first. Before you standardise deployments, standardise how you look at your clusters. Same dashboards, same alert expressions, same log structure. When the incident fires at 2 AM, you want muscle memory, not translation overhead.

Document differences, not just similarities. Your runbooks should say "on EKS, check X; on AKS, check Y; on GKE, check Z" — not pretend the clouds are the same.

Test Helm charts on all three before every major release. 20 minutes of CI is cheaper than one 4-hour incident.

Never trust "cloud-agnostic" on the label. Especially networking, storage, and identity. Those break silently rather than loudly.

The day you stop being surprised by cloud differences is the day you've actually become a multi-cloud SRE. Everything before that is just surviving the learning curve.

If you're just starting this journey: keep a war journal. Every weird behaviour, every gotcha that cost you an hour — write it down. In six months it's worth more than any certification.

Follow me on Medium for more incident stories from production.

Our EKS Node Crashed and Four Auto-Recovery Mechanisms All Failed. Here's Why.

Prateek Srivastava — Thu, 20 Aug 2026 11:56:58 +0000

⚡ TL;DR: An EKS worker node running 30+ pods crashed at 08:43 IST on Aug 16. EC2 Auto Scaling didn't act. EKS Node Auto Repair was never enabled. Cluster Autoscaler had no headroom (Min=Max=6). Our memory alert fired 82 seconds after the node was already dead. We fixed it manually at 12:30 IST — 4 hours later. Every fix is one or two CLI commands. Here's the full breakdown.

When an EKS worker node goes down, Kubernetes is supposed to handle it. That's the whole pitch — self-healing infrastructure. Pods evict, workloads reschedule, the cluster heals itself.

In a recent incident, a node running 30+ pods crashed at 08:43 IST. None of the four auto-recovery mechanisms we had in place did anything. The node sat NotReady for almost 4 hours. We discovered it 82 seconds after it died — from a memory alert that fired when the node was already gone — and fixed it manually at 12:30 IST.

The interesting part wasn't that the node crashed. It was that four different layers of protection failed for four completely different reasons.

Here's exactly what failed and why.

The Node That Looked Healthy Until It Didn't

ip-10---*.ec2.internal had been running since Jun 28. From Kubernetes' perspective, it was Ready. CPU requests at ~97%. Memory requests at ~69%. Nothing alarming.

The actual picture was very different:

Allocated resources:
  Resource    Requests       Limits
  cpu         ~97%           ~545%
  memory      ~69%           ~358%

358% memory limit overcommit. Kubernetes scheduled 37 pods there because requests showed 69% — but if those pods pushed toward their limits, the node had no chance. The requests were lies; the limits were the truth. And the truth said this node was 3.5x overloaded.

Underneath the Ready status, the containerd runtime had been in a degraded state since Jul 20, when the OOM killer had hard-killed a MongoDB process. The kernel had logged two containerd-shim deadlocks (Jul 1 and Jul 29) that the node had somehow absorbed. Neither showed up anywhere in monitoring. The control plane saw nothing wrong.

On Aug 16 at 08:22 IST, a service responsible for processing audit events started leaking memory. ELK showed node memory at 81.6% at 08:22:57 IST. By 08:43 IST the service had grown from 2.0GB to 6.2GB — a 4.2GB spike in 21 minutes — pushing the node from ~81% to ~96%.

The memory pressure triggered a third containerd-shim deadlock on the already-corrupted runtime. This time it was fatal. At 08:43:22 IST, SSM lost contact. Kubelet posted its last heartbeat. At 08:45:10, the node went NotReady.

Now watch how every auto-recovery mechanism missed it.

Failure #1 — EC2 Auto Scaling Health Check

The ASG protecting this nodegroup was configured with EC2-level health checks:

HealthCheckType: EC2
Min: 6 | Max: 6 | Desired: 6

EC2 health checks have exactly one question: is the VM powered on?

aws ec2 describe-instance-status \
  --instance-ids <id> --region <region> \
  --query 'InstanceStatuses[].[InstanceState.Name,SystemStatus.Status,InstanceStatus.Status]' \
  --output table

+---------+------+------+
|  running|  ok  |  ok  |
+---------+------+------+

The instance was running. System status: ok. Instance status: ok. The AWS hypervisor was satisfied. The deadlock inside containerd was completely invisible at the hypervisor layer — the kernel was alive, the VM was up, the NIC was responding to health probes.

EC2 health check never triggered. No replacement instance was launched.

The lesson: EC2 health checks only detect hardware failure or VM termination. They cannot detect OS-level hangs, kubelet death, containerd deadlocks, or any software failure that doesn't take down the underlying VM. For Kubernetes node health, EC2 checks are nearly useless.

Failure #2 — EKS Node Auto Repair

EKS Node Auto Repair is designed exactly for this scenario. When a node stays NotReady for a defined period, it automatically cordons, drains, and replaces it. No manual intervention needed.

aws eks describe-nodegroup \
  --cluster-name <name> \
  --nodegroup-name <nodegroup_name> \
  --query 'nodegroup.nodeRepairConfig'

null

null. Never configured. Never enabled.

This feature was available. We had just never set it up. The node sat NotReady for nearly 4 hours while the feature that would have replaced it in 10 minutes was turned off by default.

The lesson: EKS Node Auto Repair is not enabled by default. Check every nodegroup right now with the command above. If you get null, you have this gap.

Failure #3 — Cluster Autoscaler

The cluster autoscaler was running. It's supposed to handle exactly this kind of situation. But our configuration made it powerless:

Min: 6 | Max: 6 | Desired: 6

Min equals Max. The autoscaler has no headroom to operate. To replace a broken node, it needs to launch a new one first (going to 7, violating Max), drain the broken node, then terminate it (back to 6). With Max=6, it couldn't even start that sequence.

The autoscaler logs confirmed it saw the problem — and couldn't act:

I0817 12:17:57 pre_filtering_processor.go:67]
Skipping ip-*-*-*-*.ec2.internal — node group min size reached
(current: 6, min: 6)

It knew the node was bad. It had no authority to fix it.

There was a second problem: the cluster-autoscaler pod itself was running on the broken node, stuck in Terminating state alongside 25+ other pods.

kube-system  cluster-autoscaler-5********8  1/1  Terminating  0  6d5h

The thing responsible for replacing the broken node was stuck on the broken node.

The lesson: Min=Max is a zero-headroom configuration that paralyzes automated recovery. Set Max to at least Min+2 on every nodegroup.

Failure #4 — Memory Alert (Fired Too Late to Matter)

We had a memory alert configured — running on an 11-minute cron interval, watching for node memory usage above 95%.

ELK showed the node at 81.6% for hours before the incident:

IST timestamp    Node memory %
08:22:57         81.6%
08:33:57         81.6%
08:44:57         80.6%  ← node already down, stale value

The 4.2GB memory spike happened entirely between two check cycles. At 08:22 the node was fine. At 08:44 the check returned stale data because the node was already gone.

The alert fired at 08:44 IST. The node went unreachable at 08:43:22 IST. The alert was 82 seconds behind.

This is a structural problem: the metric source is the node itself. When the node dies, metrics stop. If the crash happens between alert evaluation cycles, you miss it entirely. The faster a node crashes, the more likely it dies between evaluations.

The lesson: Metric-based node alerts have a fundamental blind spot. If the node is the source of the metric, the metric disappears when the node dies.

What We're Fixing

Enable EKS Node Auto Repair (do this first)

aws eks update-nodegroup-config \
  --cluster-name <name> \
  --nodegroup-name <nodegroup_name> \
  --node-repair-config enabled=true \
  --region us-east-1

When a node stays NotReady for the configured threshold, EKS automatically cordons, drains, and replaces it. The control plane makes this decision — it doesn't depend on the broken node reporting its own failure.

Set Max > Min on Every ASG

aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name <name> \
  --min-size 6 \
  --max-size 8 \
  --region us-east-1

Max needs to be at least Min+2 to give the autoscaler room to launch a replacement before removing the broken instance.

Switch to ELB Health Checks

aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name <name> \
  --health-check-type ELB \
  --health-check-grace-period 300 \
  --region us-east-1

ELB health checks evaluate actual application-layer response, not just hypervisor state.

Replace Metric-Based Node Alert with Condition-Based Alert

# This fires even when the node is completely dead.
# kube-state-metrics reads from the API server, not from the node itself.
- alert: NodeNotReady
  expr: kube_node_status_condition{condition="Ready",status="true"} == 0
  for: 90s
  labels:
    severity: critical
  annotations:
    summary: "Node {{ $labels.node }} NotReady for >90s — EKS Auto Repair should engage"

kube-state-metrics watches the Kubernetes API server — not the node. It fires even when the node is completely unreachable.

Enforce Memory Overcommit Limits

yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: memory-ratio-limit
  namespace: your-namespace
spec:
  limits:
  - type: Container
    maxLimitRequestRatio:
      memory: "4"   # limit cannot exceed 4x request

This forces the actual memory ceiling to stay within a predictable range of what the scheduler sees. A pod with a 1Gi request cannot have a 16Gi limit.

The Broader Problem

This wasn't an unusual incident. A node accumulated damage silently for 46 days. Nothing alerted. Nothing flagged the degraded containerd state. The node reported Ready to the control plane while internally the runtime was already compromised.

When the fatal event came — a memory spike from an unrelated service — four recovery mechanisms failed in different ways:

EC2 health checks were asking the wrong question (is the VM up?)
EKS Auto Repair was never turned on
The autoscaler had no room to maneuver (Min=Max)
The metric-based alert fired 82 seconds after the node was already dead

Any one of these gaps would have slowed recovery. All four together meant 4 hours of a NotReady node and a manual reboot at noon.

The fixes are each one or two commands. None of them are complicated. They just require knowing the gaps exist.

Check Your Cluster Right Now

Run this on every nodegroup:

aws eks describe-nodegroup \
  --cluster-name <your-cluster> \
  --nodegroup-name <your-nodegroup> \
  --query 'nodegroup.nodeRepairConfig'

If you get null — you have this gap. Fix it before the next 2 AM alert.

The names, cluster/node identifiers, timestamps, instance IDs, pod counts, resource values, and other incident-specific details in this article have been intentionally changed or fictionalized for privacy and security. The failure pattern and technical lessons remain representative of the original incident.

Follow me on Medium for more incident stories from production.