The day the same Helm chart behaved differently on three clouds in the same hour was the day I stopped trusting "cloud-agnostic" as a real thing.
Nobody plans to manage Kubernetes on three clouds at the same time. It happens gradually — an acquisition brings an Azure footprint, a new business unit commits to GCP, and you already run EKS. Then one morning a page fires and you're staring at three terminal windows, three different cluster behaviours, one very angry incident bridge.
I've spent the better part of two years in exactly that situation. What follows isn't a comparison chart you can find on any vendor blog. It's the stuff I learned the hard way — the silent differences that don't show up until something breaks in production at 2 AM.
📟 The Incident That Started It All
We deployed the same Helm chart — identical values, same image tag — to EKS, AKS, and GKE within the same 20-minute release window. EKS came up healthy. AKS came up healthy. GKE's pods started, ran for about 90 seconds, then began OOMKilling in a loop. Same YAML. Same limits. Three different outcomes. Four hours to find why. That story is lesson #3.
1. Networking: The Biggest Lie — "Pods Are Just Pods"
This is where most multi-cloud pain originates. Every Kubernetes tutorial tells you pods get their own IPs and can talk to each other. What it doesn't tell you is that where those IPs come from completely changes your blast radius, subnet planning, security posture, and the failure modes you'll face at 3 AM.
EKS: Your pods eat your VPC subnet IPs
EKS uses the AWS VPC CNI by default. Every pod gets a real, routable IP from your VPC subnet. This sounds fine until you have a /24 subnet and someone deploys a service with 200 replicas. The error isn't "out of IPs" — it's an ENI attachment timeout.
The harder gotcha: each EC2 instance type has a max ENI count × max IPs per ENI cap. A t3.medium can hold 3 ENIs × 6 IPs = 18 pod IPs max, minus 1 for the node. Pods sit Pending on nodes with plenty of CPU and RAM — the constraint is invisible to the scheduler.
# Check ENI and IP capacity on a node
kubectl describe node <node-name> | grep -A5 "Allocatable"
# Check aws-node daemonset for ENI attachment errors
kubectl logs -n kube-system -l k8s-app=aws-node --tail=50 | grep -i "err\|fail\|exhaust"
Fix: Enable VPC CNI prefix delegation — one ENI prefix = 16 IPs instead of 1.
AKS: Azure CNI pre-allocates IPs whether you like it or not
AKS with Azure CNI reserves IPs per node based on --max-pods (default 30) — before any pod runs. A 100-node cluster pre-consumes 3,000 VNet IPs. Use Azure CNI Overlay mode if IP-constrained.
GKE: Clean until you need to peer networks
GKE uses alias IP ranges, isolated from your main VPC. Clean story — until you peer your GKE VPC with on-prem and discover the pod CIDR overlaps your on-prem range, because you let GKE auto-assign ranges six months ago. Always explicitly define secondary ranges.
| EKS | AKS | GKE | |
|---|---|---|---|
| Pod IP source | VPC subnet (real IPs) | VNet or overlay | Alias IP ranges |
| IP exhaustion risk | High — ENI limits | High — pre-allocated | Low — isolated |
| Common 3 AM failure | ENI attachment timeout | VNet IP exhaustion | CIDR overlap during peering |
2. IAM + RBAC: Three Identity Models, One Misconfiguration Away from Disaster
EKS: The IRSA silent fallback trap
If your IRSA annotation has a typo, the assume-role call silently fails and the pod falls through to the node's instance profile — often over-permissive. You won't see an error. Always verify:
kubectl exec -it <pod-name> -- aws sts get-caller-identity
I once spent three hours debugging why a pod had too much S3 access. A typo in the annotation. Silent fallback to the node's IAM role. The pod happily worked — with the wrong permissions.
AKS: Workload Identity federation — fragile on setup
Three pieces must align exactly: Managed Identity, federated credential (exact OIDC issuer URL, including trailing slashes), and the Kubernetes service account annotation. One mismatch = silent 401.
GKE: Node rotation breaks implicit identity
Enabling Workload Identity on a node pool removes the node's default Google service account. Any workload relying on implicit node identity breaks on the next node rotation. Audit before enabling.
3. Memory + cgroups: Why the Same Container OOMKilled on GKE but Not on EKS
Here's the opening incident. Same image, same limits, EKS and AKS healthy, GKE OOMKilling every 90 seconds.
GKE had moved to containerd with cgroup v2 on newer node images. EKS and AKS were still on cgroup v1.
Our Java service used JVM ergonomics to auto-detect heap size:
-
cgroup v1: JVM reads from
/sys/fs/cgroup/memory/memory.limit_in_bytes -
cgroup v2: JVM reads from
/sys/fs/cgroup/memory.max
Our old JVM didn't handle cgroup v2. It read host memory (64GB) instead of the container limit (2GB), allocated an 8GB heap into a 2GB container, and OOMKilled within 90 seconds of starting.
# Check which cgroup version the container sees
kubectl exec -it <pod-name> -- cat /proc/1/cgroup
# cgroup v1: "12:memory:/kubepods/..."
# cgroup v2: single line "0::/"
# Check what heap the JVM actually allocated
kubectl exec -it <pod-name> -- java -XshowSettings:all -version 2>&1 | grep -i heap
# See OOMKill events across all namespaces
kubectl get events --field-selector reason=OOMKilling -A --sort-by='.lastTimestamp'
Fix: JDK 15+ (native cgroup v2 support) or JDK 11 with -XX:+UseContainerSupport. Always explicitly set -Xmx and -Xms. This isn't just a Java problem — Go's GOMAXPROCS and Python's multiprocessing.cpu_count() have the same pattern.
| EKS | AKS | GKE | |
|---|---|---|---|
| Default node OS | Amazon Linux 2023 | Ubuntu 22.04 / Azure Linux | Container-Optimized OS |
| cgroup version | v2 (AL2023), v1 (AL2) | v2 (Ubuntu 22.04+) | v2 (COS since mid-2022) |
4. Storage: The AZ-Pinning Trap
Block storage (EBS / Azure Disk / Persistent Disk) is AZ-specific on every cloud. Your PVC and the pod using it must be in the same AZ.
The scenario I've watched happen three times:
Stateful pod uses a PVC in AZ-A. Traffic grows. Cluster Autoscaler spins up nodes in AZ-B and AZ-C. New pods stay Pending — PVC is in AZ-A, the new nodes aren't. On-call engineer stares at CPU and memory graphs seeing nothing wrong.
# The event buried in pod describe that tips you off
kubectl describe pod <pending-pod> | grep "had volume node affinity conflict"
# "0/12 nodes are available: 8 node(s) had volume node affinity conflict"
Fix: Use volumeBindingMode: WaitForFirstConsumer on all storage classes. Delays PV creation until pod scheduling — volume always lands in the same AZ as the pod.
5. Load Balancers: Same Service Type, Three Different Outcomes
Each cloud's controller uses completely different annotation namespaces for the same intent:
annotations:
# EKS — NLB instead of legacy Classic ELB
service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
# AKS — internal (private) load balancer
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
# GKE — completely different namespace
networking.gke.io/load-balancer-type: "Internal"
⚠️ We once deployed a chart to GKE with only AWS annotations. GKE ignored them and created a public Load Balancer. An internal API was internet-reachable for 40 minutes. SolarWinds monitoring caught it. It's the kind of miss that ends careers if the service is sensitive.
Rule: Validate LB visibility (internal vs public) on all three clouds in CI.
6. Autoscaling: Cluster Autoscaler Is Not the Same Everywhere
EKS scale-up path: CA detects unschedulable pods → calls ASG → ASG launches EC2 → joins cluster → kubelet registers → pods schedule. That's 4–8 minutes minimum.
GKE Node Auto Provisioning (NAP): Managed by Google, reacts faster, can create new node pools automatically. Gotcha: NAP creates pools with labels and taints you didn't ask for. Pods without matching tolerations won't schedule — even though the cluster "scaled up" in your dashboards.
What works across all three: Layer KEDA (application-level scaling, seconds) under Cluster Autoscaler (node-level, minutes). Scale pods first; nodes are the last resort.
What I'd Tell Myself Before Starting All Over
Multi-cloud Kubernetes isn't twice the work — it's about six times the work, because every subtle difference compounds.
Standardise observability first. Before you standardise deployments, standardise how you look at your clusters. Same dashboards, same alert expressions, same log structure. When the incident fires at 2 AM, you want muscle memory, not translation overhead.
Document differences, not just similarities. Your runbooks should say "on EKS, check X; on AKS, check Y; on GKE, check Z" — not pretend the clouds are the same.
Test Helm charts on all three before every major release. 20 minutes of CI is cheaper than one 4-hour incident.
Never trust "cloud-agnostic" on the label. Especially networking, storage, and identity. Those break silently rather than loudly.
The day you stop being surprised by cloud differences is the day you've actually become a multi-cloud SRE. Everything before that is just surviving the learning curve.
If you're just starting this journey: keep a war journal. Every weird behaviour, every gotcha that cost you an hour — write it down. In six months it's worth more than any certification.
Follow me on Medium for more incident stories from production.
Top comments (0)