Top Kubernetes Production Incident Scenarios & Diagnostic Runbooks (2026 Edition)

Naveed Ahmed — Tue, 15 Sep 2026 06:15:00 +0000

Originally published on Naveed Ahmed Tech Blog.

In 2026, technical interview panels at high-scale tech organizations have completely abandoned academic definition questions. 

Nobody asks *"What is a Pod?"* or *"What is a DaemonSet?"* anymore.

Instead, senior candidates are placed directly into simulated production fire drills: **silent cgroup OOM kills, CoreDNS latency under traffic surge, rolling

upgrade 502 cascades, and CNI IP exhaustion**.

Below are 5 battle-tested production incident scenarios with diagnostic CLI runbooks and structured 60-second interview elevator pitches.

---

## ☸️ Scenario 1: CoreDNS Latency Spikes & 503 Errors During Traffic Surge

**The Incident:** During a marketing traffic spike, downstream microservices report intermittent `503 Service Unavailable` and `i/o timeout` connecting to

internal APIs. Pod CPU and memory are well within limits, but cluster-wide DNS response times jump from 2ms to 3.8 seconds.

### 🛠️ Diagnostic Runbook:


bash
    # 1. Check CoreDNS replica count and CPU/Memory saturation
    kubectl get deployment coredns -n kube-system -o wide
    kubectl top pods -n kube-system -l k8s-app=kube-dns

    # 2. Check CoreDNS drop rates and lookup latency
    kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50 | grep -i "timeout\| SERVFAIL"

    # 3. Inspect ndots setting in pod resolv.conf
    kubectl exec -it <app-pod> -- cat /etc/resolv.conf

  ### 🎙️ 60-Second Elevator Pitch:
  │ *"By default, Kubernetes injects ndots:5 into container /etc/resolv.conf. When an application queries an external domain like api.stripe.com, the resolver first
  │ traverses up to 5 internal search domains before making the public query. This multiplies query volume 4x–5x, saturating CoreDNS replicas during traffic spikes.
  │
  │ We resolve this by deploying NodeLocal DNSCache as a DaemonSet to handle local lookups, setting ndots:2 on workloads querying external endpoints, and configuring
  │ horizontal pod autoscaling for CoreDNS based on cluster node count."*

  👉 Practice this scenario interactively on Interview Hub https://interview.naveedkumbhar.com/
  ──────
  ## ⚡ Scenario 2: Rolling Node Group Upgrade Triggers 502 Bad Gateway Outages

  The Incident: While performing a rolling upgrade of an Amazon EKS managed node group, the ingress controller generates thousands of 502 Bad Gateway errors for 3
  minutes. All deployments have replicas: 5 and maxUnavailable: 25%.

  ### 🛠️ Diagnostic Runbook:

    # 1. Inspect Pod Disruption Budgets (PDB)
    kubectl get pdb -A

    # 2. Check pod termination grace period and preStop lifecycle hooks
    kubectl get deployment <app> -o yaml | grep -A 8 lifecycle

    # 3. Check kube-proxy iptables synchronization status
    kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=100 | grep -i "error"

  ### 🎙️ 60-Second Elevator Pitch:

  │ *"When a node drains, the kubelet simultaneously sends SIGTERM to container processes and notifies the API server to remove the pod from Endpoints. However,
  │ iptables/eBPF routing rules across other worker nodes take 2 to 5 seconds to synchronize. If the container process shuts down immediately, incoming in-flight
  │ traffic is routed to a dead container, generating 502s.
  │
  │ We achieve zero downtime by adding a preStop sleep hook (e.g. sleep 5) allowing routing tables across the cluster to drain before the process shuts down, paired
  │ with proper readinessProbe gates and terminationGracePeriodSeconds."*
  👉 Practice this scenario interactively on Interview Hub https://interview.naveedkumbhar.com/
  ──────
  ## 🌐 Scenario 3: EKS Pods Stuck in ContainerCreating (AWS VPC CNI IP Exhaustion)

  The Incident: An HPA auto-scaling event triggers 50 new pods to handle customer load, but all newly created pods remain permanently stuck in ContainerCreating with
  the event: FailedCreatePodSandBox: failed to assign an IP address to container.
  ### 🛠️ Diagnostic Runbook:

    # 1. Check available IP addresses in worker subnets
    aws ec2 describe-subnets --subnet-ids <subnet-id> \
      --query "Subnets[0].AvailableIpAddressCount"

    # 2. Check AWS VPC CNI L-IPAM daemon logs
    kubectl logs -n kube-system -l k8s-app=aws-node --tail=100 | grep -i "failed to allocate"

    # 3. Inspect ENI allocation on the worker node
    kubectl describe node <node-name> | grep -A 5 "Allocated resources"

  ### 🎙️ 60-Second Elevator Pitch:

  │ *"The AWS VPC CNI allocates secondary private IPv4 addresses from the node's subnet directly to each pod. In dense clusters, subnets quickly exhaust available IPs.
  │
  │ We mitigate this by enabling VPC CNI Prefix Delegation (ENABLE_PREFIX_DELEGATION=true), which assigns /28 IPv4 blocks per ENI slot (16 IPs per slot) instead of
  │ individual IPs, significantly increasing pod density per node while reducing IP fragmentation, alongside configuring secondary VPC CIDRs dedicated exclusively to
  │ pod networking."*
  ──────
  ## 🛑 Scenario 4: The Silent OOMKilled Pod (Exit Code 137 Without Logs)

  The Incident: A high-throughput service exits intermittently with exit code 137. The application log buffer reveals zero errors or stack traces right before the
  process terminates.

  ### 🛠️ Diagnostic Runbook:

    # 1. Check pod termination state reason
    kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'

    # 2. Inspect kernel dmesg on the host node for cgroup oom-killer invocation
    kubectl describe node <node-name> | grep -i oom
    dmesg -T | grep -i -E 'oom[-_]killer|killed process'

    # 3. Check container memory limits vs JVM / Node heap headroom
    kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].resources.limits.memory}'

  ### 🎙️ 60-Second Elevator Pitch:

  │ *"Exit code 137 means a process was terminated by SIGKILL (128 + 9). When no application error logs exist, it indicates the Linux kernel cgroup OOM-Killer
  │ terminated the process from outside the container runtime because memory usage exceeded container resources.limits.memory.
  │
  │ To resolve this, we ensure the application runtime (e.g. JVM -XX:MaxRAMPercentage or Node --max-old-space-size) is configured with 25% headroom below the cgroup
  │ limit to account for non-heap native buffers and OS thread stacks."*
  ──────
  ## 🛡️ Scenario 5: Kubernetes Split-Brain & Etcd Quorum Loss

  The Incident: A 3-node master control plane loses a single node due to an AWS AZ network partition. Suddenly, kubectl commands time out with etcdserver: leader
  changed or context deadline exceeded.

  ### 🛠️ Diagnostic Runbook:

    # 1. Check etcd cluster health and member list
    etcdctl endpoint health --cluster --cacert=/etc/kubernetes/pki/etcd/ca.crt \
      --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key

    # 2. Check leader election status and raft index drift
    etcdctl endpoint status --cluster -w table

    # 3. Check disk write latency on etcd storage
    iostat -xz 1 5

  ### 🎙️ 60-Second Elevator Pitch:

  │ *"Etcd relies on the Raft consensus algorithm, requiring a strict majority quorum ((n/2) + 1). In a 3-node cluster, quorum is 2. If a node partitions and the
  │ remaining nodes experience disk I/O latency exceeding heartbeat thresholds (default 10ms), leader election loops occur.
  │
  │ We resolve and prevent this by dedicating high-IOPS NVMe storage (such as AWS io2 or local SSDs) to /var/lib/etcd with fdatasync latency under 10ms, tuning Raft
  │ election timeouts for cross-AZ topologies, and running 5-member control planes for multi-AZ clusters."*
  ──────
  ## 🚀 Explore the Complete Open-Source Ecosystem

  All of these scenarios and diagnostic playbooks are part of an open-source initiative to help DevOps engineers and SREs prepare for real-world production
  challenges:

  • 🧠 Interactive Interview Hub (970+ Scenarios) https://interview.naveedkumbhar.com/ — Practice mode with active recall, domain filtering across AWS, Kubernetes,
  Terraform, Docker, and Linux.
  • 📦 GitHub Repository (Star the Repo) https://github.com/naveedkumbhar/devops-production-interview-handbook — Full open-source scenario handbook on GitHub.
  • ☸️ Kubernetes Mastery Path https://k8s.naveedkumbhar.com/ — 24-module hands-on curriculum with interactive quizzes and local Minikube sandboxes.
  • ⚡ The Platform & Cloud Dispatch https://news.naveedkumbhar.com/ — Free bi-weekly newsletter: direct architectural notes, real post-mortems, and automation
  runbooks.
  ──────
  💬 What was the hardest production Kubernetes incident you've had to triage under pressure? Drop your war stories in the comments below!

DEV Community: Naveed Ahmed

Top Kubernetes Production Incident Scenarios & Diagnostic Runbooks (2026 Edition)