Top Kubernetes Production Incident Scenarios & Diagnostic Runbooks (2026 Edition)

*Originally published on [Naveed Ahmed Tech Blog](https://blog.naveedkumbhar.com/kubernetes-scenario-interview-questions-2026/).*

In 2026, technical interview panels at high-scale tech organizations have completely abandoned academic definition questions. 

Nobody asks *"What is a Pod?"* or *"What is a DaemonSet?"* anymore.

Instead, senior candidates are placed directly into simulated production fire drills: **silent cgroup OOM kills, CoreDNS latency under traffic surge, rolling

upgrade 502 cascades, and CNI IP exhaustion**.

Below are 5 battle-tested production incident scenarios with diagnostic CLI runbooks and structured 60-second interview elevator pitches.

---

## ☸️ Scenario 1: CoreDNS Latency Spikes & 503 Errors During Traffic Surge

**The Incident:** During a marketing traffic spike, downstream microservices report intermittent `503 Service Unavailable` and `i/o timeout` connecting to

internal APIs. Pod CPU and memory are well within limits, but cluster-wide DNS response times jump from 2ms to 3.8 seconds.

### 🛠️ Diagnostic Runbook:

bash
    # 1. Check CoreDNS replica count and CPU/Memory saturation
    kubectl get deployment coredns -n kube-system -o wide
    kubectl top pods -n kube-system -l k8s-app=kube-dns

    # 2. Check CoreDNS drop rate & connection timeouts
    kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100 | grep -iE "timeout|SERVFAIL"

    # 3. Inspect pod resolv.conf search domains
    kubectl exec -it deploy/api-service -n production -- cat /etc/resolv.conf

  ### 🎯 60-Second Interview Answer:

  │ "By default, Kubernetes injects ndots:5 into container /etc/resolv.conf. When an application queries an external domain like api.stripe.com, glibc traverses up to
  │ 5 internal search domains before making the public query. This generates 4–5 unnecessary DNS lookups per call, creating a connection tracking (conntrack) race
  │ condition on Linux worker nodes and saturating CoreDNS replicas.
  │
  │ To permanently resolve this: we deploy NodeLocal DNSCache as a DaemonSet to handle lookups locally via agent UDP cache, lower ndots to 2 on chatty external
  │ callers, and configure CoreDNS horizontal pod autoscaling (cluster-proportional-autoscaler) tied to cluster node count."
  👉 Practice this scenario interactively on Interview Hub https://interview.naveedkumbhar.com/#scenario-k8s-coredns-starvation
  ──────
  ## 🚀 Scenario 2: Rolling Node Group Upgrade Triggers 502 Cascades

  The Incident: During a zero-downtime rolling node upgrade on AWS EKS, ingress controllers and users report bursts of 502 Bad Gateway. The application deployment has
  replicas: 10, maxUnavailable: 10%, and passing health checks.

  ### 🛠️ Diagnostic Runbook:

    # 1. Check PodDisruptionBudget policies
    kubectl get pdb -A

    # 2. Inspect graceful shutdown & preStop hooks
    kubectl get deployment payment-service -n production -o yaml | grep -A 10 lifecycle

    # 3. Verify kube-proxy iptables sync delay
    kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=50

  ### 🎯 60-Second Interview Answer:

  │ "When a node is drained, the kubelet simultaneously sends SIGTERM to the application pod while the control plane updates the Service Endpoints slice. However,
  │ propagating endpoint deletions across all worker node iptables/IPVS tables takes 2 to 5 seconds.
  │
  │ If the container terminates immediately upon SIGTERM, in-flight traffic routed by lagging worker nodes hits closed ports, triggering 502s. We fix this by
  │ introducing a preStop hook (sleep 5) to allow iptables rules to synchronize across the mesh before the process terminates, combined with an adequate
  │ terminationGracePeriodSeconds and an ingress retry policy."

  👉 Practice this scenario interactively on Interview Hub https://interview.naveedkumbhar.com/#scenario-eks-upgrade-zero-downtime
  ──────
  ## 🌐 Scenario 3: Pods Stuck in ContainerCreating (AWS VPC CNI IP Exhaustion)

  The Incident: A sudden autoscaling event spins up 60 pods, but they sit indefinitely in ContainerCreating. Describing the pod shows: FailedCreatePodSandBox: failed
  to assign an IP address to container.

  ### 🛠️ Diagnostic Runbook:

    # 1. Check available IPs in the target VPC subnets
    aws ec2 describe-subnets --subnet-ids subnet-0123456789abcdef0 \
      --query "Subnets[*].[SubnetId,AvailableIpAddressCount]" --output table

    # 2. Inspect AWS VPC CNI DaemonSet logs on affected worker node
    kubectl logs -n kube-system -l k8s-app=aws-node --tail=100 | grep -i "no ip addresses"

    # 3. Inspect ENI allocations per worker node
    kubectl describe node <worker-node> | grep -A 5 "Allocated resources"

  ### 🎯 60-Second Interview Answer:

  │ "The AWS VPC CNI assigns native private IPv4 addresses directly to pods from the EC2 instance's VPC subnet. If pod density increases or worker node subnets have
  │ small CIDR blocks, available IPs run dry even though EC2 compute and memory are abundant.
  │
  │ Our production remediation: enable AWS CNI Prefix Delegation (ENABLE_PREFIX_DELEGATION=true) allowing each network interface slot to attach a /28 IPv4 prefix (16
  │ IPs) instead of single IPs, configure custom networking with secondary non-routable CIDRs (such as 100.64.0.0/16 Carrier-Grade NAT) exclusively for pods, or
  │ migrate to dual-stack IPv6."

  👉 Practice this scenario interactively on Interview Hub https://interview.naveedkumbhar.com/#scenario-aws-cni-ip-exhaustion
  ──────
  ## 💥 Scenario 4: Silent OOMKilled (Exit Code 137) Without High Memory Alerts

  The Incident: A Java microservice pod abruptly restarts with ExitCode: 137. However, Datadog/Prometheus graphs show memory usage was hovering at only 70% of the
  container limit.

  ### 🛠️ Diagnostic Runbook:

    # 1. Check previous container termination status and exit code
    kubectl get pod <pod-name> -n production -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

    # 2. Inspect node-level kernel dmesg for Linux OOM invocation
    kubectl debug node/<node-name> -it --image=busybox -- chroot /host dmesg -T | grep -i "killed process"

    # 3. Check cgroup memory limits vs JVM heap config
    kubectl exec -it <pod-name> -n production -- env | grep -iE "JAVA_OPTS|MAX_RAM"

  ### 🎯 60-Second Interview Answer:

  │ "Exit code 137 indicates SIGKILL (128 + 9), triggered by the Linux kernel OOM killer. Prometheus samples metrics at intervals (e.g. every 15–30s). A rapid memory
  │ spike or off-heap memory leak (such as direct byte buffers, Netty allocations, or thread metaspace) can blow past the cgroup v2 limit in milliseconds between
  │ scraping intervals.
  │
  │ We resolve this by setting -XX:MaxRAMPercentage=75.0 so JVM reserves 25% for native OS/JIT overhead, enabling cgroup OOM kill metrics (container_oom_events_total),
  │ and capturing heap dumps on OOM using -XX:+HeapDumpOnOutOfMemoryError directed to an ephemeral volume."

  👉 Practice this scenario interactively on Interview Hub https://interview.naveedkumbhar.com/#scenario-k8s-jvm-oomkill-tuning
  ──────
  ## ⚙️ Scenario 5: Multi-Tenant CPU Throttling Despite 30% CPU Utilization

  The Incident: An API service experiences p99 latency degradation from 45ms to 1,200ms. CPU usage metrics report only 30% utilization of allocated requests and
  limits, but customers experience severe sluggishness.

  ### 🛠️ Diagnostic Runbook:

    # 1. Check CFS (Completely Fair Scheduler) throttling percentage
    kubectl top pods -n production -l app=api-service

    # 2. Query Prometheus for CFS quota throttled periods
    # rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) * 100

    # 3. Inspect pod CPU limits in container spec
    kubectl get deploy api-service -n production -o jsonpath='{.spec.template.spec.containers[0].resources}'

  ### 🎯 60-Second Interview Answer:

  │ "Kubernetes enforces CPU limits using Linux CFS (Completely Fair Scheduler) quota with a default 100ms period. A multi-threaded application (like Go runtime or
  │ Node.js worker pools) might consume its allocated 100ms quota within the first 15ms of a time slice during brief request bursts, leaving all threads frozen for
  the
  │ remaining 85ms.
  │
  │ Even though the 1-minute averaged CPU metric shows only 30% utilization, the process suffers severe latency throttling. The production best practice: avoid hard
  │ CPU limits on latency-sensitive services, rely on right-sized CPU requests with HPA scaling, or tune CFS quota periods."

  👉 Practice this scenario interactively on Interview Hub https://interview.naveedkumbhar.com/#scenario-k8s-cfs-throttling-tuning
  ──────
  ## 📚 Essential SRE & DevOps Resources

  Explore more real-world production incident drills, architectural deep-dives, and automated platforms:

  • 🛠️ DevOps & SRE Production Interview Hub https://interview.naveedkumbhar.com/ — 950+ scenario-based incident playbooks, real-world troubleshooting guides, and
  practice drills.
  • ☸️ Kubernetes Mastery Hub https://k8s.naveedkumbhar.com/ — 24 structured interactive modules with guided labs and architectural deep-dives.
  • ⚡ The Platform & Cloud Dispatch https://news.naveedkumbhar.com/ — Free bi-weekly newsletter: direct architectural notes, real post-mortems, and automation
  runbooks.
  ──────
  │ 💬 What was the hardest production Kubernetes incident you've had to triage under pressure? Drop your war stories in the comments below!


    ---