DEV Community

devtocash
devtocash

Posted on Originally published at devtocash.com

Kubernetes Pod Stuck in ContainerCreating (FailedCreatePodSandBox): How to Fix It

#containers #devops #kubernetes #sre

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

What FailedCreatePodSandBox actually means

Your pod says ContainerCreating, has zero restarts, no logs, and kubectl describe is repeating a FailedCreatePodSandBox warning. What happened: the scheduler placed the pod and the node accepted it, but the kubelet couldn't build the sandbox — the pause container plus the network namespace that every container in the pod will share. Your image hasn't been pulled and your process hasn't started, because the room it runs in was never built.

Sandbox creation is a two-step handshake between the kubelet and the container runtime. The runtime (containerd) pulls the pause image and creates the sandbox container; then it calls the CNI plugin to give that namespace an IP and wire it into the cluster network. Almost every FailedCreatePodSandBox is one of those two steps failing, and the error text tells you which.

This is not the volume case. If your events say FailedMount or FailedAttachVolume, you're in the storage half of ContainerCreating — volumes get attached before the sandbox is built, so a volume failure never even reaches this stage. And if the node itself is NotReady, debug the node first; a pod can't sandbox on a node whose runtime or CNI is down. The cases below are what's left when the node is Ready, the volumes are fine, and the pod still won't come up.

Step 1: Read the sandbox error, not the ContainerCreating status 

kubectl describe pod api-7d9f-x2kq -n prod | grep -A3 FailedCreatePodSandBox
Enter fullscreen mode Exit fullscreen mode

The messages you'll actually meet, and where each one sends you:

Warning  FailedCreatePodSandBox  Failed to create pod sandbox: rpc error: code = Unknown
  desc = failed to setup network for sandbox "3f1a…": plugin type="aws-cni"
  name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
Enter fullscreen mode Exit fullscreen mode 
Warning  FailedCreatePodSandBox  Failed to create pod sandbox: rpc error: code = Unknown
  desc = failed to setup network for sandbox "3f1a…": plugin type="calico"
  failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized
Enter fullscreen mode Exit fullscreen mode 
Warning  FailedCreatePodSandBox  Failed to create pod sandbox: rpc error: code = Unknown
  desc = failed to get sandbox image "registry.k8s.io/pause:3.9": failed to pull image
Enter fullscreen mode Exit fullscreen mode 
Warning  FailedCreatePodSandBox  Failed to create pod sandbox: rpc error:
  code = DeadlineExceeded desc = context deadline exceeded
Enter fullscreen mode Exit fullscreen mode 
Warning  FailedCreatePodSandBox  Failed to create pod sandbox: rpc error: code = Unknown
  desc = failed to reserve sandbox name "web-0_prod_9c…_0": name "web-0_prod_9c…_0"
  is reserved for "a81b…"
Enter fullscreen mode Exit fullscreen mode 
Warning  FailedCreatePodSandBox  Failed to create pod sandbox: rpc error: code = Unknown
  desc = failed to get sandbox runtime: no runtime for "gvisor" is configured
Enter fullscreen mode Exit fullscreen mode

The one useful distinction: anything containing failed to setup network for sandbox is a CNI ADD failure — the sandbox container exists, but networking it failed, and the plugin name in the message (aws-cni, calico, cilium-cni, flannel) tells you whose logs to read. Everything else is a runtime failure that happened before CNI was ever called.

If the events have already rotated out, the kubelet and containerd journals on the node keep the full history:

journalctl -u kubelet --since -30m | grep -i sandbox
journalctl -u containerd --since -30m | grep -iE "sandbox|cni"
Enter fullscreen mode Exit fullscreen mode

Step 2: Confirm the failure is per-pod, per-node, or per-cluster

Before touching anything, find the blast radius — it changes the diagnosis:

kubectl get pods -A --field-selector status.phase=Pending -o wide | grep ContainerCreating
Enter fullscreen mode Exit fullscreen mode

(ContainerCreating pods report phase Pending to the API, so the field selector catches them.) Then look at the NODE column:

  • One node — that node's CNI DaemonSet pod, its IP pool, or its containerd is the problem. Fix or drain it.
  • Every node in one AZ — the subnet for that AZ is out of IPs, or a CNI change rolled out to one pool.
  • Every node — a cluster-wide CNI misconfiguration, an expired CNI credential, a registry you can't reach, or a CNI version that didn't survive the last upgrade.

For the single-node case, get the runtime's view before Kubernetes' view. crictl on the node shows sandboxes the kubelet has already given up on:

crictl pods --state NotReady           # sandboxes that were created but never became Ready
crictl inspectp <pod-sandbox-id> | jq '.status.network, .info.runtimeSpec.annotations'
Enter fullscreen mode Exit fullscreen mode

Step 3: Fix the actual cause

1. VPC CNI IP exhaustion (EKS): failed to assign an IP address to container

The single most common cause on EKS. The node still has room under max-pods, so the scheduler happily places the pod — but aws-node's IPAM daemon has no free IP to hand out. Two different pools can be empty:

The node's warm pool. The aws-node daemon pre-allocates secondary IPs per ENI; during a burst (a large rollout, a cron fan-out, a node reboot re-scheduling 60 pods at once) it can't attach ENIs and allocate IPs fast enough. This self-heals in 30–90 seconds — if the pod recovers on its own, that's what you saw. Check the pool state from the daemon's own metrics:

kubectl exec -n kube-system aws-node-x9k2l -c aws-node -- \
  curl -s localhost:61678/metrics | grep -E "awscni_(total|assigned)_ip_addresses|awscni_eni_max"
Enter fullscreen mode Exit fullscreen mode

The subnet itself. When AvailableIpAddressCount on the node's subnet hits zero, no amount of waiting helps — every new pod on every node in that subnet fails until something releases an IP:

aws ec2 describe-subnets --subnet-ids subnet-0a1b… \
  --query 'Subnets[].{id:SubnetId,az:AvailabilityZone,free:AvailableIpAddressCount}'
Enter fullscreen mode Exit fullscreen mode

A /24 gives you ~250 usable IPs for nodes, ENIs, load balancers, and pods combined — clusters outgrow that quietly. The durable fix is prefix delegation, which assigns /28 prefixes (16 IPs each) per ENI slot instead of single IPs and multiplies per-node capacity by an order of magnitude:

kubectl set env daemonset aws-node -n kube-system \
  ENABLE_PREFIX_DELEGATION=true WARM_PREFIX_TARGET=1
Enter fullscreen mode Exit fullscreen mode

Prefix delegation needs contiguous /28 blocks in the subnet, so it works best on subnets that aren't already fragmented — on a subnet that's 95% used it can fail to find a free prefix at all. For those, add a larger secondary CIDR (a /19 from the 100.64.0.0/10 range is the standard move) and put pods there via custom networking (AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true plus an ENIConfig per AZ). Either way, new capacity only applies to newly launched nodes — roll the node group afterward.

2. The CNI plugin can't reach or authenticate to its control plane

The Calico message above — connection is unauthorized: Unauthorized — is the signature of an expired or rotated service account token that the CNI config on disk still carries. Calico writes a kubeconfig into /etc/cni/net.d/calico-kubeconfig; if calico-node was down when the token rotated (or the cluster was upgraded to bound tokens), the file goes stale and every CNI ADD on that node is rejected. Restarting the CNI pod on the affected node regenerates it:

kubectl delete pod -n kube-system -l k8s-app=calico-node \
  --field-selector spec.nodeName=ip-10-0-2-114.ec2.internal
Enter fullscreen mode Exit fullscreen mode

Cilium's equivalent is unable to connect to Cilium daemon — the cilium-cni binary talks to the local agent over a Unix socket, so a crashed or restarting cilium agent pod makes every sandbox on that node fail until it's back. Either way, the debugging target is the CNI DaemonSet pod on that specific node:

kubectl get pods -n kube-system -o wide --field-selector spec.nodeName=<node> \
  | grep -Ei 'aws-node|calico|cilium|flannel'
kubectl logs -n kube-system <cni-pod> --previous | tail -50
Enter fullscreen mode Exit fullscreen mode

If it's in CrashLoopBackOff, its crash reason is your root cause. Check the plugin config and binary directories on the node too — a CNI upgrade that shipped a new plugin type name without updating the conflist, or a conflist referencing a binary missing from /opt/cni/bin, produces failed to find plugin "…" in path on every ADD:

ls -la /etc/cni/net.d/ && cat /etc/cni/net.d/10-aws.conflist
ls /opt/cni/bin/
Enter fullscreen mode Exit fullscreen mode

3. The pause image can't be pulled

failed to get sandbox image "registry.k8s.io/pause:3.9" is an image pull failure, but it doesn't show up as ImagePullBackOff, because the pause image is owned by the runtime, not by your pod spec. You hit it on air-gapped nodes, behind an egress proxy that isn't configured for containerd, or when a registry mirror goes away. The image name comes from containerd's config, and you can test the pull directly on the node:

grep sandbox_image /etc/containerd/config.toml
crictl pull registry.k8s.io/pause:3.9
Enter fullscreen mode Exit fullscreen mode

The fix is either fixing egress (containerd reads proxies from its systemd unit's Environment=HTTPS_PROXY=…, not from /etc/environment) or pointing sandbox_image at your internal mirror — which is what EKS AMIs already do with the regional ECR copy. Change it in config.toml, systemctl restart containerd, and the pending sandboxes retry on their own.

4. context deadline exceeded: the runtime is too slow to answer

A DeadlineExceeded sandbox error means containerd didn't respond to the kubelet's RunPodSandbox call inside the CRI timeout — the node's runtime is overloaded, not broken. The usual suspects are a node at 100+ pods churning through a rollout, disk I/O saturation under /var/lib/containerd, or a CNI plugin that itself hangs on a slow cloud API call (the VPC CNI attaching ENIs during an AWS API throttling event is a classic). Look at the node, not the pod:

kubectl top node <node>
iostat -x 5 3          # on the node: %util on the containerd volume
crictl info | jq '.status.conditions'
Enter fullscreen mode Exit fullscreen mode

If sandboxes are timing out alongside PLEG is not healthy in the kubelet log, the node is on its way to NotReady — cordon it and let the scheduler move work elsewhere. If it's rollout churn, slow the rollout down (maxSurge, maxUnavailable) rather than tuning runtime timeouts.

5. Leaked sandbox: name is reserved for

failed to reserve sandbox name … is reserved for "a81b…" means containerd already has a sandbox with the same pod name, namespace, and UID — usually a leftover from a kubelet restart or an earlier sandbox attempt that half-succeeded. The kubelet won't clean up what it doesn't know about, so you do it:

crictl pods --name web-0 --namespace prod        # find the stale sandbox ID
crictl stopp a81b… && crictl rmp a81b…           # stop and remove it
Enter fullscreen mode Exit fullscreen mode

The kubelet's next sync creates a fresh one. If you see this repeatedly on the same node, look at containerd's journal for why sandbox teardown is failing — a CNI DEL that errors out will strand sandboxes exactly like this, and it's the same bug family as pods stuck in Terminating.

6. RuntimeClass or security profile the node doesn't have

no runtime for "gvisor" is configured is honest: the pod asked for runtimeClassName: gvisor and containerd on that node has no [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor] section. The scheduler doesn't check runtime handlers unless the RuntimeClass declares a scheduling.nodeSelector — so add one that matches only nodes with the runtime installed:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
scheduling:
  nodeSelector:
    runtime: gvisor
Enter fullscreen mode Exit fullscreen mode

The same shape covers a seccompProfile of type Localhost referencing a file that isn't at /var/lib/kubelet/seccomp/ on the node, and an AppArmor profile not loaded there. The pod spec is asking the node for something the node's image never got; either fix the node image or stop scheduling that pod onto nodes without it.

Alert on it — a pod that never starts trips no error-rate graph

The ContainerCreating alert from the FailedMount post catches this too, since the waiting reason is the same. Add one that watches the actual capacity that runs out, before a single pod is stuck. On EKS, aws-node exposes its IPAM state to any Prometheus scrape on port 61678:

# per node: free IPs in the VPC CNI pool
awscni_total_ip_addresses - awscni_assigned_ip_addresses < 3
Enter fullscreen mode Exit fullscreen mode

Hold it for: 10m and label it by node — a single node running dry during a burst is noise, three nodes in one AZ running dry is your subnet. Pair it with a kube_pod_status_phase{phase="Pending"} count that's been rising for 15 minutes with no matching FailedScheduling events; that combination means pods are landing on nodes and dying in the sandbox stage.

A repeatable checklist

  1. kubectl describe pod and read the FailedCreatePodSandBox text. failed to setup network for sandbox = CNI ADD failure, and the plugin type names the culprit; anything else is the runtime failing before CNI. (Step 1)
  2. Scope it: one node, one AZ, or every node? Use crictl pods --state NotReady on the node for the runtime's own view. (Step 2)
  3. failed to assign an IP address: check the aws-node IP pool and the subnet's free IPs. Bursts self-heal; empty subnets need prefix delegation or a secondary CIDR, then a node-group roll. (Step 3.1)
  4. Unauthorized, unable to connect to Cilium daemon, failed to find plugin: restart or debug the CNI DaemonSet pod on that node, and verify /etc/cni/net.d and /opt/cni/bin. (Step 3.2)
  5. failed to get sandbox image: crictl pull the pause image on the node; fix containerd's proxy or sandbox_image. (Step 3.3)
  6. context deadline exceeded: node runtime overload — check I/O, pod churn, PLEG; cordon if it's heading to NotReady. (Step 3.4)
  7. name is reserved for: crictl stopp + crictl rmp the leaked sandbox. (Step 3.5)
  8. no runtime for "…" is configured: RuntimeClass with a nodeSelector so the pod only lands where the handler exists. (Step 3.6)
  9. Alert on free VPC CNI IPs per node, not just on stuck pods — the subnet runs out days before the first sandbox fails.

Related Reading

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.

Top comments (0)