DEV Community

devtocash
devtocash

Posted on • Originally published at devtocash.com

Kubernetes Pod Stuck in Pending (FailedScheduling): How to Fix It

#kubernetes #containers #devops #sre

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

What Pending actually means

A pod stuck in Pending has been accepted by the API server but has no node to run on. The kube-scheduler looked at every node in the cluster, none satisfied the pod's constraints, and it gave up for now — recording exactly why in a FailedScheduling event. Unlike a crash, nothing is wrong with your container yet. The image is never pulled, the process never starts. The problem is entirely about placement.

This is a different failure from the "my container won't stay up" family — ImagePullBackOff, CrashLoopBackOff, and OOMKilled all mean a node accepted the pod and then something broke. Pending means no node would take it in the first place. The good news: like the others, the scheduler tells you the exact reason. You never have to guess. This is the sequence I run to place a stuck pod, usually in a couple of minutes.

Step 1: Read the scheduler's verdict

Don't theorize from a Pending status. Read the event the scheduler already wrote:

kubectl describe pod payments-api-7d9f4c8b6-xk2mn
Enter fullscreen mode Exit fullscreen mode

Scroll to Events at the bottom:

  Warning  FailedScheduling  default-scheduler
    0/5 nodes are available: 2 Insufficient cpu,
    2 node(s) had untolerated taint {dedicated: gpu},
    1 node(s) didn't match Pod's node affinity/selector.
Enter fullscreen mode Exit fullscreen mode

That single line accounts for every node in the cluster and why each one rejected the pod. The scheduler evaluates nodes through filter predicates, and the message is the tally of which predicate failed where. Map the phrase to a root cause:

Message fragment Root cause Go to
Insufficient cpu / Insufficient memory Requests don't fit any node's free capacity Step 2
node(s) had untolerated taint Node is cordoned or reserved; pod lacks a toleration Step 3
didn't match Pod's node affinity/selector nodeSelector/affinity points at labels no node has Step 4
had volume node affinity conflict / unbound ... PersistentVolumeClaims Storage can't bind, or PV is zone-locked away from capacity Step 5
too many pods Node hit its max-pods cap (often ENI/IP limits) Step 6
didn't match pod topology spread constraints / anti-affinity Spread/anti-affinity rules can't be satisfied Step 7

Read this first and everything below collapses to one path. Often you'll see several fragments at once — fix them in order of how many nodes each blocks.

Step 2: Insufficient CPU or memory

The most common cause. Insufficient cpu doesn't mean the node is using all its CPU — it means the sum of pod requests already scheduled there leaves less than your pod asks for. Scheduling is based on requests, not live usage. A node at 10% actual CPU can still reject a pod if its requests are already reserved.

First, see what the pod is asking for:

kubectl get pod payments-api-7d9f4c8b6-xk2mn \
  -o jsonpath='{.spec.containers[0].resources.requests}'
Enter fullscreen mode Exit fullscreen mode 
{"cpu":"2","memory":"4Gi"}
Enter fullscreen mode Exit fullscreen mode

Then see what's actually free on the nodes:

kubectl describe node ip-10-0-1-42 | grep -A6 "Allocated resources"
Enter fullscreen mode Exit fullscreen mode 
  Resource   Requests      Limits
  cpu        3500m (87%)   6 (150%)
  memory     6Gi (78%)     10Gi
Enter fullscreen mode Exit fullscreen mode

If free CPU is below your request on every node, you have two fixes:

  • The request is oversized. A pod asking for 2 full cores that actually uses 150m is starving the scheduler of placements it could otherwise make. Right-size the request to real usage — the same discipline that keeps the Kubernetes bill down. The VPA in recommendation mode will suggest values from observed usage; see how it fits alongside HPA and KEDA in the pod autoscaling guide.
  • The cluster is genuinely full. Every node is legitimately packed. You need more capacity — enable the cluster autoscaler (or Karpenter) so a Pending pod that can't fit triggers a new node instead of waiting forever. Confirm it's reacting: 
kubectl -n kube-system logs -l app=cluster-autoscaler --tail=50 | grep -i scale
Enter fullscreen mode Exit fullscreen mode

If the autoscaler logs pod didn't trigger scale-up: max node group size reached, you've hit your node-group ceiling — raise the max, or the pod stays Pending indefinitely.

Step 3: Untolerated taints

A message like node(s) had untolerated taint {node.kubernetes.io/unschedulable} means the target nodes are tainted and your pod carries no matching toleration. Taints repel pods unless the pod explicitly tolerates them. See the taint:

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
Enter fullscreen mode Exit fullscreen mode

Two very different situations:

  • The node is cordoned. node.kubernetes.io/unschedulable appears after kubectl cordon (or a drain during maintenance). If the cordon was intentional, that's expected — the pod schedules once you kubectl uncordon the node or new capacity arrives. A pod stuck Pending during a node upgrade is usually this.
  • The node is deliberately reserved — GPU nodes, a dedicated=gpu:NoSchedule taint, spot-instance pools. If your pod should run there, add a toleration: 
spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"
Enter fullscreen mode Exit fullscreen mode

A toleration lets a pod land on a tainted node; it does not force it there. Pair it with a nodeSelector or affinity (Step 4) when you want the pod to actually target that pool.

Step 4: Node affinity / selector mismatch

didn't match Pod's node affinity/selector means your nodeSelector or nodeAffinity names a label that no available node carries. Usually a typo, a decommissioned node pool, or a label that was never applied. Check what the pod demands:

kubectl get pod payments-api-7d9f4c8b6-xk2mn -o jsonpath='{.spec.nodeSelector}'
Enter fullscreen mode Exit fullscreen mode 
{"disktype":"ssd","topology.kubernetes.io/zone":"us-east-1a"}
Enter fullscreen mode Exit fullscreen mode

Then check which nodes actually have those labels:

kubectl get nodes -l disktype=ssd
Enter fullscreen mode Exit fullscreen mode

If that returns nothing, no node matches. Either the label is wrong on the pod, or it was never set on the nodes:

kubectl label node ip-10-0-1-42 disktype=ssd
Enter fullscreen mode Exit fullscreen mode

A subtle trap: pinning a pod to a single zone with topology.kubernetes.io/zone means it can only schedule in that zone. If that zone is full and your autoscaler scales a different zone, the pod stays Pending forever. Prefer preferredDuringScheduling over requiredDuringScheduling unless the constraint is truly hard — a preference degrades gracefully instead of deadlocking.

Step 5: Unbound volumes and zone conflicts

pod has unbound immediate PersistentVolumeClaims or volume node affinity conflict means storage is blocking placement. Storage and scheduling are coupled: a pod can only run where its volume can attach. Check the PVC:

kubectl get pvc
Enter fullscreen mode Exit fullscreen mode 
NAME            STATUS    VOLUME   CAPACITY   STORAGECLASS
data-payments   Pending                       gp3
Enter fullscreen mode Exit fullscreen mode

A Pending PVC means no PersistentVolume satisfied the claim. Common causes:

  • No matching StorageClass or no dynamic provisioner. kubectl get storageclass — if there's no default class and the PVC names none, nothing provisions the volume. Set a default or name a valid class.
  • volume node affinity conflict — the PV already exists in us-east-1a, but the only node with free capacity is in us-east-1b, and an EBS volume can't cross zones. Fix by setting volumeBindingMode: WaitForFirstConsumer on the StorageClass so the volume is provisioned after the scheduler picks the node, in the same zone — instead of binding first and pinning the pod to the wrong zone.

Step 6: Node is out of pod slots (too many pods)

too many pods means the node hit its max-pods limit even though it has spare CPU and memory. On AWS EKS with the VPC CNI, the ceiling is often driven by how many IP addresses the instance's ENIs can hold — a small instance type may cap at 8–17 pods regardless of how much RAM it has. Check the node's capacity and current count:

kubectl get node ip-10-0-1-42 -o jsonpath='{.status.capacity.pods}'
kubectl get pods --all-namespaces --field-selector spec.nodeName=ip-10-0-1-42 \
  --no-headers | wc -l
Enter fullscreen mode Exit fullscreen mode

If the count equals capacity, the node is full of pods, not resources. Fixes: use larger instance types (more ENIs = more IPs = more pods), enable prefix delegation on the VPC CNI to pack far more IPs per node, or add nodes. This is an easy one to miss because kubectl top node shows the node as nearly idle.

Step 7: Topology spread and anti-affinity deadlocks

didn't match pod topology spread constraints or an anti-affinity rejection means your own placement rules can't be satisfied. A classic self-inflicted deadlock: a podAntiAffinity with requiredDuringScheduling that says "no two replicas on the same node" while you have 5 replicas and only 3 nodes. The 4th and 5th can never place.

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:   # hard rule — deadlocks
      - topologyKey: kubernetes.io/hostname
Enter fullscreen mode Exit fullscreen mode

Switch required to preferred so replicas spread when possible but still schedule when they can't, or add nodes so the hard rule is satisfiable. The same reasoning applies to topologySpreadConstraints with whenUnsatisfiable: DoNotSchedule — that's a hard gate; ScheduleAnyway degrades gracefully.

A repeatable checklist

When a pod is stuck Pending, run this in order:

  1. kubectl describe pod <pod> → read the FailedScheduling event; note every N node(s) ... fragment.
  2. Insufficient cpu/memory → right-size requests or enable the cluster autoscaler; check it isn't at max node-group size (Step 2).
  3. untolerated taint → uncordon the node, or add the matching toleration (Step 3).
  4. didn't match node affinity/selector → fix the label on the pod or apply it to nodes; avoid hard zone pins (Step 4).
  5. unbound PVC / volume node affinity conflict → fix the StorageClass; set WaitForFirstConsumer (Step 5).
  6. too many pods → node hit its IP/pod cap; bigger instances, prefix delegation, or more nodes (Step 6).
  7. Topology/anti-affinity rejection → relax required/DoNotSchedule to preferred/ScheduleAnyway (Step 7).

Don't let Pending pods hide

A pod that can't schedule is invisible to a health check that only watches running pods — it never crashes, it just quietly never runs. That's how a scaled-up deployment silently serves at half capacity. Alert on pods stuck Pending past a threshold:

kube_pod_status_phase{phase="Pending"} == 1
  and on(pod, namespace) (time() - kube_pod_created) > 300
Enter fullscreen mode Exit fullscreen mode

Wire that into the Prometheus + Grafana monitoring setup so a five-minute-stuck pod pages before a rollout stalls, and bake the seven-step triage into your incident runbook. A cluster that's chronically full and can't place pods is one of the quiet Kubernetes mistakes that cost companies millions — cheap to catch, expensive to discover during a traffic spike.

Related Reading

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.

Top comments (0)