DEV Community

aiunplugged
aiunplugged

Posted on Originally published at aiunplugged.in

Kubernetes Pods stuck in Pending — 8 causes and exact fixes

#kubernetes #debugging #devops #k8

A Pod stuck in Pending means Kubernetes cannot schedule it onto any node. The Pod is created but no container is running. Here are the 8 real causes ranked by frequency, with exact diagnostic commands.

First — always run this 

kubectl describe pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

Scroll to the Events: section at the bottom. The scheduler explains WHY it can't place the Pod. Every fix below starts from this event message.

Cause 1 — Insufficient CPU or memory on nodes

Event message:

0/3 nodes are available: 3 Insufficient cpu.
Enter fullscreen mode Exit fullscreen mode

The Pod requests more resources than any node has free.

Diagnose:

kubectl top nodes
kubectl describe nodes | grep -A 5 "Allocated resources"
Enter fullscreen mode Exit fullscreen mode

Fix:

  • Reduce Pod's resources.requests.cpu / memory if over-requested
  • Scale cluster (add nodes)
  • Free capacity by removing unused Deployments

Cause 2 — Node selector or affinity doesn't match

Event message:

0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector.
Enter fullscreen mode Exit fullscreen mode

Pod requires nodes with specific labels but none have them. (The scheduler uses the same NodeAffinity plugin for both nodeSelector and nodeAffinity, hence the combined message.)

Diagnose:

kubectl get nodes --show-labels
kubectl get pod <pod-name> -o yaml | grep -A 5 nodeSelector
Enter fullscreen mode Exit fullscreen mode

Fix:

  • Add the required label to a node: kubectl label node <node> disktype=ssd
  • Or remove the nodeSelector from Pod spec if not needed

Cause 3 — Taints without matching tolerations

Event message:

0/3 nodes are available: 3 node(s) had untolerated taint.
Enter fullscreen mode Exit fullscreen mode

Nodes have taints (e.g. NoSchedule) that Pod doesn't tolerate.

Diagnose:

kubectl describe nodes | grep -A 2 Taints
Enter fullscreen mode Exit fullscreen mode

Fix:

  • Add matching toleration to Pod spec: 
tolerations:
- key: "special"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"
Enter fullscreen mode Exit fullscreen mode
  • Or remove the taint from node: kubectl taint node <node> special-

Cause 4 — PersistentVolumeClaim not bound

Event message:

persistentvolumeclaim "data-pvc" not found
Enter fullscreen mode Exit fullscreen mode

or

0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
Enter fullscreen mode Exit fullscreen mode

Pod references a PVC that doesn't exist or has no matching PV.

Diagnose:

kubectl get pvc
kubectl describe pvc <pvc-name>
Enter fullscreen mode Exit fullscreen mode

Fix:

  • Create the missing PVC
  • Check StorageClass exists: kubectl get storageclass
  • Verify dynamic provisioning is working (CSI driver installed)

Cause 5 — Image pull secret missing (private registry)

Event message:

Failed to pull image: unauthorized
Enter fullscreen mode Exit fullscreen mode

Pod's image is in a private registry, no credentials configured. Note: kubectl get pods will show STATUS as ErrImagePull or ImagePullBackOff for this case — the Pod's .status.phase is technically still Pending, but readers watching the STATUS column won't see the word "Pending." Include this in your triage when a Pod visibly appears not Pending but is stuck.

Diagnose:

kubectl get pod <pod-name> -o yaml | grep imagePullSecrets
kubectl get secrets | grep regcred
Enter fullscreen mode Exit fullscreen mode

Fix:

kubectl create secret docker-registry regcred \
  --docker-server=<registry-url> \
  --docker-username=<user> \
  --docker-password=<pass>
Enter fullscreen mode Exit fullscreen mode

Add to Pod spec:

spec:
  imagePullSecrets:
  - name: regcred
Enter fullscreen mode Exit fullscreen mode

Cause 6 — Pod anti-affinity conflicts

Event message:

0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules.
Enter fullscreen mode Exit fullscreen mode

Pod has anti-affinity requiring separation from other Pods, but no eligible node.

Diagnose:

kubectl get pod <pod-name> -o yaml | grep -A 20 affinity
kubectl get pods -o wide
Enter fullscreen mode Exit fullscreen mode

Fix:

  • Add more nodes (so anti-affinity has room to spread)
  • Relax anti-affinity from requiredDuringScheduling to preferredDuringScheduling

Cause 7 — Namespace ResourceQuota rejects Pod creation

Important distinction: unlike causes 1-6 and 8, a ResourceQuota violation prevents the Pod object from being created at all. The API server's admission controller rejects the create request with 403 Forbidden, so there's no Pod stuck in Pending — the Pod never exists. Include this in your triage when kubectl create / kubectl apply returns an error and no Pod appears in kubectl get pods.

Error message (from the CLI, not the Pod events):

Error from server (Forbidden): pods "foo" is forbidden: exceeded quota: <quota-name>, requested: ..., used: ..., limited: ...
Enter fullscreen mode Exit fullscreen mode

Diagnose:

kubectl describe quota -n <namespace>
kubectl apply -f pod.yaml   # observe the Forbidden error text
Enter fullscreen mode Exit fullscreen mode

Fix:

  • Delete unused Pods in namespace to free quota
  • Increase quota: kubectl edit quota <quota-name>
  • Deploy to a different namespace with headroom

Cause 8 — Scheduler broken or overloaded

Rare, but happens on stressed clusters.

Diagnose:

kubectl get pods -n kube-system | grep scheduler
kubectl logs -n kube-system <scheduler-pod-name>
Enter fullscreen mode Exit fullscreen mode

Fix:

  • Restart scheduler pod
  • Check for scheduler configuration issues
  • On managed K8s (EKS/GKE), this usually resolves itself

The universal debug checklist

Copy this and run through it every time:

# Basic status
kubectl get pod <pod-name>

# Detailed events (THE most important)
kubectl describe pod <pod-name>

# Node status
kubectl get nodes
kubectl describe nodes

# Recent events across cluster
kubectl get events --sort-by='.lastTimestamp' | tail -20

# Related resources
kubectl get pvc
kubectl get storageclass
kubectl get secrets
Enter fullscreen mode Exit fullscreen mode

Reproduce this yourself (browser sandbox)

Free Kubernetes playground: https://killercoda.com/playgrounds/scenario/kubernetes

Create a pod that will be Pending:

kubectl run stuck-pod --image=nginx --overrides='{"apiVersion":"v1","spec":{"nodeSelector":{"nonexistent":"label"}}}'
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl describe pod stuck-pod
Enter fullscreen mode Exit fullscreen mode

Screenshot the Events section. Note the "didn't match node selector" message.

Clean up:

kubectl delete pod stuck-pod
Enter fullscreen mode Exit fullscreen mode

Prevention

Set sensible defaults from the start:

  • Resource requests — not too high, not too low. Start with 100m CPU, 128Mi memory. Adjust based on actual usage.
  • No nodeSelector unless truly needed. Default schedulers work well.
  • Test PVCs before deploying real workloads — verify storage class works.
  • Use kubectl top regularly to see node capacity trends.

Bottom line

Every Pending pod has a specific reason. The scheduler always tells you what it is via kubectl describe. Match the event message to one of the 8 causes above. Apply the corresponding fix. Done in 5 minutes.

Top comments (0)