Your Pods Are Running. They Will Never Be Ready.

Over on r/kubernetes, in a crosspost titled “CNI - NETWORK POLICY”, an engineer was watching pods sit happily in Running while the Ready count stayed stubbornly at 0/1, restarts ticked upward, and screenshots of manifests flew back and forth looking for a fix.

Picture the scene — invented, but you have lived a version of it. It is 2 AM. The deployment rolled out cleanly an hour ago, green ticks all the way down the pipeline. But kubectl get pods tells a passive-aggressive story: every pod shows Running, the READY column refuses to budge past 0/1, and the restart counter is quietly climbing. What is the app team doing? Exactly what app teams always do first: tailing application logs, hunting for a bug that does not exist. They are blaming the furniture when the floor under their feet has vanished.

First, read the symptom properly

Kubernetes does not have one big bucket called “healthy”. It hands out two very different health verdicts, and you must know which one is talking to you.

The first is the readiness probe: “may this pod take traffic right now?” If it fails, Kubernetes does not panic. It marks the pod unready and pulls its address out of the Service’s endpoints list (the list of pod addresses a Service sends traffic to). Nothing restarts. The container keeps humming in the corner, denied any work.

The second verdict belongs to the liveness probe, which asks a harsher question: “is this process wedged beyond repair?” If it fails, the kubelet (the agent on each node that runs pods) kills the container and restarts it. Two verdicts, two completely different punishments — and your terminal output is already telling you which story you are in, if you read the columns instead of skimming them.

• Running, 0/1 Ready, RESTARTS stays 0: The readiness probe is failing, but the liveness probe is fine. The container is alive, but it cannot pass its readiness checks to receive traffic.

• Running, RESTARTS climbing: The liveness probe is failing too, or the application process is crashing internally, throwing an exit code, and dying inside its own health endpoint.

• Whole namespace goes unready at the same minute: This is never an application bug. Microservices written by twenty different developers do not coordinate a synchronized strike. Apps do not break in formation. Something underneath the floor changed.

• Only pods on one node are sick: Stop staring at the deployment manifest. Suspect that specific node’s CNI agent, not the workload scheduled on top of it.

• Pods fine, Service still dead: The pods are Ready and endpoints are populated, yet nobody can connect. The path itself, not the pod, is broken.

Check the floor before the furniture

Before you touch the deployment, look at the CNI (Container Network Interface — the plugin system that gives pods their network; here, Calico). If the fabric cannot wire a pod, application logs are a waste of chai.

When Calico is installed via the Tigera operator, its core machinery lives quietly inside the calico-system namespace. There are three moving parts you must look at immediately: calico-node (the agent that programs routes and policy on that node; runs as a DaemonSet with one pod on every node), calico-typha (the middleman that fans out datastore events to the agents so your API server does not melt), and calico-kube-controllers. Run these three checks before you do anything else:

kubectl get pods -n calico-system
kubectl get tigerastatus
kubectl get nodes -o wide

The golden rule: if a calico-node pod is sitting at 0/1 or restarting, every workload pod on that node inherits the sickness — no interfaces configured properly, no policy updates, no traffic moving.

Next, the BGP check. Calico nodes exchange routes using BGP (Border Gateway Protocol — the internet’s route-sharing protocol). If a session drops, routes vanish from the node’s table, and packets set out for a destination nobody can reach.

calicoctl node status

Every peer should show Established. The classic error is BIRD is not ready: BGP not established (BIRD is Calico’s route speaker) — that node cannot reach a peer, full stop. The fix order never changes: node first, CNI agent second, workload last.

Test the DNS path, because it is always DNS

The old operations joke says it is never DNS. Inside a cluster, it is DNS often enough that you check it early and feel no shame. Cluster DNS (the name-to-address phone book) is CoreDNS: pods labelled k8s-app=kube-dns behind a Service named kube-dns in kube-system.

Consider how readiness checks usually work. An app rarely just says “I am here” — its /ready endpoint tries to reach its database at a name like postgres.db.svc.cluster.local. If the pod cannot resolve that name, the check times out, the probe reports failure, the kubelet marks the pod unready, and the app looks completely guilty. But the app did not fail; the phone book was stolen.

Do not guess. Prove it directly from inside the network path using a disposable throwaway pod:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl run netprobe --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default.svc.cluster.local
kubectl exec <your-pod> -- cat /etc/resolv.conf

Healthy output: nslookup resolves the name in milliseconds, no retries, no timeouts — and /etc/resolv.conf shows its nameserver line pointing at the ClusterIP of the kube-dns Service (for example 10.96.0.10 — the exact address depends on your cluster’s service range). If nslookup hangs and complains that no servers could be reached, while the CoreDNS pods sit pretty and Running in kube-system, stop debugging CoreDNS. The packets are being dropped on the floor between your pod and CoreDNS. Which brings us to the policies.

Audit the policies: allow-list arithmetic

Here is where many platform engineers shoot their own feet: NetworkPolicy rules in Kubernetes are an allow-list, and only an allow-list. There are no explicit deny rules. There is no rule priority or ordering. By default, every pod in a cluster is non-isolated; it can talk to everything, and everything can talk to it. The very second any NetworkPolicy selects a pod for egress (outbound traffic), that pod flips its behavior instantly to: “nothing leaves this container except what an applied policy explicitly allows.” If three policies select the pod, the allowed rules simply add together.

Here is the trap people walk into every week: a well-meaning default-deny egress policy lands in the namespace. Somebody adds an allow policy for the app’s ports — 8080 to the database, 443 to the outside APIs. Nobody adds DNS.

Now the nuance most posts botch: a NetworkPolicy normally cannot block the kubelet’s health check itself. Probe traffic originates from the node, and traffic to and from a pod’s own node is always allowed — the kubelet’s knock reaches the door just fine. What the policy strangles is everything the app tries to call while answering that knock. The app reaches for the database name, the DNS packet is dropped by the egress rules, the connection hangs, the probe times out, the pod goes unready. The probe failed honestly. The application is innocent.

The minimal, battle-tested fix is to explicitly permit egress to CoreDNS. Here is the exact YAML you need to drop into the namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: app
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Audit your policies cleanly. Run kubectl get networkpolicy -A to list every policy active across the cluster, and run kubectl describe networkpolicy <name> -n <namespace> on the suspects. And remember the golden rule of allow-list arithmetic: both sides must allow a conversation. The sender must have an egress rule that permits the packet out, and the receiver must have an ingress rule that permits the packet in. If either side forgets, the packet dies silently.

Know which dataplane is actually moving your packets

Before you run around tracing packets, you must know what dataplane is running under the hood. The dataplane is simply the machinery in the kernel that actually moves and filters packets.

Traffic heading to a Kubernetes Service — remember, a Service is just a stable virtual address sitting in front of shifting pod addresses — has to be intercepted and directed to an actual pod IP. In standard clusters, kube-proxy handles this translation using one of three modes: iptables (rule lists written into the kernel firewall; the long-time default), IPVS (an enterprise-grade in-kernel load balancer designed for massive scale), or nftables (the modern replacement for iptables in newer Linux kernels).

Calico adds its own layer of complexity to this mix. Calico’s standard dataplane can enforce policies using standard Linux iptables, it offers an nftables mode, and it also features a native eBPF dataplane (eBPF = small programs the kernel runs directly on packets as they pass through network hooks). When Calico runs in eBPF mode, it handles Services natively and replaces kube-proxy entirely.

Why it matters at 2 AM: the same NetworkPolicy manifest is executed by entirely different machinery depending on the mode, and a mode change mid-incident rewrites the floor while you are standing on it. Inspection commands do not transfer either — iptables -L shows you a comforting empty list on a cluster happily running the eBPF dataplane. So check the kube-proxy configuration or the Tigera Installation resource and name your dataplane before you read a single dropped packet.

The two quiet killers: an empty pool and a wrong-sized pipe

If agents are running, DNS is responsive, and policies are completely open, you are usually dealing with one of two subtle platform failures.

The first quiet killer is IPAM (IP address management — the bookkeeping of who got which pod address). Calico hands out pod addresses from configured blocks called IP pools. When a pool runs dry, new pods get no address and freeze in ContainerCreating, while kubectl describe pod shows events complaining the CNI plugin failed to allocate an IP.

calicoctl ipam show

The output will show you total addresses, allocations, and whether your CIDR block is at 100% capacity. The fixes are straightforward: grow the IP pool by adding an additional CIDR block to the cluster, or clean up leaked and orphaned IP reservations left behind by dead, terminated pods.

The second quiet killer is MTU (maximum transmission unit — the largest packet the path carries in one piece). Under the hood, overlay networks wrap every pod packet inside an outer packet to send it between nodes. VXLAN adds about 50 bytes of overhead, IPIP eats about 20 bytes, and WireGuard encryption adds roughly 60 bytes. Because of this outer wrapping, the internal MTU inside the pod must be strictly smaller than the physical MTU of the underlying host network.

Set the pod MTU wrong — say, left at 1500 on a 1500-byte physical network that still has to carry the wrapping — and you get the spookiest symptom in networking: small packets sail through, big packets vanish without a sound. Ping works. DNS works. The TCP handshake completes. Then the app sends a real payload, or a fat TLS certificate, and the connection hangs half-open forever — while somebody swears the network is fine because “ping works.”

kubectl exec <pod> -- ping -M do -s 1452 <other-pod-ip>

The -M do flag sets the Don’t Fragment bit. Walk the size parameter down from 1452 bytes until the ping successfully receives replies. Once you know your path’s real ceiling, update the MTU field inside the Tigera operator’s Installation custom resource to match reality. Calico will roll out the corrected MTU across your nodes, and large packets will stop vanishing into the ether.

The data-plane-first checklist

Pin this above the runbook. Work it top to bottom; do not skip to the deployment.

1. Read the symptom: Restarts climbing means liveness is failing; stuck at 0/1 with zero restarts means readiness is failing; a whole namespace dying at once means infrastructure broke.

2. Check calico-node on every node: Run kubectl get pods -n calico-system; one sick agent makes one node’s worth of pods sick.

3. Check tigerastatus and BGP: Ensure all peers show Established in calicoctl node status, or routes are missing and packets fall into holes.

4. Prove DNS from inside a pod: Run an nslookup from a throwaway pod inside the cluster, not from your laptop’s terminal.

5. Read resolv.conf: Check /etc/resolv.conf inside the container; the pod must point directly at the kube-dns Service address.

6. List every NetworkPolicy: Run kubectl get networkpolicy -A; assume the newest policy is guilty until proven innocent.

7. Confirm DNS egress exists: Verify an explicit allow rule for port 53 (both UDP and TCP) to kube-dns in every namespace that contains a default-deny policy.

8. Name your dataplane: Know whether you are running iptables, IPVS, nftables, or eBPF before trying to interpret a single dropped packet.

9. Check the IP pool: Run calicoctl ipam show; an exhausted pool masquerades as a container scheduling issue, but it is purely an IPAM shortage.

10. Test the MTU: Run ping -M do at descending packet sizes; if small packets pass and large payloads hang, your network pipe is the wrong size.

Running is a statement about a process. Ready is a statement about a network.

Debug the floor, not the furniture

Watch how threads like that one always go: the deployment gets rolled. Then rolled again, just in case. Teams always do this — it is human nature to turn the key again when the engine sputters. But every rollout lands on the exact same networking floor, produces the exact same 0/1 Ready symptom, and burns debugging time while the platform layer sits there holding the actual answer.

Make it a reflex: pods Running but never Ready means your first commands target calico-system, kube-system and the NetworkPolicies — and the deployment stays untouched until the floor is proven solid.

Staging warning: test network policies in a staging namespace first, with the same default-deny you plan for production — a policy that has never met DNS egress in staging will meet it at 2 AM in production.

---

© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

Kubernetes, DevOps, SRE, and platform engineering — explained the practical way.

Which layer was the culprit the last time your pods got stuck in 0/1 Ready limbo — a missing DNS egress rule, a silent MTU mismatch, or a sick CNI agent? Let me know in the comments below.

Examples are illustrative — validate in your own environment before running in production.

Disclaimer

• This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.

• I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.

• Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.

• Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.

• Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.