Why Does kubectl Say "created" When Nothing Is Running?

You run kubectl apply -f deployment.yaml. Half a second later: deployment.apps/web created. Nothing you deployed exists yet. No ReplicaSet, no Pod, no container, no IP.

"Created" means one thing: a record was written to etcd. Everything after that happens across 11 components, and most of them never talk to each other directly. They all watch the API server and react to what the previous one wrote.

kubectl ──► API server ──► etcd            "created" is printed here
                │
   watches ◄────┘  the controllers below only talk through the API server
                │
  Deployment controller ──► ReplicaSet
  ReplicaSet controller ──► 3 Pods (Pending, no node)
  scheduler             ──► Pod gets a nodeName
  kubelet (on the node) ──► container runtime ──► CNI plugin (IP)
                            (these three call each other directly)
                            init containers, then app containers
  readiness probe       ──► Pod is Ready
  EndpointSlice ctrl    ──► endpoint flips to ready
  kube-proxy            ──► every node can route to it
                                              first request lands here

kubectl to etcd (steps 1 to 3)

1. kubectl checks whether web exists. If not, it sends a create. If so, it diffs your file, the live object, and the last-applied-configuration annotation, and sends a patch.
2. The API server runs four gates in order: authentication, authorization, mutating admission (sidecar injectors live here), then validation and validating admission. Any gate can reject the request.
3. etcd confirms the write once a majority of its members have it. kubectl prints created. Your part is done.

Controllers and the scheduler (steps 4 to 6)

1. The Deployment controller creates a ReplicaSet named web-<pod-template-hash>. That hash is how rollouts and rollout undo tell versions apart.
2. The ReplicaSet controller creates the Pods in doubling batches (1, 2, 4) and stops at the first error, so a broken template gives you one error instead of hundreds.
3. The scheduler filters nodes by the pod's requests (actual usage is ignored), taints, and affinity, scores the rest, and binds the pod to one. No fit? The pod stays Pending with 0/5 nodes are available: 3 Insufficient cpu.

On the node (steps 7 to 9)

1. The kubelet sees a pod bound to its node and waits for volumes to attach and mount.
2. The container runtime (via CRI) creates the pod sandbox, the tiny "pause" container that holds the pod's network. Then it runs init containers one at a time, pulling each image just before it starts, then pulls and starts your app containers.
3. The CNI plugin is called while the sandbox is created and gives the pod its IP. Until that works, you're in ContainerCreating.

Running is not Ready

kubectl get pods now says Running. Still no traffic.

The kubelet runs your readiness probe (every 10 seconds by default) and marks the container ready on the first success. No readiness probe? Then "started" counts as Ready, and traffic can hit an app that is still booting.

Traffic (steps 10 and 11)

1. The EndpointSlice controller has listed the pod's IP since it got one, marked not ready. When the pod turns Ready, it flips that endpoint to ready.
2. kube-proxy on every node watches EndpointSlices and updates its iptables or nftables rules. Each node does this on its own, so for a moment some nodes route to the new pod and some don't.

Now a request can arrive.

Where it gets stuck

This table is the reason I wrote the post. The symptom tells you which step to look at.

What you see	Which step	First thing to check
kubectl apply errors	2. API server	RBAC, a webhook that rejected or timed out, a policy
Deployment exists, no pods	4 or 5. Controllers	kubectl describe replicaset: quota or Pod Security rejections
Pending	6. Scheduler	kubectl describe pod events: Insufficient cpu/memory, taints, affinity
ContainerCreating forever	7 to 9. Kubelet, CNI, volumes	Volume attach errors, CNI out of IPs, a missing Secret or ConfigMap volume
ImagePullBackOff	8. Runtime	Image name or tag, registry credentials, rate limits
CrashLoopBackOff	8. Your app	kubectl logs --previous: it started and exited
Running but 0/1 ready	Readiness	Wrong probe path or port, or a slow start
Ready, but no traffic	10 or 11. Endpoints	Service selector doesn't match pod labels, or wrong targetPort

One more thing: put rollout status in your pipeline

kubectl apply returning 0 means the request was accepted. It does not mean anything is running. If new pods never become Ready, the Deployment waits 600 seconds (progressDeadlineSeconds), marks itself as not progressing, and rolls nothing back on its own.

kubectl rollout status deployment/web is the command that actually waits for steps 4 to 11 to finish. If your CI only runs apply, your pipeline goes green before your app exists.

This is part of a series on what Kubernetes does on its own clock. The other direction is Why Does My Pod Take Exactly 30 Seconds to Die?. There's an illustrated version with diagrams of each step on Medium. I used AI tools to help research and draft this; every step above is checked against the Kubernetes source and docs.

Which row in that table has cost you the longest debugging session, and what finally gave it away?

Comments

[Noah Grinstein]

The one that bites the most people: a Service selector of app: web-api while the pods are labeled app: webapi. Every pod Ready, endpoints empty, and the ingress gets blamed first. kubectl get endpointslices shows it in one line. What's yours?