Open a terminal on your workstation, point your environment at a production cluster, and execute a single declarative command:
$ kubectl apply -f deployment.yaml
deployment.apps/web-service configured
Within four seconds, three containers running your application have materialized across three separate physical servers in different availability zones. They have private IP addresses, health checks, CPU limits, shared storage volumes, and internal load-balancer endpoints routing traffic to them.
To the developer typing in the terminal, it feels like an imperative remote procedure call: Kubernetes, go spin up my containers right now.
The reality is fundamentally different.
There is no central orchestrator pushing containers onto servers. The kubectl binary does not talk to your worker nodes. The servers running your workload never received a command telling them to start your app. In fact, if every single control plane node suddenly lost power thirty seconds after you executed that command, your containers would continue running completely uninterrupted.
Kubernetes is not an imperative management tool. It is an asynchronous distributed state engine built on declarative consensus, optimistic concurrency control, and continuous reconciliation loops.
When you run kubectl apply, you are not starting software. You are recording a desire in a distributed database, and triggering a cascade of autonomous control loops that continuously drag reality toward your declared intent.
Here is the mechanical breakdown of every step that unfolds between your keystroke and a live container.
1. The End-to-End Lifecycle: An Overview
Before diving into each subsystem, examine the complete path your manifest travels across the cluster:
The journey divides into five distinct stages:
-
Client Preparation:
kubectlperforms schema discovery and computes a 3-way strategic merge patch before sending an HTTP request. -
The API Server Gauntlet:
kube-apiserverruns the payload through authentication, authorization, mutating webhooks, schema validation, and validating webhooks before committing toetcd. - Controller Reconciliation: The Deployment Controller detects the updated spec and creates a ReplicaSet, which then generates individual Pod manifests with unassigned node names.
-
Scheduling: The
kube-schedulerwatches for pending Pods, filters out unsuitable nodes, scores the survivors, and writes a binding back to the API server. -
Node Execution: The
kubeletdaemon on the target worker node observes the assignment, directs the Container Runtime Interface (CRI) to launch a Pause container, configures networking via CNI, and starts your application containers.
Let us inspect the gears of each stage.
2. Client-Side Mechanics: The 3-Way Strategic Merge Patch
Long before any network packet leaves your laptop, kubectl executes critical validation logic.
When you execute kubectl apply, the CLI does not blindly upload your YAML file. First, it queries the cluster Discovery API (/openapi/v2 or /openapi/v3), caching the cluster's OpenAPI schema locally in ~/.kube/cache/discovery/. It validates that your fields exist, your indentation is valid, and your types match the cluster version.
Next, kubectl solves a fundamental distributed systems dilemma: concurrent modifications.
Imagine you apply a deployment with replicas: 3. An hour later, a Horizontal Pod Autoscaler (HPA) scales the deployment to replicas: 10. If you subsequently edit the container image in your local YAML and run kubectl apply, should your command overwrite replicas back to 3, or should it preserve the HPA's 10 replicas?
To solve this without locking resources, Kubernetes implements a 3-Way Strategic Merge Patch using three distinct data sources:
- The Local Configuration: The YAML file on your workstation disk.
- The Live Cluster State: The current object as recorded in the cluster database.
-
The Last-Applied Configuration: A serialized JSON snapshot stored inside an annotation on the live object:
kubectl.kubernetes.io/last-applied-configuration.
[ 1. Local YAML ]
│
├── (Diff) ──┐
│ ▼
[ 3. Last-Applied JSON ] ───► [ Strategic Merge Patch Engine ]
│ ▲
├── (Diff) ──┘
│
[ 2. Live State ]
The patch engine calculates two separate diffs:
- What changed between your previous local file and your current local file?
- What changed between your previous local file and the live cluster state?
If a field (like replicas) was modified in the live cluster but was not modified in your local file, kubectl preserves the live cluster value. If a field was modified in your local file, kubectl updates it. If a field was deleted from your local file that existed in the last-applied annotation, kubectl removes it from the cluster.
Once the patch is computed, kubectl serializes it into a JSON Patch payload and issues an HTTP PATCH request to the API server endpoint:
PATCH /apis/apps/v1/namespaces/default/deployments/web-service
Content-Type: application/strategic-merge-patch+json
3. Inside kube-apiserver: The Admission Control Gauntlet
The kube-apiserver is the front door of Kubernetes. It is also completely stateless. It does not run controllers, it does not schedule pods, and it does not manage containers. Its sole responsibility is acting as a secure, transactional gateway to the cluster database.
Every request that reaches kube-apiserver must survive six sequential gates:
Stage 1: Authentication (Who are you?)
The API server inspects the incoming TLS handshake and HTTP headers. It passes the credentials through a chain of configured authenticators:
- X.509 Client Certificates: Often used for cluster administrators (extracting the Common Name as username and Organization as group).
- OpenID Connect (OIDC): Bearer tokens issued by providers like Okta, Google, or Keycloak.
- ServiceAccount Tokens: Signed JWTs issued by the cluster token controller.
The first authenticator that successfully validates the request attaches an identity struct (User: alice, Groups: ["devs", "system:authenticated"]) to the request context. If all authenticators fail, the request terminates immediately with 401 Unauthorized.
Stage 2: Authorization (Are you allowed to do this?)
Next, the identity passes to the Role-Based Access Control (RBAC) engine. The authorizer extracts four attributes:
- User/Group: Who is making the request?
-
Verb: What action are they taking? (
get,list,create,update,patch,delete) -
Resource: Which object type is targeted? (
deployments,pods,services) - Namespace: In which administrative boundary?
The engine evaluates all RoleBinding and ClusterRoleBinding objects. If no rule grants explicit permission for that user, verb, and resource in that namespace, the request is rejected with 403 Forbidden.
Stage 3: Mutating Admission Webhooks (Modifying the object)
If authorized, the request enters the admission control chain. This phase begins with Mutating Admission Controllers.
These plugins can modify the object before it is stored. For example:
- Setting default values for omitted fields (e.g., setting
imagePullPolicy: IfNotPresent). - Injecting service mesh sidecars: When you deploy an application into an Istio-enabled namespace, a mutating webhook catches your Pod spec and injects an Envoy proxy container alongside your application container.
Because mutating webhooks can modify the object, they run in a loop until the object stabilizes.
Stage 4: Object Schema Validation
Once mutations settle, the API server performs strict schema validation against the internal Golang struct definitions of the target resource. If a field contains an invalid type (such as a string where an integer is expected) or violates regex constraints, the request fails with 422 Unprocessable Entity.
Stage 5: Validating Admission Webhooks (Policy enforcement)
The modified, validated object now reaches Validating Admission Controllers.
Unlike mutating controllers, validating controllers are strictly read-only decision gates. Tools like Open Policy Agent (OPA) Gatekeeper or Kyverno inspect the final object against organization security policies:
- Does this container try to run as root (
runAsNonRoot: false)? - Does the container image originate from an approved corporate registry?
- Did the developer declare mandatory CPU and memory resource limits?
If any validating webhook rejects the object, the entire transaction aborts.
Stage 6: Persistence in etcd
If the object passes all five gates, the API server serializes it into Protocol Buffers (protobuf) format and commits it to the distributed key-value store, etcd.
The API server responds to your terminal with HTTP 200 OK or HTTP 201 Created.
At this exact moment, your kubectl apply command exits with a success message. Yet on your servers, not a single container has been launched yet.
4. The Source of Truth: etcd, Raft Quorum, and Resource Versions
Every piece of state in a Kubernetes cluster lives in etcd. If etcd is healthy, the cluster can rebuild itself from scratch. If etcd is lost without backups, your cluster state is gone forever.
etcd is a strongly consistent, distributed key-value store that coordinates using the Raft consensus algorithm.
[ etcd Node 1 ] (Leader)
├── Raft Heartbeat (RPC) ──► [ etcd Node 2 ] (Follower)
└── Raft Heartbeat (RPC) ──► [ etcd Node 3 ] (Follower)
In a production 3-node or 5-node etcd cluster, one node is elected Leader. When the API server writes your Deployment object, the Leader appends the entry to its Raft log and replicates it to the follower nodes. The write is only committed once a quorum (a majority: $(N/2) + 1$) acknowledges the log entry. This is why production clusters always run an odd number of control plane nodes (3 nodes tolerate 1 failure, 5 nodes tolerate 2 failures).
Optimistic Concurrency Control (resourceVersion)
Databases like PostgreSQL handle concurrent updates with row-level locks or transaction isolation levels. In a distributed control plane where dozens of controllers inspect the same object simultaneously, database locks would cause crippling bottlenecks.
Kubernetes avoids locks entirely using Optimistic Concurrency Control (OCC).
Every object stored in etcd has a mandatory metadata attribute: metadata.resourceVersion. This is a monotonically increasing integer tracked by etcd.
metadata:
name: web-service
namespace: default
resourceVersion: "4829104"
When a controller reads an object, it records the resourceVersion. When it sends an update, the API server checks if the incoming resourceVersion matches the current version in etcd.
If another controller updated the object in the intervening millisecond, the version numbers do not match. The API server immediately rejects the write with 409 Conflict. The rejected controller does not crash; it simply re-reads the fresh state, re-evaluates its calculation, and retries.
The Watch API: Eliminating Polling
In early distributed systems, clients constantly polled the database to check for updates (GET /pods every 2 seconds). With thousands of pods, polling rapidly saturates CPU and network bandwidth.
Kubernetes replaces polling with the etcd Watch API, exposed through kube-apiserver over persistent HTTP/2 gRPC streams.
When a controller connects, it requests a Watch stream starting at a specific resourceVersion:
Watch(resource="pods", startingResourceVersion="4829104")
The API server holds this connection open and pushes compact change events (ADDED, MODIFIED, DELETED) in real time as commits occur in etcd.
5. The Controller Manager: Level-Triggered Reconciliation Loops
If the API server is just a database gateway, what actually makes things happen?
The answer is kube-controller-manager: a single binary executing dozens of independent, concurrent reconciliation loops.
A reconciliation loop follows a simple, infinite loop:
$$\text{Observe Current State} \longrightarrow \text{Compare with Desired State} \longrightarrow \text{Act to Eliminate the Diff}$$
Level-Triggered vs Edge-Triggered Systems
To understand why Kubernetes is exceptionally reliable, you must understand the difference between edge-triggered and level-triggered architectures:
- Edge-Triggered (Message-Driven): Reacts to state transitions. If a message says "Add 1 Pod," you add 1 pod. If a network blip drops that message, your system falls out of sync permanently.
- Level-Triggered (State-Driven): Reacts to current levels. It does not care what happened in the past; it only cares about the current snapshot. It asks: "The user declared 3 pods. Right now, there are 2 pods. Therefore, I will create 1 pod."
Because Kubernetes is level-triggered, it is naturally self-healing. If a network partition drops twenty events, the controller does not get confused. On the next reconciliation pass, it looks at the live level, spots the discrepancy, and repairs it.
The client-go Informer Pipeline
Controllers never read directly from etcd. If fifty controllers queried etcd on every loop iteration, etcd would collapse under read pressure.
Instead, every controller uses the client-go Informer library:
-
Reflector: Establishes a
ListWatchconnection to the API server. On startup, it lists all objects once, then watches for updates. - DeltaFIFO: A queue that receives change events and deduplicates them.
- Indexer (Local Cache): An in-memory cache holding the controller's view of cluster objects. When a controller needs to read an object, it reads from this local RAM cache in microseconds with zero network calls.
-
WorkQueue: A thread-safe, rate-limiting queue that holds the keys (
namespace/name) of objects that need reconciliation.
The Controller Cascade
When you apply a Deployment, a sequence of controllers execute like dominoes:
-
Deployment Controller: Watches Deployments and ReplicaSets. It spots your new Deployment in its Indexer cache. It calculates that no ReplicaSet exists matching the new pod template hash. It calls the API server:
POST /apis/apps/v1/namespaces/default/replicasets. -
ReplicaSet Controller: Watches ReplicaSets and Pods. It notices the new ReplicaSet with
replicas: 3. It checks its local cache and finds zero pods matching the ReplicaSet's label selector. -
Action: The ReplicaSet controller loops three times. In each iteration, it calls the API server:
POST /api/v1/namespaces/default/pods.
The API server validates each Pod and writes it to etcd.
At this point, three Pod objects exist in etcd. But if you inspect their YAML, you will notice a glaring missing field:
spec:
nodeName: "" # Empty! No node has been chosen.
The pods exist only as records on disk. They are unassigned.
6. The kube-scheduler: The Two-Phase Placement Engine
Enter kube-scheduler. The scheduler is another independent controller that watches for Pods where spec.nodeName is empty.
When an unscheduled Pod appears in its queue, the scheduler executes a sophisticated two-phase algorithm to find the single best physical node in the cluster:
Phase 1: Filtering (Predicates)
The scheduler takes the list of all nodes in the cluster (which could be hundreds or thousands) and evaluates hard constraints. Any node that fails a single filter is instantly disqualified.
Key filtering plugins include:
-
NodeResourcesFit: Does the node have enough unallocated CPU and RAM to satisfy the Pod'sresources.requests? -
NodeName: Did the pod request a specific node viaspec.nodeName? -
NodePorts: Does the Pod request a host port already bound on this node? -
TaintsTolerations: If the node has a taint (key=value:NoSchedule), does the Pod have a matching toleration? -
NodeAffinity: Does the node match the Pod's requirednodeSelectorornodeAffinitylabels?
If 96 out of 100 nodes fail these filters, the scheduler is left with 4 feasible candidate nodes.
Phase 2: Scoring (Priorities)
If multiple nodes survive filtering, the scheduler ranks them on a scale of 0 to 100 using scoring plugins:
-
NodeResourcesBalancedAllocation: Favors nodes where scheduling this Pod will balance the ratio of CPU and RAM usage, avoiding stranded resources. -
ImageLocality: Awards higher scores to nodes that have already downloaded the required container image layers, dramatically speeding up startup times. -
PodTopologySpread: Awards higher scores to nodes in availability zones that have fewer replicas of this application, maximizing resilience against cloud rack failures.
The scheduler computes a weighted sum across all scoring plugins. The node with the highest score wins.
The Binding Subresource
Once a winner is selected (e.g., worker-node-03), the scheduler does not log into the node. It simply issues an HTTP POST request to the API server's binding subresource:
POST /api/v1/namespaces/default/pods/web-service-7d4f-x8k2j/binding
Content-Type: application/json
{
"target": {
"kind": "Node",
"apiVersion": "v1",
"name": "worker-node-03"
}
}
The API server updates the Pod object in etcd, writing spec.nodeName: "worker-node-03".
The scheduler's job is complete.
7. The Worker Node: Kubelet, CRI, and the Secret of the Pause Container
Now, attention shifts to the physical worker node: worker-node-03.
Running on this machine is the kubelet: a native binary executing outside containers as a systemd service. The kubelet is the true bridge between Kubernetes declarations and local Linux primitives.
The kubelet runs an internal event loop called the syncLoop. Through its own Watch connection to the API server, it spots that a Pod has been assigned to its own node name (spec.nodeName == "worker-node-03").
Here is the exact sequence the kubelet executes:
Step 1: Volume Preparation
If the Pod requests Secrets, ConfigMaps, or Persistent Volume Claims, the kubelet creates a local directory in /var/lib/kubelet/pods/<pod-uid>/volumes/ and mounts the required filesystems.
Step 2: The Container Runtime Interface (CRI)
The kubelet does not talk directly to Docker. Since the deprecation of Dockershim, the kubelet communicates over a local UNIX domain socket (/run/containerd/containerd.sock) using gRPC to a CRI runtime like containerd or CRI-O.
The kubelet issues a gRPC call: RunPodSandbox.
Step 3: Spawning the Pause Container
This is one of the most brilliant architectural details in Kubernetes.
Before launching your application, the container runtime starts a tiny, specialized container using the image registry.k8s.io/pause:3.9.
The Pause container does almost nothing. Its compiled C source code is roughly 30 lines long: it installs signal handlers, calls pause(), and goes to sleep forever.
Why does it exist?
The Pause container is the anchor for the Pod's Linux namespaces.
As explored in our deep dive on container primitives, a container is simply an isolated process. A Pod is a group of containers that share the exact same Network and IPC namespaces.
When the runtime launches the Pause container, it creates a new Linux Network namespace. It assigns an IP address to that namespace (e.g., 10.244.1.42) and opens loopback (lo).
When your actual application containers start, the runtime passes a configuration directive to join the existing namespaces:
"linux": {
"namespaces": [
{ "type": "network", "path": "/proc/<pause-pid>/ns/net" },
{ "type": "ipc", "path": "/proc/<pause-pid>/ns/ipc" }
]
}
Because your main container and your sidecar container share the Pause container's network namespace:
- They share the exact same IP address (
10.244.1.42). - They can talk to each other over
localhost:8080via memory sockets. - If your main application crashes and restarts, the Pod IP address does not change, because the Pause container never crashed.
Step 4: Container Network Interface (CNI)
Before containers can communicate across nodes, the network must be plumbed.
The runtime invokes a Container Network Interface (CNI) plugin (such as Calico, Cilium, or Flannel). The CNI plugin creates a Virtual Ethernet (veth) pair:
- One end plugs into the Pause container's network namespace as
eth0. - The other end attaches to the host virtual bridge or eBPF routing hook.
- The plugin queries its IP Address Management (IPAM) pool and assigns
10.244.1.42to the interface.
Step 5: Launching Application Containers
Finally, the kubelet issues CRI gRPC calls: CreateContainer followed by StartContainer.
The runtime calls runc, allocates Linux control groups (cgroups v2) to enforce CPU and memory limits, mounts the layered root filesystem via OverlayFS, mounts the volume directories, and executes the application binary (nginx -g daemon off;).
The kubelet updates the Pod status in the API server: status.phase: Running.
Your container is now live.
8. Service Discovery: How Traffic Finds the Pod
Your pods are running with private IPs like 10.244.1.42. But pods are ephemeral; if a node crashes, the ReplicaSet creates a replacement pod with a completely different IP.
How do other services send traffic to your application reliably?
This is where Services and kube-proxy complete the picture.
When you create a Kubernetes Service with spec.type: ClusterIP, Kubernetes allocates a stable virtual IP (e.g., 10.96.0.100) from a dedicated subnet. This IP does not belong to any physical network interface in the cluster. It is a ghost IP.
On every worker node, a daemon called kube-proxy watches Services and Endpoints:
[ Client Pod ] ──► HTTP Request to 10.96.0.100:80
│
(Host Kernel)
│
[ iptables / IPVS / eBPF Hook ]
│
(Rewrites Destination IP to)
│
▼
[ Target Pod IP: 10.244.1.42:8080 ]
Historically, kube-proxy programmed Linux iptables chains. When a packet addressed to 10.96.0.100 enters the host network stack, iptables executes Destination Network Address Translation (DNAT), randomly rewriting the destination IP to one of the live Pod IPs (10.244.1.42).
In large clusters with 5,000 services, sequential $O(n)$ iptables rule evaluation causes noticeable packet latency. Modern Kubernetes clusters have largely replaced iptables with IPVS (hash-table lookups in kernel space) or eBPF (programs like Cilium running directly at the Linux socket layer), achieving line-rate packet forwarding.
9. Production Realities: Common Traps
Understanding this architecture turns mysterious Kubernetes errors into straightforward debugging steps:
The OOMKilled Exit Code 137
When a container terminates with Exit Code 137, it did not crash from a bug in your code. It hit the Linux cgroup memory.max boundary. The host kernel Out-Of-Memory (OOM) killer dispatched a SIGKILL ($128 + 9 = 137$) directly to the process.
Always distinguish between resources.requests (which the scheduler uses to filter nodes) and resources.limits (which the kubelet passes to Linux cgroups to throttle or kill processes).
The CrashLoopBackOff Loop
If your container exits with code 1 or 137, the kubelet catches the dead process. In accordance with its restartPolicy, it restarts it. If it fails again immediately, the kubelet applies an exponential backoff delay (10s, 20s, 40s... up to 300s).
The Pod is not stuck in Kubernetes; the kubelet is intentionally throttling restart frequency to prevent CPU thrashing on the host.
GitOps Drift and Merge Collisions
When teams combine automated GitOps tooling (like ArgoCD or Flux) with ad-hoc kubectl edit commands in the terminal, the 3-Way Strategic Merge Patch becomes a double-edged sword.
If an engineer patches a live field that is not declared in Git, the next automated reconciliation loop will calculate that the field does not exist in the source of truth, triggering an automatic reversion. Enforce all configuration changes through your version control pipeline.
The Mental Shift
The beauty of Kubernetes lies not in its container runtime, but in its mathematical approach to distributed consensus.
When you type kubectl apply:
-
kubectlcalculates a 3-way strategic merge patch. -
kube-apiserverauthenticates, authorizes, mutates, and validates the schema. -
etcdlogs the desire across a Raft quorum using optimistic concurrency control. - Independent controllers wake up through local Informer caches and reconcile differences.
- The scheduler filters candidate nodes with predicates and scores the winners.
- The kubelet uses CRI to spawn a Pause container, establishes shared Linux namespaces, configures CNI interfaces, and executes application binaries.
-
kube-proxyprograms kernel packet routing rules to balance network traffic.
There is no magic in the cloud. There are only state machines, persistent consensus logs, and continuous reconciliation loops ensuring that what you declared is what actually runs.
What was the most baffling Kubernetes issue you ever diagnosed in production? Did you ever track down an admission webhook timeout or unpick an iptables routing clash? Share your war stories and architecture insights in the comments below.





Top comments (1)
Because Kubernetes is doing much more behind the scenes than most people realise.
When we run a simple command like:
"kubectl apply -f deployment.yaml"
It might look like we're just sending a configuration file to a cluster. But that single command triggers an entire chain of operations involving the Kubernetes API Server, authentication, authorization, admission controllers, etcd, the scheduler, and various controllers.
Understanding this process is what separates someone who can simply execute Kubernetes commands from someone who can actually troubleshoot and operate a Kubernetes environment.
For example:
These are not just theoretical questions. They are the kinds of problems developers and DevOps engineers encounter in real production environments.
The interesting part is that Kubernetes is a declarative system. You're not directly telling it every step to execute. You're describing the desired state, and its control loops continuously work to make the actual cluster state match that desired state.
That's the real magic behind Kubernetes.
Learning "kubectl apply" is easy. Understanding the entire lifecycle behind it is where the real engineering knowledge begins.
If you're learning Kubernetes, don't just memorise commands. Follow the request from your terminal all the way to a running container.
Because when something breaks in production, understanding what's happening under the hood is far more valuable than knowing which command to copy and paste.
My question for the community: When you first started learning Kubernetes, what confused you the most: the API Server, etcd, scheduling, controllers, or networking?