DEV Community

James Lee
James Lee

Posted on

Kubernetes Control Flow: How Resources Are Created, Deleted, Modified & Queried

#kubernetes #architecture #tutorial #devops

In the previous article we covered Kubernetes' logical architecture — two node types, their components, and how they communicate. Now let's trace what actually happens inside the cluster when you run kubectl apply, kubectl delete, or kubectl get. This is the control flow: the complete lifecycle of a resource operation.

1. Two Categories of Resources

Kubernetes resources fall into two levels:

Level Examples
Infrastructure Control plane nodes, worker nodes
Application workload Pod, Deployment, ReplicaSet, Service, Endpoint, ServiceAccount, Secret, Volume, ...

The control flow differs slightly between these two categories, and also between read and write operations. Let's break them all down.

2. Infrastructure-Level Control Flow

Control Plane Nodes

Control plane nodes follow a strict rule: you cannot use one control plane node to add or remove another.

  • Adding/removing control plane nodes requires manual intervention (human operator)
  • This is only relevant when building a high-availability (HA) cluster
  • Control plane node count follows the formula: $$2n - 1$$ where $$n > 1$$ (i.e., 1, 3, 5, 7, ...) 
Single control plane  →  3 control planes  →  5 control planes
(no HA)                  (HA, tolerates 1 failure)  (tolerates 2 failures)
Enter fullscreen mode Exit fullscreen mode

Why $$2n-1$$? Control plane nodes are stateless and cannot directly sense each other. They rely on etcd for consensus. etcd uses the Raft protocol, which requires a majority quorum — so an odd number is required to avoid split-brain.

Worker Nodes

Worker nodes are fully manageable through Kubernetes tooling:

Operation Tool Notes
Add node kubeadm join No $$2n-1$$ constraint; max 5,000 nodes per cluster
Remove node kubeadm reset Can scale down to 0 worker nodes
Query node kubectl get nodes Common operation
Modify/Delete node kubectl Possible but rarely used directly

Scaling to 0 worker nodes: The control plane cannot schedule workloads by default (due to its NoSchedule taint). To run workloads on the control plane, you must explicitly remove the taint:

kubectl taint nodes <control-plane-node> node-role.kubernetes.io/control-plane:NoSchedule-

3. Application-Level Control Flow

Most Kubernetes resources (Deployment, ReplicaSet, Pod, Service, etc.) share the same underlying control flow pattern. Since nearly all resources ultimately serve Pods, we'll use Pod operations as the primary example.

3.1 Resource Creation Flow 

kubectl apply -f pod.yaml
     │
     ▼
① kubectl / REST client
     Serialize resource → API object format
     POST /api/v1/namespaces/{ns}/pods
     │
     ▼
② kube-apiserver
     Validate + authenticate + authorize
     Write resource object to etcd (path: /registry/pods/{ns}/{name})
     │
     ▼
③ kube-controller-manager
     Watches etcd for new unbound resources
     Relevant controller (e.g. ReplicaSet controller) enqueues the resource
     │
     ▼
④ kube-scheduler
     Watches the scheduling queue
     Runs filter + score algorithms across available nodes
     Selects optimal node
     Writes binding result → etcd (/registry/pods/{ns}/{name}/binding)
     │
     ▼
⑤ kubelet (on selected node)
     Watches etcd for new bindings on its node
     Calls container runtime (containerd/CRI-O) to create containers
     Monitors running state → reports back to apiserver
Enter fullscreen mode Exit fullscreen mode

Key insight: No component talks directly to another. Every step is mediated through etcd (write) and watches (read). This event-driven, decoupled design is what makes Kubernetes so resilient.

3.2 Resource Deletion Flow 

kubectl delete pod {name}
     │
     ▼
① kubectl / REST client
     DELETE /api/v1/namespaces/{ns}/pods/{name}
     │
     ▼
② kube-apiserver
     Write deletion marker to etcd
     │
     ▼
③ kubelet (on the node running the Pod)
     Watches etcd for deletion events on its node
     Checks whether the target resource exists locally
     │
     ▼
④ kubelet
     Calls container runtime to stop + remove containers
     Resource is deleted  ✅
Enter fullscreen mode Exit fullscreen mode

Note: Deletion is simpler than creation — it does not go through the scheduler. The kubelet on the node that owns the resource handles it directly.

3.3 Resource Modification Flow

Modification follows the same path as creation — because updating a resource may require rescheduling (e.g., changing resource requests may make the current node unsuitable):

kubectl apply -f updated-pod.yaml
     │
     ▼
① kubectl / REST client
     PATCH /api/v1/namespaces/{ns}/pods/{name}
     │
     ▼
② kube-apiserver
     Write updated resource object to etcd
     │
     ▼
③ kube-controller-manager
     Detects modified resource in etcd
     Enqueues it for rescheduling
     │
     ▼
④ kube-scheduler
     Re-evaluates node fitness
     Selects optimal node (may be same or different)
     Writes new binding → etcd
     │
     ▼
⑤ kubelet (on target node)
     Detects new binding
     Applies changes to running containers
     Resumes monitoring
Enter fullscreen mode Exit fullscreen mode

3.4 Resource Query Flow

Query is fundamentally different from write operations. There are two types of queryable resources:

Type Examples Storage
Persisted static resources Node, Pod, Deployment, Service, Secret, Volume, ... etcd
Real-time dynamic resources CPU/memory usage, Pod logs, exec sessions Live on node (kubelet)

Querying Persisted Resources 

kubectl get pod {name}
     │
     ▼
① kubectl / REST client
     GET /api/v1/namespaces/{ns}/pods/{name}
     │
     ▼
② kube-apiserver
     Looks up /registry/pods/{ns}/{name} in etcd
     Returns serialized resource object
     │
     ▼
③ kubectl / REST client
     Receives and displays result  ✅
Enter fullscreen mode Exit fullscreen mode

Querying Real-Time Resources 

kubectl top pod {name}
kubectl logs {name}
kubectl exec {name} -- ...
     │
     ▼
① kubectl / REST client
     GET /api/v1/namespaces/{ns}/pods/{name}/log  (or /exec, /metrics)
     │
     ▼
② kube-apiserver
     Looks up pod → finds node binding in etcd
     │
     ▼
③ kube-apiserver
     Proxies / forwards request to kubelet on the target node
     │
     ▼
④ kubelet (on target node)
     Queries container runtime / cgroups / log files
     Returns live data
     │
     ▼
⑤ kubectl / REST client
     Receives real-time result  ✅
Enter fullscreen mode Exit fullscreen mode

Why the proxy step? Real-time data (logs, CPU, memory) lives on the node, not in etcd. The apiserver acts as a secure proxy — clients never talk directly to kubelet, maintaining a single authenticated entry point.

4. The Full Picture 

┌─────────────────────────────────────────────────────────────────┐
│                    Operation Type Map                           │
│                                                                 │
│  CREATE  →  apiserver → etcd → controller-manager              │
│                              → scheduler → etcd                │
│                                          → kubelet  ✅         │
│                                                                 │
│  DELETE  →  apiserver → etcd → kubelet  ✅                     │
│                                                                 │
│  UPDATE  →  apiserver → etcd → controller-manager              │
│                              → scheduler → etcd                │
│                                          → kubelet  ✅         │
│                                                                 │
│  GET (static)    →  apiserver → etcd  ✅                       │
│  GET (realtime)  →  apiserver → etcd → kubelet (proxy)  ✅     │
└─────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode
Operation Goes through scheduler? Goes through controller-manager? Data source
Create ✅ Yes ✅ Yes etcd → node
Delete ❌ No ❌ No etcd → node
Update ✅ Yes ✅ Yes etcd → node
Query (static) ❌ No ❌ No etcd
Query (realtime) ❌ No ❌ No kubelet (proxied)

5. Summary

The Kubernetes control flow embodies a single architectural principle: all state lives in etcd, all components react to state changes via watches.

  • kube-apiserver is the only component that reads/writes etcd
  • kube-controller-manager and kube-scheduler are event-driven reactors — they watch for state and act
  • kubelet is the executor — it makes desired state real on the node
  • Real-time data (logs, metrics) bypasses etcd entirely, proxied through apiserver to kubelet

This decoupled, watch-based architecture is what gives Kubernetes its self-healing, eventually-consistent behavior.

Next in this series: Kubernetes Resource Scheduling: Filters, Scoring & Affinity Rules (Part 4)

Follow the series for more deep dives into Kubernetes internals.

Top comments (0)