Subtitle
A first-principles mental model for understanding Kubernetes controllers, desired state, actual state, and why this changes the way you debug production systems.
Introduction
When I first learned Kubernetes, I mostly thought about it in terms of containers:
Pods run containers.
Deployments create Pods.
Services expose Pods.
Scheduler puts Pods on nodes.
Those statements are true.
But they don't really explain what Kubernetes is doing underneath.
The mental model that made Kubernetes much easier for me was this:
Kubernetes is a reconciliation system.
At its core, you declare the state you want, and Kubernetes continuously works to make the actual system match that desired state.
Once this clicks, many Kubernetes concepts start fitting together naturally.
1. Desired State vs Actual State
Imagine I create a Deployment with:
spec:
replicas: 5
I'm not really saying:
“Kubernetes, immediately start five containers.”
I'm declaring:
“I want five replicas of this workload to exist.”
That's the desired state.
The cluster has some actual state.
Maybe currently:
Desired:
5 Pods
Actual:
5 Pods
Everything matches.
But now imagine one Pod crashes:
Desired:
5 Pods
Actual:
4 Pods
Something changed.
Kubernetes doesn't need me to manually say:
“Hey Kubernetes, one of my Pods died. Please create another one.”
The system observes the difference and works toward the desired state again:
Desired: 5
Actual: 4
↓
Controller notices difference
↓
New Pod is created
↓
Scheduler finds a suitable node
↓
Kubelet starts the workload
↓
Actual state approaches 5
That is reconciliation.
2. The Core Loop
A useful simplified mental model is:
┌─────────────────┐
│ Desired State │
│ replicas: 5 │
└────────┬────────┘
↓
Kubernetes API
↓
┌─────────────────┐
│ Controller │
└────────┬────────┘
↓
Take an action
↓
┌─────────────────┐
│ Actual State │
│ 4 Pods │
└────────┬────────┘
↓
Observe
│
└──────────────→ Reconcile again
The important word here is continuously.
Kubernetes isn't executing a one-time script.
It's continuously observing the system and responding to differences between what should exist and what actually exists.
3. Where does the Desired State live?
Kubernetes objects describe desired state.
For example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 5
The YAML itself isn't Kubernetes.
It's simply a representation of an object that we submit to the Kubernetes API.
The API Server becomes the front door through which Kubernetes components interact with the cluster state.
A simplified flow is:
kubectl / GitOps / API client
↓
API Server
↓
Cluster state
↓
Controllers
The important thing is that Kubernetes components don't all independently maintain some secret copy of what the cluster should look like.
They interact through the Kubernetes API and its stored state.
4. Controllers Are the Reconciliation Engine
This is where the idea becomes really powerful.
Different controllers are responsible for different kinds of resources.
For a Deployment, a simplified chain looks like:
Deployment
↓
ReplicaSet
↓
Pods
Suppose the Deployment says:
replicas = 5
The Deployment controller works toward that desired state.
The ReplicaSet is responsible for maintaining the desired number of Pods associated with it.
So if one Pod disappears:
Desired:
5 Pods
Actual:
4 Pods
↓
ReplicaSet reconciles
↓
Create replacement Pod
This is why the controller model is more important than memorizing individual Kubernetes commands.
5. Kubernetes Doesn't Guarantee That Desired State Is Possible
This is an important production lesson.
Declaring:
replicas: 100
doesn't mean Kubernetes has magically created the capacity for 100 Pods.
You are declaring:
“I want 100.”
Kubernetes then tries to make that happen.
But the cluster might not have enough resources.
For example:
Desired replicas: 100
Running: 70
Pending: 30
The desired state is still 100.
But the actual state is only 70 running Pods.
Why?
Maybe the scheduler can't find nodes with enough requested CPU or memory.
Maybe node constraints don't match.
Maybe the cluster needs to scale.
Maybe some other scheduling constraint prevents placement.
The important distinction is:
Kubernetes can maintain a desired state only when the underlying system has the ability to satisfy it.
This is one reason simply looking at:
kubectl get pods
isn't enough when debugging production.
You need to understand why actual state differs from desired state.
6. Running Doesn't Mean Healthy
The reconciliation model also helps explain something that initially confused me:
Pod phase = Running
doesn't necessarily mean:
“My application is healthy.”
A Pod can be running while:
- the application is returning errors
- a dependency is unavailable
- the application isn't ready to receive traffic
- the process is alive but stuck
That's where concepts such as readiness and liveness become important.
For example:
Pod exists
↓
Container running
↓
Readiness probe
↓
Can this workload receive traffic?
Kubernetes may successfully maintain:
Desired: 5 Pods
Actual: 5 Pods
while the application itself is still unhealthy.
This distinction is critical:
Kubernetes can reconcile infrastructure state without understanding your application's business correctness.
7. The Same Idea Appears Everywhere
Once you start seeing Kubernetes through the reconciliation lens, many features become easier to understand.
Deployment
Desired replicas
↓
Controller
↓
Actual Pods
Horizontal Pod Autoscaler
Desired workload capacity
↓
HPA
↓
Deployment replicas
Cluster Autoscaler
Desired node capacity
↓
Autoscaler
↓
Actual nodes
KEDA
A workload metric changes:
Queue grows
↓
KEDA
↓
HPA
↓
More replicas
The details differ, but the broader pattern remains:
Observe → Compare → Act → Observe again
That's the mental model.
8. A Better Way to Debug Kubernetes
This is probably the most useful part of the mental model for me.
When something is wrong, instead of immediately running random kubectl commands, start with:
Question 1 — What is the desired state?
For example:
Deployment wants:
100 replicas
Question 2 — What is the actual state?
Maybe:
70 Running
30 Pending
Now we've established the mismatch.
Question 3 — Who is responsible for closing the gap?
Is it:
- a Deployment controller?
- a scheduler?
- HPA?
- KEDA?
- Cluster Autoscaler?
- the application itself?
Question 4 — Why can't the next reconciliation step succeed?
For example:
100 desired
70 running
30 pending
↓
Scheduler says:
Insufficient CPU
Now we've moved from:
“Kubernetes isn't working.”
to:
“The desired state is 100 replicas, but 30 Pods cannot be scheduled because available node capacity doesn't satisfy their resource requests.”
That's a completely different debugging conversation.
9. The Mental Model I Keep
I now think about Kubernetes like this:
DESIRED STATE
│
↓
Kubernetes API
│
↓
CONTROLLERS
│
↓
ACTION
│
↓
ACTUAL STATE
│
↓
OBSERVE
│
└──────────→ RECONCILE
And when something breaks:
What do we want?
↓
What do we actually have?
↓
Where is the difference?
↓
Which controller/component owns that difference?
↓
Why can't it reconcile it?
That is a much more useful Kubernetes mental model than memorizing commands.
Conclusion
Kubernetes is obviously a platform for running workloads and containers.
But I think that description hides the more important idea.
The deeper abstraction is:
Kubernetes continuously reconciles the actual state of a distributed system toward a declared desired state.
Once you understand that, Deployments, ReplicaSets, scheduling, autoscaling, self-healing, and many debugging workflows become easier to reason about.
And perhaps the biggest lesson is this:
When Kubernetes isn't doing what you expect, don't just ask “what command should I run?” Ask “what state does Kubernetes want, what state does it have, and why can't it reconcile the difference?”
Top comments (0)