DEV Community: Sheryar Ahmed

Why Kubernetes Says ‘Insufficient CPU’ When the Node Looks Idle

Sheryar Ahmed — Wed, 07 Oct 2026 08:29:04 +0000

Your deployment is stuck. The Pod has been Pending for several minutes, and the event looks conclusive:

0/3 nodes are available: 3 Insufficient cpu.

You open the dashboard. The nodes are using 30–40% CPU.

So where did the rest of the CPU go?

Nowhere. Kubernetes and your dashboard are simply answering different questions:

Your dashboard: How much CPU is being used right now?
The scheduler: How much CPU has already been accounted for, and can this Pod's request fit on one eligible node?

That difference—usage versus requests—is the entire mystery. But it leads to a few traps that are easy to miss in production.

The short answer

The Kubernetes scheduler does not place a Pod based on live CPU usage. It primarily uses the Pod's CPU request and each node's allocatable CPU.

In simplified form, a Pod fits when:

new Pod request
    <=
node allocatable CPU - CPU already requested on that node

Notice what is missing from that calculation: current CPU usage.

This is deliberate. If the scheduler packed nodes according to a quiet moment in a monitoring graph, several workloads could spike together and overwhelm the node.

A node can be idle and still be “full”

Imagine a node with 8 CPU cores allocatable to Pods:

Workload	CPU request	Current usage
API	2 CPU	0.7 CPU
Worker	2 CPU	0.5 CPU
Search	2 CPU	0.6 CPU
Total	6 CPU	1.8 CPU

Your monitoring system reports roughly 22.5% CPU usage. The node looks quiet.

Now you deploy another Pod requesting 3 CPU:

6 CPU already requested + 3 CPU requested = 9 CPU

The node has only 8 CPU allocatable, so the Pod does not fit. The scheduler reports Insufficient cpu even though actual usage is below 2 CPU.

There is no contradiction. The dashboard is showing activity; the scheduler is checking commitments.

One nuance matters here: a CPU request is not necessarily a dedicated core locked away for that container. Think of it as an entry in the scheduler's capacity ledger. At runtime, it also influences the container's CPU weight when the node is contended.

Request, limit, and usage are three different numbers

Consider this container:

resources:
  requests:
    cpu: "500m"
  limits:
    cpu: "2"

In Kubernetes CPU units, 1000m is 1 CPU, so 500m is half a CPU.

The three numbers mean:

Number	What it answers
Request	How much CPU should the scheduler account for when placing the Pod?
Limit	How much CPU time may the container consume before it is throttled?
Usage	How much CPU is the container consuming at this moment?

The scheduler uses the request for placement. It does not use kubectl top as a live packing signal, and a higher CPU limit does not normally require that whole limit to be available before scheduling.

This also explains two opposite failure modes:

Requests set too high: Pods are hard to schedule and the cluster looks underused.
Requests set too low: Pods schedule easily, but too many may land together and fight for CPU during a traffic spike.

The goal is not to make requests as small as possible. The goal is to make them honest.

Capacity is not allocatable CPU

A 16-core machine does not always offer all 16 cores to normal Pods.

Kubernetes distinguishes between:

Capacity: the resources present on the node.
Allocatable: the portion available to Pods after reservations for the operating system, kubelet, container runtime, and eviction thresholds.

Check both with:

kubectl describe node <node-name>

Look for output similar to:

Capacity:
  cpu: 16
Allocatable:
  cpu: 14

For scheduling, 14 CPU is the useful number.

Further down, kubectl describe node includes an Allocated resources section. Its requests column is much more relevant to this incident than the node's live utilization graph:

Allocated resources:
  Resource  Requests       Limits
  cpu       12500m (89%)   22000m (157%)

Seeing CPU limits above 100% can be valid because CPU limits may be overcommitted. The requests column is the important scheduling signal.

Debug it in five minutes

1. Read the complete scheduling event

kubectl describe pod <pod-name> -n <namespace>

Do not stop after reading Insufficient cpu. The complete event may say:

0/6 nodes are available:
2 Insufficient cpu,
2 node(s) had untolerated taint,
2 node(s) didn't match Pod's node affinity.

In that case, CPU is only part of the story. Your Pod effectively has just two CPU-eligible nodes, not six.

2. Inspect the Pod's actual request

kubectl get pod <pod-name> -n <namespace> -o yaml

Check resources.requests.cpu for every container, including sidecars. A Pod's normal-container CPU request is generally the sum of its containers' requests.

Also inspect init containers. Because init containers run sequentially, Kubernetes uses the largest relevant init-container request when calculating the Pod's effective request, rather than simply adding all init containers together. A heavyweight migration or setup container can therefore make a seemingly small application Pod difficult to schedule.

3. Check each eligible node's allocatable and requested CPU

kubectl describe node <node-name>

Compare:

allocatable CPU
- existing CPU requests
= schedulable headroom

Then compare that headroom with the new Pod's effective request.

4. Use live usage for diagnosis—not for placement math

kubectl top nodes
kubectl top pods -A --sort-by=cpu

These commands help answer a different but valuable question: are requests aligned with reality?

If a service requests 2 CPU per replica but stays near 100m even during representative peaks, its request may be oversized. Do not resize it from one quiet snapshot, though. Look at a meaningful time window, peak traffic, startup behavior, batch jobs, and latency under CPU contention.

Four traps that make this more confusing

1. Free CPU is fragmented across nodes

Suppose three nodes each have 2 CPU of schedulable headroom. The cluster has 6 CPU free in total, but a single Pod requesting 3 CPU still cannot run.

A Pod must fit on one node. Cluster-wide free CPU cannot be pooled across nodes for one Pod.

2. A namespace policy may add requests for you

A LimitRange can inject default requests and limits when a workload omits them. Also, when a container specifies a CPU limit but no request, Kubernetes can assign a request equal to that limit.

Check for defaults with:

kubectl get limitrange -n <namespace> -o yaml

Always inspect the created Pod, not only the Helm values or Deployment template you expected to be applied.

3. DaemonSets consume space on every node

Logging agents, security agents, CNI components, and monitoring exporters often run as DaemonSets. Their requests count too. A small per-node request becomes significant when node sizes are small or the list of agents grows.

4. “Insufficient CPU” may appear beside other hard constraints

Node selectors, required affinity, pod anti-affinity, taints, topology spread rules, and volume topology can shrink the set of valid nodes before CPU is considered.

The cluster may have plenty of CPU—just not on a node your Pod is allowed to use.

What should you fix?

The event tells you the symptom, not the correct remedy. Choose the fix that matches the evidence:

Requests are unrealistic: Right-size them using historical usage, representative peaks, load tests, and latency or throughput goals. A Vertical Pod Autoscaler in recommendation mode can provide another data point.
The request is legitimate: Add or scale suitable nodes, or use larger nodes that can fit the Pod.
Capacity is fragmented: Rebalance workloads, reconsider placement rules, or provide a node shape large enough for the biggest Pod.
Only a narrow node pool is eligible: Revisit affinity, selectors, taints, and topology rules—or scale that specific pool.
A rollout temporarily needs extra room: Account for maxSurge and other deployment behavior in your capacity plan.

What you should not do is blindly lower the request until the Pod schedules. That can turn a visible scheduling problem into a less obvious production performance problem.

The mental model to keep

When a node looks idle but Kubernetes says Insufficient cpu, remember:

usage       = what is happening now
request     = what the scheduler plans around
allocatable = what the node offers to Pods

Then ask four questions:

What is this Pod's effective CPU request?
How much CPU is allocatable on each eligible node?
How much CPU has already been requested there?
What other rules make nodes ineligible?

Once you separate live usage from scheduler accounting, Insufficient cpu stops looking like a Kubernetes lie. The node can be quiet and still have no room for the promise your new Pod is asking Kubernetes to make.

Why a Kubernetes Pod Being 'Running' Doesn't Mean It's Healthy

Sheryar Ahmed — Thu, 01 Oct 2026 09:11:37 +0000

Why a Kubernetes Pod Being `Running` Doesn't Mean It's Healthy

A Pod can be Running while your application is broken, unable to receive traffic, or quietly failing every job.

One of the easiest Kubernetes mistakes to make is also one of the most dangerous:

kubectl get pods

You see:

NAME                          READY   STATUS    RESTARTS
worker-7f9    1/1     Running   0

And you think:

"The Pod is healthy."

Not necessarily.

Running is not a health check.

It's a statement about Pod lifecycle.

And understanding that difference completely changes how you debug Kubernetes.

The first mental model

A Kubernetes Pod has several different questions that can be answered independently:

Is the Pod running?
        ↓
Is the container ready?
        ↓
Can it receive traffic?
        ↓
Is the application actually functioning?
        ↓
Can it successfully perform its real work?

These are not the same question.

Think about a normal server.

A process can be alive:

PID 1234

while the application is:

stuck
unable to connect to a dependency
returning errors
unable to process requests
consuming jobs and failing them

Kubernetes is no different.

`Running` means something very specific

Kubernetes Pod phases include:

Pending
Running
Succeeded
Failed
Unknown

When a Pod is Running, Kubernetes knows that the Pod has been scheduled and that its containers have started.

It does not automatically mean:

✓ application is healthy
✓ application can serve traffic
✓ dependencies are working
✓ requests are succeeding
✓ background jobs are succeeding

That's the first distinction to remember:

Running describes lifecycle. Healthy describes behavior.

So how does Kubernetes know whether a container is healthy?

This is where probes enter the picture.

Kubernetes gives us three primary probe types:

Startup Probe
Readiness Probe
Liveness Probe

They answer three different questions.

1. Startup Probe

"Has the application finished starting?"

Imagine your application takes 40 seconds to initialize.

During those 40 seconds:

container = alive
application = still starting

If you immediately run a liveness check, Kubernetes might conclude:

"This application is broken."

And restart it.

Then it starts again.

Then the probe fails again.

Then Kubernetes restarts it again.

You can accidentally create:

start
 ↓
probe fails
 ↓
restart
 ↓
start
 ↓
probe fails
 ↓
restart

A startup probe protects slow-starting applications from that situation.

Example:

startupProbe:
  httpGet:
    path: /health/live
    port: 8080
  periodSeconds: 2
  failureThreshold: 15

With these settings, Kubernetes gives the application roughly:

2 × 15 = 30 seconds

to successfully start before treating startup as failed.

2. Readiness Probe

This is one of the most important concepts in production Kubernetes.

Readiness asks:

"Should this Pod receive work right now?"

Suppose you have:

Service
   ↓
Pod A
Pod B
Pod C

If Pod B fails its readiness probe:

Pod A → Ready
Pod B → NotReady
Pod C → Ready

Kubernetes can stop sending Service traffic to Pod B.

But here's the important part:

A failed readiness probe does not restart the container.

That's intentional.

Maybe the application is temporarily unable to serve traffic but is expected to recover.

For example:

database temporarily unavailable
        ↓
readiness fails
        ↓
Pod stays alive
        ↓
Pod removed from traffic
        ↓
database recovers
        ↓
readiness succeeds
        ↓
Pod receives traffic again

That's very different from liveness.

3. Liveness Probe

Liveness asks:

"Is this application so broken that restarting the container is appropriate?"

For example, suppose your application enters a deadlock:

process = alive
HTTP server = technically running
application = permanently stuck

A liveness probe can detect that condition.

If it repeatedly fails:

liveness fails
      ↓
Kubernetes restarts container
      ↓
application starts fresh

That's useful.

But it can also be dangerous.

The dangerous liveness probe

Imagine your liveness endpoint checks:

database
Redis
external API
third-party service

Now Redis goes down.

Your application itself is perfectly capable of running, but your liveness probe says:

Redis unavailable
      ↓
liveness fails
      ↓
restart application
      ↓
application starts
      ↓
Redis unavailable
      ↓
liveness fails
      ↓
restart

You've just turned a dependency outage into a restart storm.

That's why liveness should generally answer:

"Is this process/application locally stuck or irrecoverably broken?"

rather than:

"Are all of my dependencies healthy?"

The three probes in one picture

Remember this:

                 APPLICATION
                     │
          ┌──────────┼──────────┐
          ↓          ↓          ↓
       STARTUP    READINESS   LIVENESS
          │          │          │
          ↓          ↓          ↓
     "Can you      "Should     "Should I
      start?"       receive      restart
                    work?"      you?"

Or even shorter:

Startup   → Can you start?
Readiness → Can you work?
Liveness  → Should I restart you?

That's the mental model.

But there's another trap

Here's where real production systems become interesting.

Imagine a worker that consumes jobs from a queue.

Its architecture is:

Queue
  ↓
Worker
  ↓
Browser
  ↓
Result

Now imagine the browser inside the worker becomes unhealthy.

You might create a readiness check:

browser unhealthy
       ↓
readiness = false

Sounds correct.

But what happens to the worker?

If the worker is still consuming jobs:

Queue
 ↓
Worker
 ↓
Browser ❌

the worker may continue receiving jobs even though it cannot successfully process them.

Now you've created a worse situation:

job arrives
   ↓
worker receives it
   ↓
browser unhealthy
   ↓
job fails
   ↓
next job
   ↓
fails
   ↓
next job

The Pod can be:

STATUS = Running

while the actual business operation is failing.

This is exactly why readiness needs to be designed around how the workload actually receives work, not simply around whether a health endpoint returns 200. Your infrastructure notes flag this as an area requiring careful review.

The subtle lesson: Readiness doesn't stop everything

This is especially important for background workers.

For a normal HTTP application:

Load Balancer
     ↓
Service
     ↓
Readiness
     ↓
Pod

Readiness has a very direct effect.

But for a worker:

Pub/Sub / Queue
      ↓
Worker

the application may pull work directly.

There may be no Kubernetes Service sitting between the queue and the worker.

Therefore:

readiness = false

doesn't automatically mean:

stop consuming jobs

That's an application-level behavior.

This is one of the biggest differences between:

"Not receiving HTTP traffic"

and

"Not receiving background work."

Kubernetes health is layered

A better mental model is:

Layer 1
Pod lifecycle
      ↓
Running?

Layer 2
Container health
      ↓
Startup / Liveness

Layer 3
Traffic readiness
      ↓
Ready?

Layer 4
Application health
      ↓
Can requests actually succeed?

Layer 5
Business health
      ↓
Are jobs actually completing successfully?

And that last layer is often forgotten.

A Pod can look perfectly healthy and still be failing

Consider:

NAME                    READY   STATUS
worker-abc              1/1     Running

Everything looks good.

But your metrics say:

jobs_received = 10,000
jobs_completed = 0
jobs_failed = 10,000

From Kubernetes' perspective:

Pod = Running
Container = alive

From your business perspective:

System = broken

Both can be true at the same time.

That's why production observability cannot rely on:

kubectl get pods

alone.

What should you actually monitor?

For a production workload, I'd think in layers.

Kubernetes signals

Pod phase
Container restarts
OOMKilled
CPU throttling
Memory usage
Pending Pods
Scheduling failures

Probe signals

Startup failures
Readiness failures
Liveness failures
Time spent NotReady

Application signals

Request success rate
Error rate
Latency
Dependency failures
Queue consumption

Business signals

Jobs completed
Jobs failed
Jobs expired
Queue wait time
Success rate
Customer-visible failures

The further down you go, the closer you get to the question:

"Is the system actually doing what users need?"

A practical debugging flow

When someone says:

"The Pod is unhealthy."

Don't immediately restart it.

Ask:

Step 1: Is it running?

kubectl get pod <pod>

Step 2: What are the conditions?

kubectl describe pod <pod>

Look at:

Conditions
Events
Restart Count
Last State
Reason

Step 3: Which probe is failing?

Look for:

Startup probe
Readiness probe
Liveness probe

Step 4: Test the endpoint yourself

For example:

curl localhost:8080/health/live
curl localhost:8080/health/ready

Step 5: Check application logs

kubectl logs <pod>

Step 6: Check actual business behavior

For a worker:

Are jobs being consumed?
Are jobs succeeding?
Are they failing?
Are they expiring?

That's where you stop debugging Kubernetes and start debugging the actual system.

A useful production rule

Here's the rule I now keep in my head:

Don't design health checks around what is easy to check. Design them around what Kubernetes needs to know.

For example:

Readiness

Good:

Can this Pod safely receive work?

Potentially dangerous:

Is every dependency in the entire system healthy?

Liveness

Good:

Is this process/application fundamentally stuck?

Potentially dangerous:

Is every external dependency healthy?

Business monitoring

Good:

Are jobs actually succeeding?

This belongs in metrics and alerting, not necessarily in liveness.

The final mental model

Next time you see:

STATUS: Running

don't translate it to:

"Everything is healthy."

Translate it to:

"The Pod is alive according to Kubernetes' lifecycle state. Now I need to determine whether it is ready, responsive, and actually doing useful work."

That's a much safer production mindset.

The 10-second revision

If you remember nothing else:

Running
  ≠
Healthy

Startup   → Can you start?
Readiness → Can you receive work?
Liveness  → Should you be restarted?
Business  → Are you actually succeeding?

And the most important production lesson:

Kubernetes can tell you that your process is alive. Your application metrics need to tell you whether your system is actually working.

What I learned from this

The interesting part of Kubernetes isn't memorizing:

Pod
Deployment
Service
Probe

It's understanding the failure semantics between them.

A readiness failure doesn't restart a container.

A liveness failure does.

A Pod being Running doesn't make it Ready.

A Pod being Ready doesn't guarantee that a business operation succeeds.

And for background workers, being NotReady doesn't necessarily mean the worker has stopped consuming work.

That's where Kubernetes knowledge starts becoming production engineering.

Kubernetes Isn't a Container Manager, It's a Reconciliation System

Sheryar Ahmed — Thu, 24 Sep 2026 11:07:22 +0000

Subtitle

A first-principles mental model for understanding Kubernetes controllers, desired state, actual state, and why this changes the way you debug production systems.

Introduction

When I first learned Kubernetes, I mostly thought about it in terms of containers:

Pods run containers.
Deployments create Pods.
Services expose Pods.
Scheduler puts Pods on nodes.

Those statements are true.

But they don't really explain what Kubernetes is doing underneath.

The mental model that made Kubernetes much easier for me was this:

Kubernetes is a reconciliation system.

At its core, you declare the state you want, and Kubernetes continuously works to make the actual system match that desired state.

Once this clicks, many Kubernetes concepts start fitting together naturally.

1. Desired State vs Actual State

Imagine I create a Deployment with:

spec:
  replicas: 5

I'm not really saying:

“Kubernetes, immediately start five containers.”

I'm declaring:

“I want five replicas of this workload to exist.”

That's the desired state.

The cluster has some actual state.

Maybe currently:

Desired:
5 Pods

Actual:
5 Pods

Everything matches.

But now imagine one Pod crashes:

Desired:
5 Pods

Actual:
4 Pods

Something changed.

Kubernetes doesn't need me to manually say:

“Hey Kubernetes, one of my Pods died. Please create another one.”

The system observes the difference and works toward the desired state again:

Desired: 5
Actual: 4
       ↓
Controller notices difference
       ↓
New Pod is created
       ↓
Scheduler finds a suitable node
       ↓
Kubelet starts the workload
       ↓
Actual state approaches 5

That is reconciliation.

2. The Core Loop

A useful simplified mental model is:

        ┌─────────────────┐
        │  Desired State  │
        │   replicas: 5   │
        └────────┬────────┘
                 ↓
          Kubernetes API
                 ↓
        ┌─────────────────┐
        │    Controller   │
        └────────┬────────┘
                 ↓
          Take an action
                 ↓
        ┌─────────────────┐
        │   Actual State  │
        │    4 Pods       │
        └────────┬────────┘
                 ↓
              Observe
                 │
                 └──────────────→ Reconcile again

The important word here is continuously.

Kubernetes isn't executing a one-time script.

It's continuously observing the system and responding to differences between what should exist and what actually exists.

3. Where does the Desired State live?

Kubernetes objects describe desired state.

For example:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: my-app

spec:
  replicas: 5

The YAML itself isn't Kubernetes.

It's simply a representation of an object that we submit to the Kubernetes API.

The API Server becomes the front door through which Kubernetes components interact with the cluster state.

A simplified flow is:

kubectl / GitOps / API client
          ↓
     API Server
          ↓
     Cluster state
          ↓
      Controllers

The important thing is that Kubernetes components don't all independently maintain some secret copy of what the cluster should look like.

They interact through the Kubernetes API and its stored state.

4. Controllers Are the Reconciliation Engine

This is where the idea becomes really powerful.

Different controllers are responsible for different kinds of resources.

For a Deployment, a simplified chain looks like:

Deployment
     ↓
ReplicaSet
     ↓
Pods

Suppose the Deployment says:

replicas = 5

The Deployment controller works toward that desired state.

The ReplicaSet is responsible for maintaining the desired number of Pods associated with it.

So if one Pod disappears:

Desired:
5 Pods

Actual:
4 Pods

        ↓

ReplicaSet reconciles

        ↓

Create replacement Pod

This is why the controller model is more important than memorizing individual Kubernetes commands.

5. Kubernetes Doesn't Guarantee That Desired State Is Possible

This is an important production lesson.

Declaring:

replicas: 100

doesn't mean Kubernetes has magically created the capacity for 100 Pods.

You are declaring:

“I want 100.”

Kubernetes then tries to make that happen.

But the cluster might not have enough resources.

For example:

Desired replicas: 100

Running:          70
Pending:           30

The desired state is still 100.

But the actual state is only 70 running Pods.

Why?

Maybe the scheduler can't find nodes with enough requested CPU or memory.

Maybe node constraints don't match.

Maybe the cluster needs to scale.

Maybe some other scheduling constraint prevents placement.

The important distinction is:

Kubernetes can maintain a desired state only when the underlying system has the ability to satisfy it.

This is one reason simply looking at:

kubectl get pods

isn't enough when debugging production.

You need to understand why actual state differs from desired state.

6. Running Doesn't Mean Healthy

The reconciliation model also helps explain something that initially confused me:

Pod phase = Running

doesn't necessarily mean:

“My application is healthy.”

A Pod can be running while:

the application is returning errors
a dependency is unavailable
the application isn't ready to receive traffic
the process is alive but stuck

That's where concepts such as readiness and liveness become important.

For example:

Pod exists
   ↓
Container running
   ↓
Readiness probe
   ↓
Can this workload receive traffic?

Kubernetes may successfully maintain:

Desired: 5 Pods
Actual: 5 Pods

while the application itself is still unhealthy.

This distinction is critical:

Kubernetes can reconcile infrastructure state without understanding your application's business correctness.

7. The Same Idea Appears Everywhere

Once you start seeing Kubernetes through the reconciliation lens, many features become easier to understand.

Deployment

Desired replicas
       ↓
Controller
       ↓
Actual Pods

Horizontal Pod Autoscaler

Desired workload capacity
       ↓
HPA
       ↓
Deployment replicas

Cluster Autoscaler

Desired node capacity
       ↓
Autoscaler
       ↓
Actual nodes

KEDA

A workload metric changes:

Queue grows
   ↓
KEDA
   ↓
HPA
   ↓
More replicas

The details differ, but the broader pattern remains:

Observe → Compare → Act → Observe again

That's the mental model.

8. A Better Way to Debug Kubernetes

This is probably the most useful part of the mental model for me.

When something is wrong, instead of immediately running random kubectl commands, start with:

Question 1 — What is the desired state?

For example:

Deployment wants:
100 replicas

Question 2 — What is the actual state?

Maybe:

70 Running
30 Pending

Now we've established the mismatch.

Question 3 — Who is responsible for closing the gap?

Is it:

a Deployment controller?
a scheduler?
HPA?
KEDA?
Cluster Autoscaler?
the application itself?

Question 4 — Why can't the next reconciliation step succeed?

For example:

100 desired
70 running
30 pending

        ↓

Scheduler says:
Insufficient CPU

Now we've moved from:

“Kubernetes isn't working.”

to:

“The desired state is 100 replicas, but 30 Pods cannot be scheduled because available node capacity doesn't satisfy their resource requests.”

That's a completely different debugging conversation.

9. The Mental Model I Keep

I now think about Kubernetes like this:

             DESIRED STATE
                  │
                  ↓
            Kubernetes API
                  │
                  ↓
             CONTROLLERS
                  │
                  ↓
               ACTION
                  │
                  ↓
             ACTUAL STATE
                  │
                  ↓
               OBSERVE
                  │
                  └──────────→ RECONCILE

And when something breaks:

What do we want?

        ↓

What do we actually have?

        ↓

Where is the difference?

        ↓

Which controller/component owns that difference?

        ↓

Why can't it reconcile it?

That is a much more useful Kubernetes mental model than memorizing commands.

Conclusion

Kubernetes is obviously a platform for running workloads and containers.

But I think that description hides the more important idea.

The deeper abstraction is:

Kubernetes continuously reconciles the actual state of a distributed system toward a declared desired state.

Once you understand that, Deployments, ReplicaSets, scheduling, autoscaling, self-healing, and many debugging workflows become easier to reason about.

And perhaps the biggest lesson is this:

When Kubernetes isn't doing what you expect, don't just ask “what command should I run?” Ask “what state does Kubernetes want, what state does it have, and why can't it reconcile the difference?”

When something breaks across 20 microservices, one question decides how fast you recover

Sheryar Ahmed — Mon, 03 Aug 2026 17:15:15 +0000

It's 2 A.M. and Your Dashboard Just Says "Something Is Wrong"

A field guide to observability — logs, metrics, and traces — for everyone who has ever debugged twenty microservices with grep and prayer.

It's 2:14 a.m. Your phone is doing the thing.

Caption: "on-call, living the dream."

Checkout is failing. Not for everyone, not all the time, but enough that support is drowning and someone has already tweeted a screenshot with the words "is this app down or is it just me." You open your laptop, one eye still asleep, and pull up the dashboard.

A big red number is climbing. Error rate: up. Something is wrong.

And that is all it tells you. Something.

Caption: "Dashboard: 'Something is wrong.' Me: great, love that for us, very specific."*

Here's the trap you're standing in. Your system isn't one program anymore. It's twenty services, spread across a cluster, written by different people in different languages, calling each other in a chain you couldn't draw from memory if your life depended on it. One "place order" click quietly fans out into a dozen network hops. One of them is failing. Maybe two. Your dashboard knows the house is on fire. It has absolutely no idea which room.

So you do what everyone does at 2 a.m. You start SSH-ing into boxes. You grep logs. You guess. "Probably the payment service." You read a thousand lines and find nothing. "The database?" Fine. The clock ticks, users keep failing to check out, and somewhere in that chain of twenty services the actual problem is sitting there, arms crossed, invisible, waiting for you to trip over it.

This — this exact moment — is what observability is for. Not to look impressive on a screen. To answer one question, fast: where is the problem, and why?

Let me actually teach you how it works, start to finish, so that nights like this stop being three-hour horror movies and start being ninety-second fixes. No prior knowledge needed. Just bring the caffeine.

First, a distinction that everything else hangs on

People throw around "monitoring" and "observability" like they mean the same thing. They don't, and the gap between them is the whole story.

Monitoring is the smoke alarm. You decided in advance what to watch — error rate, CPU, latency — and built a dashboard for it. It's brilliant at one job: screaming when a known thing goes wrong. "There is smoke." But a smoke alarm cannot tell you that the wiring in the upstairs bathroom shorted out. It only knows: smoke.

Observability is being able to walk through the burning house and ask questions you never planned for. "Why are only Android users, only in one region, only after 6 p.m., only when paying by card, getting slow checkouts?" Nobody builds a dashboard that specific ahead of time. But if your system is constantly describing itself — emitting enough signal — you can start at the alarm and follow the smoke all the way to the one shorted wire, after the fire starts, without shipping new code to investigate.

Monitoring tells you the house is on fire. Observability tells you which room. You need both, but at 2 a.m., the alarm was never the hard part. Finding the room is.

And to find the room, you need three different kinds of signal. Each answers a different question, and the entire skill is knowing which one to grab.

The three signals (a.k.a. your detective kit)

Treat your broken checkout like a crime scene. You've got three tools, and they are not interchangeable.

Metrics — the vital-signs monitor

Metrics are numbers over time: requests per second, error rate, latency. Cheap, always running, glanceable. Metrics are what spiked and woke you up. Their job is to answer is something wrong, and how bad?

What metrics cannot do is tell you which patient or why. A metric is an aggregate — the heart-rate line jumping — and it will never name the one request that died, because to stay cheap it blended that request into a million others. That's not a bug. That's the deal: you traded per-request detail for a signal that runs 24/7 and costs almost nothing.

If you only ever watch a handful of metrics, watch these four — the industry calls them the Four Golden Signals: latency (how slow), traffic (how much demand), errors (how many are failing), and saturation (how full your system is). Those four are the vital signs of basically any service. Get them on a dashboard and you have honest, always-on vitals.

Traces — the detective tailing the suspect through the building

A trace is the story of one single request as it travels across all your services, broken into timed steps called spans. The gateway received it (span one). The gateway called orders (span two). Orders called payments (span three). Each span has a start, a duration, and a parent. Lay them out and you get a waterfall — a visual timeline of exactly where that one request spent its time, and exactly where it fell over.

When your metrics scream "checkout is slow," the trace is what walks you down the hallway, past the five services that were perfectly fine, and stops you at the door of the one that wasn't. Traces answer the question metrics can't: where?

Logs — the security-camera footage

When the trace stops you at the payment service's door, the log tells you what actually happened inside the room: "payment processor declined: connection timed out after 5000ms." Logs are the detailed, event-level story of one specific thing. Richest signal you have, and the most expensive to keep — which is why they're only useful if they're written like data, not diary entries.

Here's the difference, because it matters. This is a log written like a diary entry:

2026-08-02 02:14  payment failed for bob

Useless. To find every failed payment you're back to grep and regex. Now here's the same event written like data:

{"level":"error","service":"payments","trace_id":"a1b2c3…","msg":"processor declined","reason":"timeout"}

That you can query: "give me every error from the payments service in the last hour." The rule for logs, then, is short: log JSON, one event per line, and put a trace ID in every single one. (Hold onto that trace ID. It's about to become the hero of the story.)

So: metrics say is it broken. Traces say where. Logs say what. Three tools, three questions, one investigation.

Now for the part almost every tutorial skips — the three ideas that separate people who merely own dashboards from people who can actually use them. They're all a little counterintuitive, and once they click, you will never look at a system the same way.

Truth #1: Averages are filthy liars

Say your checkout handles a hundred requests. Ninety-nine come back in 100 milliseconds. One takes ten full seconds.

The average? About 200 milliseconds. You glance at it and think: healthy. Ship it.

But one in every hundred customers just waited ten seconds for a page. At scale that's thousands of furious people a day, and your dashboard smiled the entire time. The average didn't lie by being wrong — it lied by being an average. It smeared one catastrophe across ninety-nine good experiences until it vanished.

There's an old line: a statistician drowned crossing a river that was, on average, three feet deep.

Caption: "the average response time is fine, sir."

This is why people who run big systems almost never look at averages. They look at percentiles. The p50 (median) is your typical experience. The p95 is what most users feel — only one in twenty is slower. The p99 is the painful tail, the one-in-a-hundred where the angry tweets are born. When an engineer says "our p99 blew up," they mean "the average looks fine, but our worst-served users are suffering, and I can see it." That visibility is the whole difference between a system you think is healthy and one you know is.

Truth #2: The thing that makes metrics cheap can also blow them to pieces

Metrics are cheap because they're aggregated. But there's a landmine buried in that word, and it takes down real systems.

Every metric can be sliced by labels: error rate by service, by status code, by region. Each unique combination of labels is a separate line the database must store and hold in memory. Slice by service and status? Maybe sixty lines. Fine. But the second someone adds a label like user_id, and you have a million users, you've just asked your metrics database to track sixty million lines. It falls over and dies. This has a name — cardinality explosion — and it's one of the most popular ways teams accidentally nuke their own monitoring.

Caption: "me, after adding user_id as a metric label."*

The lesson is a rule that keeps you alive: labels are for things with a small, bounded set of values — service, status code, region. They are never for unbounded things — user IDs, emails, request IDs. And this is exactly where the three signals snap together into a system: when you want per-user, per-request detail, you don't cram it into a metric. That's what logs and traces are for. Every signal has a job. The whole art is not forcing one to do another's.

Truth #3: The real magic is the thread that ties all three together

Here's the idea that turns this from "three separate tools" into a superpower.

Remember that trace ID I told you to keep in every log line? It also rides along on every metric and every span. One string, following one request across every service it touches. And when all three signals carry the same one, something genuinely magical happens.

You're staring at a metric spike. You click it — and you're looking at an example trace from that exact moment. You see the payment span glowing red. You click that — and you're reading the exact log lines that request produced: "processor timeout." You never opened a terminal. You never guessed. You followed one thread from "something's wrong" all the way to "here's the precise line and the precise reason," across a system written in three languages, in about ninety seconds.

Caption: "…all carrying the same trace_id."*

For that thread to survive, one thing has to work. When service A calls service B, it has to pass the trace ID along, tucked inside the request. There's an open standard for this — a header literally called traceparent — and every service in the chain reads it and continues the same story instead of starting a new one. Break that hand-off anywhere (an ancient library, a queue that drops the header) and your one beautiful trace shatters into disconnected fragments and the magic dies.

The good news is you don't wire this by hand per vendor. OpenTelemetry is the open, vendor-neutral standard for producing all three signals and passing that thread along. You instrument your code once against it, and you can send the data to any backend you like without rewriting anything. Instrument once, send anywhere, never get locked in. So the entire discipline, in one breath: instrument everything with OpenTelemetry, and never drop the thread.

Truth #4 (bonus): Alert on what your users feel, not what your servers feel

This one saves your sleep, so it earns a spot.

You could alert on a thousand things. Most of them will wake you at 2 a.m. for absolutely nothing. The rule that rescues your sanity: alert on symptoms your users feel, not on internal causes.

High error rate is a symptom — someone can't check out right now. Wake me. Slow p95 is a symptom — the app feels broken. Wake me. But CPU at 90%? That is not a symptom. A server pinned at 90% while happily serving fast, successful responses is a server doing its literal job. Page a human for that and you've taught your team that alerts are noise — so the day a real one fires, everyone scrolls past it. CPU and memory are for diagnosis, something you check after a symptom already pulled you in. They should never be the thing that pulls you in.

And give every alert a little patience: "error rate above 5% for two minutes," not "for one second." That short wait absorbs the meaningless blips so the only things that ever wake you are real. An alert that isn't worth acting on isn't an alert. It's a notification you'll learn to hate.

What it looks like when it's all actually running

Concepts are slippery until you watch them move, so let me paint the finished picture.

Imagine a small but real distributed system — three services, and on purpose written in two languages: a gateway in Node.js, an orders service in Python, and a payments service back in Node.js. A single request to the gateway fans out through all three, exactly like your checkout. Around them sits the modern stack: Prometheus collecting metrics, Loki holding logs, Tempo storing traces, an OpenTelemetry Collector routing all of it, and Grafana on top as one single pane of glass.

Send one request, and you watch it become all three signals at once. It shows up as a spike on the metrics dashboard. It shows up as structured log lines. And it shows up as one distributed trace whose waterfall runs from Node, into Python, and back into Node — living proof that the thread survived the trip across languages. Click the trace, jump to its logs. Click a log line, jump back to its trace. That round trip, in one tool, is the entire idea made physical. Watching a single request stitch three languages into one story is the moment observability stops being a buzzword and becomes obvious.

Now rewind to 2 a.m.

Same night. Same fire. But this time your system can explain itself.

Your phone buzzes. Error rate climbing. You open the dashboard — but instead of shrugging at "something is wrong," you click into an example failing trace from that exact minute and watch the waterfall light up the payment span in angry red. You click it. The logs are right there: "payment processor declined: connection timed out." You check the processor's status page — they're having an incident. You flip a feature flag to fail over to your backup provider, checkout recovers, and you are back in bed before the coffee would have finished brewing.

Caption: "found it in 90 seconds. good night."

Same fire. Same twenty services. The difference isn't that you got smarter at 2 a.m. It's that your system could finally tell you where it hurt.

Where to start, if you're starting from zero

Don't try to boil the ocean. Do these three, in this order:

First, put the Four Golden Signals — latency, traffic, errors, saturation — on one dashboard, so you have honest vital signs. Second, make your logs structured: JSON, one event per line, a trace ID in every one, so they're queryable instead of grep-bait. Third, add tracing with OpenTelemetry, so you can follow one request across every service and jump between all three signals.

Do that, and the next time something breaks across your services, you won't be the person SSH-ing into boxes and guessing at 2 a.m. You'll be the person who clicks three times, points at the exact failing service, says "found it," and goes back to sleep while everyone else is still reaching for the terminal.

That, in the end, is all observability really is: giving your system a voice, so that when it's hurting, it can finally just tell you where.

If you'd like to see all of this running — the three services, the cross-language trace, every signal lighting up in Grafana — I built the whole thing as a hands-on demo and walk through it live. Even better, it runs with one command, so you can break it on purpose and watch it tell on itself. Link's in the comments.

"an AI-interview experience with zero AI and zero cost." Opens with the three bills you didn't pay (TTS, Whisper, proctoring vendor)

Sheryar Ahmed — Tue, 07 Jul 2026 20:57:07 +0000

A mock interviewer that speaks, listens, and proctors with no AI bill

Demo: https://drive.google.com/file/d/13WEiJXqwq7O6XzouhUR0LpNfgz67znd9/view?usp=sharing
Mentors on my platform wanted to run practice interviews with their mentees: a voice
asks a question, the candidate answers out loud, it gets recorded, and — because it's
practice for the real thing the session is proctored. Every off-the-shelf version of
this hands you three metered bills: a TTS service to read the question, an STT
service (usually Whisper) to transcribe the answer, and a proctoring vendor to watch the
room. Three subscriptions for a feature people use in short bursts.

The naive plan is to just pay them. Wire up a cloud TTS voice, POST each audio answer to
a speech-to-text endpoint, embed a proctoring SDK, and eat the per-minute cost forever.
It works, but every one of those is a recurring charge, a data-egress path, and a privacy
footprint — for something that's fundamentally the candidate's own browser talking to
the candidate's own microphone. I didn't want a network round-trip sitting between a
mentee and a practice question.

The decision

I asked how much of the stack the browser already ships, and the answer was: nearly all
of it. The whole "AI interviewer" experience is three native Web APIs in a trench coat.

TTS — the interviewer speaks. window.speechSynthesis renders the question on-device.
The mentor even tunes the pitch/rate so the interviewer ("Aria") has a personality. No
audio is generated on a server; nothing is downloaded.

STT — the answer transcribes itself. SpeechRecognition (webkitSpeechRecognition
in Chromium) does live, streaming transcription on-device — interimResults means
words land on screen as they're spoken, the same engine behind your phone keyboard's mic
button. No upload, no Whisper invoice.

Audio streaming — the recording is the ground truth. MediaRecorder captures the mic
to a chunked audio/webm blob. That blob is the only byte that touches my server, and
it goes to storage I already pay for.

// TTS: the interviewer reads the prompt (mentor-tuned voice)
const u = new SpeechSynthesisUtterance(prompt);
u.rate = interviewer.rate; u.pitch = interviewer.pitch;
speechSynthesis.cancel();            // never overlap two questions
speechSynthesis.speak(u);

// STT: live, streaming transcript — no server in the loop
const rec = new (window.SpeechRecognition || window.webkitSpeechRecognition)();
rec.continuous = true; rec.interimResults = true;
rec.onresult = (e) => setTranscript([...e.results].map(r => r[0].transcript).join(''));

// Audio streaming: MediaRecorder is the ground truth if STT is unsupported
const mr = new MediaRecorder(stream, { mimeType: 'audio/webm' });
mr.ondataavailable = (e) => e.data.size && chunks.push(e.data);

The key judgment call is layering, not just "use the free thing." STT support is uneven
(Firefox notably lags), so transcription is never the source of truth — the recording
is. STT degrades to a nicety; the feature never breaks. And a green-room mic check runs a
lightweight VAD-style level meter off the Web Audio AnalyserNode (RMS on the time-
domain data) so a candidate can see the mic is live before the clock starts — again,
zero network, pure getByteTimeDomainData.

Tally: TTS $0, STT $0, proctoring $0. The only recurring cost is the audio blob storage I
already had.

The gotcha

Proctoring is where "free browser signals" nearly bit me. The harness itself is cheap and
honest — fullscreen lock, tab/blur focus-loss counting, copy/paste/right-click blocking,
periodic webcam snapshots, all buffered client-side and flushed in batches. The trap was a
kit option I was quietly proud of: "require camera."

I shipped it, then during testing found the guarantee was hollow. A candidate could allow
the camera at the start, then turn it off mid-interview from the browser's site
controls. The session sailed on — and worse, my snapshot loop kept firing, dutifully
uploading black frames. On the surface everything looked compliant. "Require camera"
that you can switch off after ten seconds isn't a requirement; it's decoration.

The fix is one signal: a MediaStreamTrack's health. When a camera is turned off, covered,
or unplugged, its track goes muted/ended and readyState stops being 'live'. Poll
that, and two behaviors fall out of the same check.

const track = (video.srcObject as MediaStream)?.getVideoTracks?.()[0];
const live = !!track && track.readyState === 'live' && track.enabled && !track.muted;

// (1) block instead of silently continuing
if (!live) { setCameraLive(false); log('camera_off'); }   // → full-screen "turn it back on" overlay

// (2) never upload a black frame from a dead camera
if (!track || track.readyState !== 'live' || track.muted) return; // skip the snapshot

Now when the camera drops, the runner throws a full-screen block — "your camera is off,
turn it back on to continue" with a one-click Retry camera — and the candidate can't
answer or advance until it's back. The clock keeps running and the off→on gap is logged,
so turning the camera off to buy thinking time is a recorded act, not a free one. And the
mentor's proctor gallery gets a clean "Camera turned off" flag instead of a wall of
black rectangles. Same track state, both jobs.

What I'd do next

STT confidence + a manual-correction pass. On-device recognition trails Whisper on accents and noise. I'd surface a confidence score and let the candidate fix the transcript, keeping the audio as the authority.
Real VAD, not just a level meter. The RMS reading is enough for a "mic is live" cue; proper voice-activity detection could auto-trim dead air and auto-advance on silence.
Honest limits on proctoring. Client-side signals raise the cost of cheating; they don't make it impossible (a second device still exists). The goal is deterrence and an auditable timeline for a mentor — not a courtroom — and the UI should say so.

The meta-lesson: before you add an AI line-item, check what window already does for free.
The browser quietly grew a whole speech-and-media stack — TTS, streaming STT, MediaRecorder,
Web Audio VAD, MediaStreamTrack health — and most of us keep reaching for an API instead.

What's the last feature you paid a vendor for that the platform could already do TTS,
STT, or something else?

I added dark mode by editing CSS variables not 100 components

Sheryar Ahmed — Tue, 30 Jun 2026 13:21:06 +0000

I added dark mode by editing CSS variables - not 100 components

My app had ~1,200 hardcoded indigo-* and ~2,600 hardcoded slate-* class usages across
130-odd components. The brief was small to say and brutal to do: "ship dark mode, and
let me rebrand the accent color whenever I want."

The naive plan is a sweep: open every component, add dark: variants, swap colors. That's
weeks of work, and worse, it's permanent work - every new component re-incurs it, and
the design drifts the moment two people touch it. I wanted dark mode to be a property of
the theme, not of every component.

The decision

Two ideas, both "edit the tokens, not the markup":

1. One brand token. Instead of components knowing the color indigo, they reference a
semantic brand scale. I did a one-time codemod - indigo- → brand- across app/,
components/, lib/ (~1,230 mechanical edits, no logic change) - and defined the brand
scale once as CSS variables. Now recoloring the entire app, globally or per-user, is a
variable swap. No component ever needs editing again.

:root {
  --brand-50:#EEF4FF; /* … */ --brand-500:#0066FF; --brand-600:#0052D6; /* … */
}
/* Tailwind v4: expose them so bg-/text-/border-/ring-brand-* all resolve to the vars */
@theme inline {
  --color-brand-500: var(--brand-500);
  --color-brand-600: var(--brand-600);
  /* …the full scale… */
}

Per-user accent "vibes" then become trivial - each preset just re-defines that scale under
an attribute selector, so picking one recolors the app instantly with zero component churn:

html[data-accent="emerald"] { --brand-500:#059669; --brand-600:#047857; /* … */ }

2. Dark mode as a token remap, not a component sweep. This is the part that surprised
me.

The mechanism (and the wrong turn I took first)

My first attempt failed silently. I tried mapping Tailwind's slate scale through my own
indirection in @theme inline - and text stayed dark in dark mode. Tailwind's default
slate-* utilities weren't picking up my override.

Then I read the compiled CSS, and the trick fell out of it. Tailwind v4 emits its own
palette as real CSS variables, and the utilities reference them:

.text-slate-900 { color: var(--color-slate-900); }

So I don't need to touch a single utility. I just redefine the slate ramp inside the
.dark selector - reversed, so "dark text" becomes light and "light background" becomes
dark:

.dark {
  --color-slate-900: #f1f5f9;  /* was near-black → now near-white text */
  --color-slate-700: #cbd5e1;
  --color-slate-200: #3a4659;  /* was light border → now a dark border */
  --color-slate-50:  #273345;  /* was page-white → now an elevated surface */
  color-scheme: dark;          /* native scrollbars/date pickers follow too */
}

Every text-slate-900, bg-slate-50, border-slate-200 in the codebase flips
automatically. Thousands of usages, one block of CSS.

The gotcha

A reversed ramp isn't a straight inversion - surface hierarchy breaks if you're naive.
My first ramp made slate-50 darker than my card color, so every "subtle gray" inner box
(inside drawers and cards) suddenly looked like the page bleeding through the card. The
fix was to treat the low slate shades as elevated surfaces that sit above the card,
not below it: canvas (#0b1120) < card (#1e293b) < slate-50/100/200. Dark mode isn't
"flip the colors" - it's "preserve the depth ordering with a dark palette." Miss that and
everything looks flat and muddy.

A second, smaller one: native <select> option lists. Components set a dark background on
the control, but the option popup inherited the dark bg without a light text color -
dark-on-dark, invisible. One global rule fixed all of them at once:

select option { background-color: var(--card); color: var(--card-foreground); }

What I'd do next

A no-flash inline script in <head> to set the theme before first paint (right now a provider gates render, which avoids the flash but costs a frame).
Audit the handful of intentionally dark elements (a dark CTA, a tooltip) that I had to pin to fixed colors so the remap couldn't invert them - those are the exceptions that prove the rule.
A contrast-ratio check in CI so a future accent preset can't ship an unreadable pair.

The meta-lesson: if changing a visual property means editing many files, the property is
in the wrong layer. Push color and theme into tokens, and "rebrand the app" or "add dark
mode" turns from a sprint into a diff.

What's your dark-mode strategy - dark: everywhere, CSS variables, or something else?

DEV Community: Sheryar Ahmed

Why Kubernetes Says ‘Insufficient CPU’ When the Node Looks Idle

The short answer

A node can be idle and still be “full”

Request, limit, and usage are three different numbers

Capacity is not allocatable CPU

Debug it in five minutes

1. Read the complete scheduling event

2. Inspect the Pod's actual request

3. Check each eligible node's allocatable and requested CPU

4. Use live usage for diagnosis—not for placement math

Four traps that make this more confusing

1. Free CPU is fragmented across nodes

2. A namespace policy may add requests for you

3. DaemonSets consume space on every node

4. “Insufficient CPU” may appear beside other hard constraints

What should you fix?

The mental model to keep

Further reading

Why a Kubernetes Pod Being 'Running' Doesn't Mean It's Healthy

Why a Kubernetes Pod Being Running Doesn't Mean It's Healthy

The first mental model

Running means something very specific

So how does Kubernetes know whether a container is healthy?

1. Startup Probe

"Has the application finished starting?"

2. Readiness Probe

Readiness asks:

3. Liveness Probe

The dangerous liveness probe

The three probes in one picture

But there's another trap

The subtle lesson: Readiness doesn't stop everything

Kubernetes health is layered

A Pod can look perfectly healthy and still be failing

What should you actually monitor?

Kubernetes signals

Probe signals

Application signals

Business signals

A practical debugging flow

Step 1: Is it running?

Step 2: What are the conditions?

Step 3: Which probe is failing?

Step 4: Test the endpoint yourself

Step 5: Check application logs

Step 6: Check actual business behavior

A useful production rule

Readiness

Liveness

Business monitoring

The final mental model

The 10-second revision

What I learned from this

Kubernetes Isn't a Container Manager, It's a Reconciliation System

Subtitle

Introduction

1. Desired State vs Actual State

2. The Core Loop

3. Where does the Desired State live?

4. Controllers Are the Reconciliation Engine

5. Kubernetes Doesn't Guarantee That Desired State Is Possible

6. Running Doesn't Mean Healthy

7. The Same Idea Appears Everywhere

Deployment

Horizontal Pod Autoscaler

Cluster Autoscaler

KEDA

8. A Better Way to Debug Kubernetes

Question 1 — What is the desired state?

Question 2 — What is the actual state?

Question 3 — Who is responsible for closing the gap?

Question 4 — Why can't the next reconciliation step succeed?

9. The Mental Model I Keep

Conclusion

When something breaks across 20 microservices, one question decides how fast you recover

It's 2 A.M. and Your Dashboard Just Says "Something Is Wrong"

First, a distinction that everything else hangs on

The three signals (a.k.a. your detective kit)

Metrics — the vital-signs monitor

Why a Kubernetes Pod Being `Running` Doesn't Mean It's Healthy

`Running` means something very specific