Last time I told you to bring coffee. I did. This one didn't start at 2 a.m. — it started at 10:40 on a perfectly ordinary Tuesday, which, if anything, is worse. There's a particular kind of dread that only arrives in full daylight, when you're well-rested and caffeinated and therefore have no excuse for not understanding what's happening. And I did not understand what was happening.
Welcome back to Troubleshooting Kubernetes. Today's episode is about the most unsettling category of outage there is: the one where nothing is broken. Every light is green. Every pod is Running. Every dashboard is calm. And your users are getting 500s.
I've been doing this long enough — a couple of decades and change, across a few places I've been lucky to work — to have developed a healthy distrust of green. Green is not a fact. Green is a claim. And on that Tuesday, something was lying to me with a very straight face.
The symptom that wasn't there
The error rate on the checkout API had crept from basically-zero to about 18% over twenty minutes. Not a spike. A creep — the kind that makes you wonder if it's real or if the graph is just having feelings. So, first move, always: is this real, or is the monitoring lying?
# hit it through the front door, a bunch of times
for i in $(seq 1 20); do
curl -s -o /dev/null -w "%{http_code}\n" https://checkout.internal/api/health-of-a-real-endpoint
done
Real. About one in five came back 500. And "about one in five" is a clue, not just a number — an intermittent failure spread across requests usually means it's spread across replicas. Some of the pods behind the Service were fine. Some were not. The load balancer was cheerfully dealing everyone a random card from a deck that had a few jokers in it.
So I went to look at the pods, fully expecting to find a couple of them crash-looping or NotReady or otherwise waving their hands.
kubectl get pods -n checkout -o wide
Five replicas. Running. 1/1. Ready. Every last one of them, calm as a millpond. Zero restarts. According to Kubernetes, this service was in perfect health. According to the customers, it was on fire. Those two things cannot both be true, and yet there they were, both being true, mocking me over my second coffee.
Finding the jokers
If the Service is routing to five pods and some are bad, I want to know which ones — go around the load balancer and interrogate each pod directly. The endpoints list tells you who's actually in rotation:
kubectl get endpoints checkout-api -n checkout -o wide
All five pod IPs, present and accounted for, all listed as ready targets. Fine. Let's talk to them one at a time.
# talk to a single pod, bypassing the Service
kubectl port-forward pod/checkout-api-6c8b9-x2k7p 8080:8080 -n checkout
curl -s localhost:8080/api/checkout/quote -d '{...}' # returns 500
curl -s localhost:8080/healthz # returns 200 OK
There it was. On the bad replicas, the real endpoint threw a 500, and the health endpoint returned a serene, confident 200 OK. The pod was, in the most literal sense, telling Kubernetes it was ready to serve — while being completely unable to serve.
Two of the five were doing this. And Kubernetes, being a faithful and literal machine, kept both of them in the endpoints list and kept dealing them to real customers, because they had passed the only test it knew to run.
The lie, and who told it
Here's the readiness probe those pods were configured with — and I'd bet real money it looks familiar, because it's the one everybody writes on their first day and never revisits:
readinessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
And here, roughly, is what /healthz did:
GET /healthz → "am I an HTTP server that is currently running?" → 200
That's it. That's the whole check. It confirms the process is up and the socket is listening. It does not ask the one question that actually matters to a customer: can you do your job right now? It never touches the database. It never checks the connection pool. It is a smoke detector wired to confirm that it has electricity.
What had actually happened: the database had done a failover about twenty-five minutes earlier — routine, expected, mostly graceful. Most of the app replicas noticed, dropped their stale connections, and reconnected to the new primary. Two of them didn't. Their connection pools were wedged full of dead connections to an IP that no longer answered, so every real query timed out into a 500. But the HTTP server? Still up. Still listening. Still cheerfully answering /healthz with a 200.
So the pods weren't broken from Kubernetes' point of view. They were doing exactly what they'd promised. The lie wasn't Kubernetes'. Kubernetes was the most honest actor in the whole incident — it did precisely what it was told, with no imagination whatsoever. The lie was in the probe. We had told it that "the web server is up" meant "this pod can serve customers," and it had believed us, because why wouldn't it. The green dashboard was technically, uselessly accurate.
I have a lot of respect for that, honestly. It's the same trait that saved my data in the last installment — Kubernetes doing the literal thing, refusing to be clever. It'll protect you from yourself and it'll faithfully execute your mistakes with equal diligence. The machine isn't the problem. The machine is a mirror.
Getting the jokers out of the deck
Two things to do, in order: stop the bleeding, then fix the actual bug.
Stop the bleeding. The two wedged replicas just needed their connection pools reset, and the fastest way to reset a pod's everything is to let the Deployment give you a fresh one:
kubectl delete pod checkout-api-6c8b9-x2k7p checkout-api-6c8b9-9m4tz -n checkout
New pods came up, connected cleanly to the new primary, and the 500s stopped inside a minute. Error rate back to zero. Crisis over. And this is exactly the moment where a tired engineer declares victory, closes the incident, and goes to lunch having fixed nothing — because the bug wasn't the wedged pool. Connection pools wedge. Databases fail over. That's Tuesday. The bug was that a pod that couldn't serve was allowed to keep serving.
Fix the actual bug. The readiness probe has to check what actually matters. A real /readyz that returns 503 when the pool can't hand out a working connection:
readinessProbe:
httpGet:
path: /readyz # checks a DB connection, not just the socket
port: 8080
periodSeconds: 5
failureThreshold: 3
Now a wedged replica reports itself not ready, Kubernetes pulls it from the Service endpoints automatically, and traffic routes only to pods that can actually do the work. The self-healing you thought you already had, you now actually have.
But — and this is the part people skip, the part that turns a fix into a second outage — do not get greedy with your readiness probe. If every replica shares one database, and you wire readiness directly to that database, then the next time the DB hiccups for fifteen seconds, all five pods simultaneously report not-ready, Kubernetes yanks every endpoint, and now you've converted a fifteen-second blip into a total, self-inflicted outage with a thundering-herd reconnect on the far side. The probe should reflect this pod's ability to serve, with enough tolerance (failureThreshold, sane timing) that a shared, transient dependency wobble doesn't get amplified into a coordinated group suicide. Readiness pulls a sick pod from rotation. It should not pull a nervous one.
And keep liveness and readiness doing different jobs. Liveness answers "should I be restarted?" — reserve it for genuinely wedged, unrecoverable states, because a liveness probe tied to a shared dependency is how you turn a database blip into a cluster-wide restart storm. Readiness answers "should I get traffic?" Conflate them and you'll eventually get both failure modes at once, which is a bad day I'll tell you about some other time.
Why green lies, and why I still love this
The lesson underneath the lesson: a health check is a promise your application makes about itself, and your monitoring is only ever as honest as that promise. The dashboard wasn't wrong. It was faithfully reporting a claim that happened to be worthless. If you take one thing from this: write the probe that checks the thing your users actually depend on, not the thing that's easy to check. The gap between those two is where 10:40-on-a-Tuesday lives.
That gap — the difference between "the process is up" and "the thing actually works" — is basically the whole reason I started keeping structured notes on this stuff, and eventually turned them into a site. When you're staring at a green dashboard and a red reality, the useful thing isn't a metrics graph, it's a hypothesis: "some replicas can't reach a dependency; check the endpoints and probe each pod directly." I'll sometimes paste the raw logs into the AI incident assistant I keep on the site just to get to that first hypothesis faster — not because it knows the answer, but because at minute three of an incident, a decent starting question is worth more than another dashboard. Sanitize your secrets before you paste, obviously. The judgment is still yours; the tool just hands you the thread to pull.
Here's the thing I keep coming back to, twenty-odd years in: this incident was beautiful, in the specific way that only a good bug is beautiful. Nothing was broken. Every component behaved exactly as designed. And the emergent result was still wrong, because of a single lazy assumption baked into six lines of YAML that someone — probably me, honestly — wrote in a hurry two years ago and never looked at again. Untangling that, watching the green dashboard finally mean something, is a feeling I have never once gotten tired of. The pages are annoying. The 10:40 Tuesdays are worse. And I would not trade this job for a quieter one, because a quieter one wouldn't let me do that.
Next time in Troubleshooting Kubernetes: the DNS lookup that resolved perfectly from my laptop, resolved perfectly from the node, and failed only — only — from inside the pod. Bring coffee. Bring patience. Bring a copy of the ndots documentation you've never actually read.
The running set of Kubernetes runbooks, probe patterns, and error guides I keep so I'm not re-deriving them mid-incident lives on the Kubernetes toolkit. Green is a claim. Make your probes tell the truth.
This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.
Top comments (0)