Kubernetes health checks tell you whether a workload satisfies the narrow condition the probe was designed to test. They do not tell you whether the business objective that workload exists to serve actually happened. That distinction sounds pedantic until it's the reason a deployment shows green across every dashboard while the checkout flow silently stopped completing orders forty minutes ago.
Passing health checks prove participation in orchestration. They do not prove the intended outcome occurred.
What Kubernetes Actually Guarantees
Strip away the operational folklore and a readiness or liveness probe answers a narrow question: did the process respond to this specific check, within this specific timeout, in this specific way? A liveness probe confirms the process hasn't deadlocked or hung badly enough to fail its own restart threshold. A readiness probe confirms the process is willing to accept traffic right now. A startup probe confirms initialization completed before the kubelet starts evaluating the other two.
None of these checks execute business logic. None of them touch a real code path a customer would traverse. A readiness probe hitting /healthz and getting a 200 tells Kubernetes that the container is ready to receive traffic — it does not tell Kubernetes "I can complete a checkout," "I can authenticate a user," or "I can write a record that will still be readable tomorrow." The probe's contract is with the control plane, not with the business objective the control plane exists to serve.
This is the correct design, not a flaw. Kubernetes has no way to know what your business objective is, and it would be architecturally wrong for the orchestrator to try to infer it. The control plane's job is process lifecycle management: keep the right number of willing replicas in rotation, and route traffic only to the ones that said yes. It does that job well. The failure isn't in what Kubernetes guarantees — it's in what architects quietly assume it guarantees once the dashboard turns green, and closing that gap is squarely a cloud architecture strategy decision, not an implementation detail to delegate downward.
What Kubernetes Does Not Guarantee
The gap between "process is participating" and "objective is being met" isn't unique to Kubernetes. It shows up at every layer of the stack where a proxy signal stands in for direct measurement of the thing you actually care about. Six of the most common false-positive patterns:
| Signal | What It Confirms | What Still Fails Underneath |
|---|---|---|
| Readiness probe | Process accepts a TCP connection or returns 200 on a health endpoint | Downstream dependency (database, cache, upstream API) is unreachable; traffic routes to a pod that can't complete a real request |
| DB connection check | Pool has a live connection, SELECT 1 returns |
Replication lag has pushed reads stale, or a lock is queuing writes indefinitely while the ping itself never touches the locked table |
| LB target health check | Target instance responds on the configured health path/port | Application logic returns malformed data or a 5xx on the actual transaction path the health path never exercises |
| Circuit breaker | Breaker reports closed — failure rate hasn't crossed the trip threshold | Individual transactions fail below the statistical threshold the breaker was tuned to catch; the breaker is doing its job correctly and that job doesn't cover this case |
| Backup job | Process exits 0 | Backup is incomplete, unusable, or not restorable even though the job reported successful completion |
| Auth/identity service | Port is reachable, service responds to a liveness ping | Token validation logic can fail open under certain error conditions, and a liveness ping never exercises those conditions |
Six control points, six different ways a passing signal hides a real failure.
The pattern repeats because every one of these checks was built to answer a cheap, fast, narrow question — is the endpoint alive — rather than an expensive, slow, broad one — did the operation actually accomplish what it was supposed to. That tradeoff is usually correct. The problem is treating the cheap question's answer as though it settled the expensive one. The readiness-probe row above isn't hypothetical, either — the Network Loop in the Rack2Cloud Method's Day-2 operations guide documents the same failure mode from the routing side: a misconfigured readiness probe sending traffic to a pod that can't actually serve it.
Why Healthy Systems Still Fail
Here's the scenario that makes this concrete. A deployment rolls out. Every readiness probe passes within its configured threshold. The rollout controller reports success. The dashboard shows 100% of replicas ready, zero restarts, latency within SLO. By every signal Kubernetes was designed to emit, this deployment succeeded.
Forty minutes later, support tickets start arriving. Checkout is broken — not down, broken. Users click "place order," get a spinner, and the order never appears in the downstream fulfillment system. The readiness probe on the checkout service never checked whether it could reach the fulfillment queue. It checked whether the process was up. The process was up. The queue connection, established once at startup and never re-validated, had gone stale after a network policy change nobody connected to this deployment.
The severity isn't hypothetical. The proxy can remain healthy while the business function is already broken. Nothing in the orchestration layer was lying — the rollout genuinely succeeded by every metric Kubernetes tracks. That's the uncomfortable part — this isn't a monitoring gap you close by adding more Kubernetes-native checks. It's a category gap. You can add ten more readiness probes and still never catch this, because readiness probes operate at the orchestration-participation layer and the failure lives at the business-outcome layer. No amount of depth within the wrong category closes a gap between categories.
Contrast this against the inverse case: a probe that fires correctly and gets misdiagnosed as something else entirely. Kubernetes Day-2 Incidents documents a CrashLoopBackOff traced back to an IAM permission gap, where exit code 0 on a crash loop means the readiness probe itself is failing — a different problem entirely, and one where the probe is doing exactly what it should. That post is about misdiagnosing which layer causes a visible failure. This one is about a signal correctly reporting success while the objective still fails. Both are real; they're not the same failure.
The False Completion Pattern
What just happened has a name, and it's not specific to Kubernetes. This is False Completion — the failure mode in which an operation reports success by system metrics while the underlying objective was not actually met.
The pattern generalizes cleanly. Swap the readiness probe for an ALB health check, a backup job's exit code, or a circuit breaker's closed state, and the shape of the failure is identical: a signal designed to answer a narrow, cheap question gets treated as authoritative for a broad, expensive one it was never built to answer. The same pattern shows up applied to HTTP response codes — a 200 status reporting success while the actual transaction underneath it failed. Kubernetes is the evidence layer in this post, not the root cause — the root cause is architectural, and it recurs everywhere a proxy metric substitutes for direct measurement of the objective.
False Completion tends to survive in production specifically because the proxy signal is healthy. A system that fails loudly gets fixed. A system that reports success while silently missing its objective can run for weeks. The gap surfaces at incident time, when a human notices the business symptom and has to work backward through several layers of "but the dashboard was green" to find it. That's the most expensive possible time to discover it, and it's discovered there specifically because nothing upstream was built to surface it earlier.
The architectural lesson isn't "distrust Kubernetes." It's: identify every point in your stack where a proxy signal stands in for a business outcome, and treat the distance between them as a permanent, unclosed risk unless you've specifically instrumented for it — the discipline that operational architecture exists to enforce once a system is already running.
What To Measure Instead
Closing the gap means adding a layer that measures the objective directly, not adding more instances of the same proxy signal.
01 — Synthetic Transactions
Run the actual business operation — a real checkout, a real login, a real write-then-read — on a schedule, from outside the cluster's health-check surface, and alert on the transaction's outcome, not on whether an endpoint responded.
02 — User-Journey Validation
Instrument the multi-hop path a real user takes — not the single service boundary a probe checks — so a failure three hops downstream from the deployed service still surfaces against that deployment.
03 — Dependency-Aware Checks
Extend readiness logic to actually exercise the dependency it currently assumes — a real query against the database it needs, a real round-trip to the queue it writes to — rather than confirming the process can reach its own health endpoint.
04 — Outcome-Based SLOs
Define the SLO against the business outcome — orders completed, tokens validated, backups restorable — not against pod readiness percentage or request latency, which measure the orchestration layer's health, not the objective's.
Note the boundary on #03: dependency-aware readiness is a real improvement over a bare /healthz endpoint, and Kubernetes' own probe model explicitly supports checking required backend services. But it still isn't business-outcome validation. A readiness check can verify that the database is reachable. It still doesn't prove that the checkout transaction completed successfully. Extending readiness logic closes one gap without closing this post's gap — the two are adjacent, not identical.
None of this replaces Kubernetes-native health checks. It sits on top of them, measuring the thing they were never designed to measure. The 30% of this post that's remediation isn't the argument — it's the proof the argument has a floor. The thesis is the gap; this is just where it closes.
Measuring the objective directly closes a gap that adding more proxy checks cannot.
Architect's Verdict
A system can satisfy every signal it was designed to emit while failing the objective the business actually cares about. That's not a monitoring bug. It's what happens when a cheap, narrow proxy question gets treated as though it answered an expensive, broad one — and Kubernetes health checks are just the most common place that substitution happens quietly, because the green dashboard is so convincing.
What most architects miss isn't the existence of the gap — everyone will nod along to "health checks aren't business logic" in the abstract. What they miss is that the gap doesn't announce itself. It survives in production precisely because the proxy signal stays healthy, which means the systems most exposed to False Completion are the ones that look, on every dashboard you already trust, like they're working fine.
Passing health checks prove participation in orchestration. They do not prove the intended outcome occurred.
Originally published at rack2cloud.com
Top comments (0)