This is article 5 of 10 in Beyond Kubernetes Defaults. Read article 4: Kubernetes Secrets Are Distribution, Not Custody first, or start with article 1.
Kubernetes gives us useful workload states: scheduled, Running, Ready, unhealthy, terminating. They are operationally important and precisely defined, but narrower than many application state machines.
The misleading green Pod
Consider an illustrative workload that is Running but unable to participate safely in its application. The process exists, the container started, and a health endpoint answers. Yet participation may still depend on enrollment, current workload identity, correct tenant and placement binding, available credentials, remote admission, desired state, current sessions and durable recovery.
The process is healthy. The workload is not yet admitted.
Five different questions
- Liveness: Is the process functioning well enough that a restart is not warranted?
- Kubernetes readiness: Should this Pod be eligible for Service traffic?
- Application admission: Has the workload joined its system correctly?
- Application authority: May it perform this operation now?
- Convergence: Has it applied the state and generation expected of it?
Those answers can legitimately differ. A follower can be healthy and authenticated but not authoritative. An agent disconnected from management can be alive but unable to receive work. An expired credential should fail closed even if its process is reachable.
What the probes actually measure
The Kubernetes probe documentation describes three kinds:
- A startup probe gates liveness and readiness until it first succeeds.
- A failing liveness probe triggers a container restart, subject to the Pod's restart policy.
- A failing readiness probe makes the Pod ineligible for normal Service traffic; it does not restart the container. Exact routing depends on how traffic reaches the Pod, and direct Pod access may behave differently.
None of these alone establishes enrollment, credential validity or the expected application generation. But Kubernetes can express additional state: custom Pod conditions and readiness gates can participate in readiness, and status conditions can carry structured information. API admission controllers and Pod Security Admission answer a different question: whether a Pod specification is permitted, not whether its running application has joined its system.
Define application readiness first. Then decide how probes, conditions and status should reflect it.
One boolean often loses the reason
A generic Ready/not Ready result can hide the cause. Bounded reason codes can distinguish a stale heartbeat, expired credential, missing recovery state, an ahead-of-runtime desired generation, retirement or invalid authorization. Conditions, metrics and status APIs can expose the explanation alongside the traffic decision.
The opposite mistake is making one component's readiness depend on everything downstream. A control component need not become unready because a customer target returns HTTP 500; restarting the control process will not heal that target. A component responsible for delivery may make a different decision. Each layer should derive readiness from the facts it owns.
Startup ordering is not semantic readiness
Init containers completing and dependencies accepting connections do not necessarily mean a workload is ready for its role. An application may still need to issue and verify identity, validate binding, receive desired state, and apply the expected generation. Making those steps explicit avoids sending work merely because a process opened a port.
Make restart behavior testable
A valid ordinary restart should recover the existing identity and state. Re-enrollment is a separate, authorized lifecycle action, not a silent fallback. If recovery state is corrupt or a credential was revoked, a healthy process must not silently regain authority.
Useful restart assertions include identity continuity, no generation rollback, no resurrection of retired state, no leader-only action by a follower, and a bounded reason while convergence is incomplete.
An illustrative scenario
A queue worker starts, returns 200 from its health endpoint and passes readiness. The Service sends an event, but the worker discovers its session credential expired during startup. If the only signal is a green Pod, the instinct may be to restart it. With explicit application state, the worker can report the expired credential and leave its traffic-eligible state; the operator sees that re-issuance, not a restart, is needed.
Observation and traffic eligibility can remain separate: a health endpoint can expose process status, readiness can govern Service traffic, and richer admission and convergence state can be available through status or metrics.
A practical checklist
- Does liveness answer process questions?
- Does readiness derive from facts this component owns?
- Are admission and convergence observable with useful reasons?
- Is restart recovery distinguishable from re-enrollment?
- Does a downstream failure affect only the layers that own it?
- Can an operator distinguish a stopped process from a healthy follower or an unadmitted workload?
Running and Ready have precise meanings. Define admission and authority for your application, expose them as first-class state, and let Kubernetes probes reflect that model.
Top comments (5)
Great breakdown of the gap between “process alive” and “system-ready.” One clarification that maybe help others: in Kubernetes, “admission” specifically means API-server admission control at object creation time. It checks whether the Pod spec may exist, not whether the running app can join a business process. Maybe distinguish “API admission” vs “application admission” explicitly from the start, so readers don’t think a ValidatingWebhook can inspect running Pod state. Still, the core point, that you need an application-owned state beyond "Ready" is spot on.
thanks for the link, another very relevant article
A Running Pod doesn't mean a healthy or authorized workload.
In Kubernetes, being operational is more than passing health checks. Authorization, readiness, fencing, and state consistency matter just as much.
Don't just ask whether your Pod is running. Ask whether it's safe and authorized to do its job.
That's the difference between a deployment that works and a system you can trust.
Exactly—that distinction between “the process is alive” and “it can safely do its job” is what motivated the article.
Fencing is a great example: a replica can be healthy and authenticated while no longer having authority to perform a leader-only action. We explored that distinction in an earlier article: A Kubernetes Lease Is Not a Fencing Token.
I’d be interested in your thoughts if you get a chance to take a look!
Absolutely, I'll take a look! The distinction between coordination and actual fencing authority is especially important in distributed systems. A Lease can indicate ownership, but without enforcement at the resource level, it doesn't guarantee safety. Looking forward to reading it!