Kubernetes is one of the best pieces of infrastructure engineering we use.
It can schedule workloads, reconcile desired state, expose health probes, elect a leader, distribute configuration and Secrets, apply network policy, and give controllers a powerful declarative model.
But while building a distributed delivery system, we kept finding the same class of bug:
A Kubernetes primitive was being treated as if it guaranteed more than it actually did.
That isn't a Kubernetes defect.
In most cases, Kubernetes is deliberately giving us a lower-level mechanism and leaving application semantics to the application.
The difference starts to matter when your system needs to answer questions like:
- Which replica is authorized to act, not merely elected?
- Does returning HTTP 200 mean an event will survive a process crash?
- Is a running Pod actually admitted into the application trust model?
- Does mounting a Secret prove that the workload has the right identity?
- Can replay protection survive the same request landing on another replica?
- Who owns a field when desired, observed, and applied state disagree?
- Can we prove that the container running is the exact artifact we qualified?
Those aren't scheduling questions.
They're application correctness questions.
The examples below came out of building and qualifying Zen Mesh, but the boundaries are general distributed-systems problems.
Here are ten guarantees we had to make explicit.
1. Garbage collection: deletion is easy; policy is harder
Kubernetes already has several excellent cleanup mechanisms.
Owner references let child resources disappear with their owner. Jobs can use ttlSecondsAfterFinished. CronJobs have history limits. Finalizers let controllers delay deletion until cleanup is complete.
For many systems, those primitives are exactly enough.
Our problem appeared when cleanup crossed resource classes and safety boundaries.
A qualification Job may be disposable after a short TTL. A generated ConfigMap may have a different lifecycle. An abandoned temporary environment may be best handled by destroying the entire environment.
But evidence, dead-letter records, or customer-owned state should never enter a generic cleanup policy merely because they're old.
That changes the question from:
How do I delete Kubernetes objects?
to:
Which classes may be collected, under what policy, with which protections, and who is allowed to decide?
The rule we ended up with is conservative:
Use native Kubernetes cleanup whenever it expresses the lifecycle correctly. Add a higher-level policy only when the lifecycle is genuinely cross-resource or policy-driven.
2. Leader election: holding the Lease is not the entire authority model
Kubernetes Lease objects and controller-runtime leader election solve an important problem:
Which participant currently holds leadership?
But distributed systems often need to answer a harder question:
How does every authoritative operation prove that an old leader is no longer allowed to act?
Imagine two replicas, A and B.
A is leader at generation 12.
A loses leadership.
B becomes leader at generation 13.
But A remains alive long enough to execute one delayed reconciliation.
If our application only asks:
Was A once a valid authenticated replica?
the answer is still yes.
That isn't sufficient.
What we need is something closer to:
A / leadership generation 12 -> DENY
B / leadership generation 13 -> ACCEPT
If A later becomes leader again, it should do so under a newer authority generation, not by reviving stale authority.
That forced us to separate concepts that are easy to collapse:
- workload authentication;
- leadership selection;
- leadership generation;
- session generation;
- credential generation;
- application authority.
A follower can have a perfectly valid identity and still be forbidden from performing leader-only operations.
Kubernetes solves leader selection. The application still has to solve stale-authority rejection where that matters.
3. Secrets: distribution is not custody
Kubernetes Secrets are useful. We use them where they're appropriate.
The dangerous leap is treating:
the Secret exists and is mounted
as a complete security architecture.
A Secret answers questions about storing and distributing sensitive bytes.
It does not, by itself, answer:
- Should this private key ever leave a custody boundary?
- Is possession of the key equivalent to workload identity?
- Who is authorized to use the key?
- What happens during rotation?
- What happens after local state loss?
- Can a stale workload continue using old authority?
- Can we prove which identity performed an operation?
Earlier versions of our architecture placed more semantic weight on possession of shared secret material.
Over time we separated those responsibilities.
Key custody became one concern.
Workload identity became another.
Authorization became another.
Enrollment and recovery became protocols of their own.
The useful lesson is not:
Kubernetes Secrets are insecure.
It's:
Secret distribution, key custody, workload identity, and authorization are different security problems.
Collapsing them makes rotation, recovery, and incident analysis much harder to reason about.
4. Pod readiness: alive is not the same as admitted
Kubernetes probes are excellent process- and service-health primitives.
They are not automatically application admission proofs.
We saw this clearly in our own runtime.
A process could be up.
Its container could be running.
Its health endpoint could answer.
And the workload could still be unable—or unauthorized—to participate in the system.
Application readiness depended on facts such as:
- identity being valid;
- enrollment or admission being complete;
- tenant and runtime binding being correct;
- credentials being current;
- heartbeat state being fresh;
- desired state being received;
- observed and applied generations being compatible;
- required persistence being available.
A useful mental model became:
process alive
!= service reachable
!= workload authenticated
!= workload admitted
!= workload authoritative
!= workload ready for customer traffic
Kubernetes probes remain important.
We simply stopped asking one boolean to represent all of those states.
5. Replicas: high availability can expose correctness bugs
Adding a second replica often improves availability.
It can also expose assumptions that were invisible with one process.
Replay protection is a good example.
Suppose an ingress request contains a valid signature and nonce.
Replica A accepts it and records the nonce in process-local state.
The same signed request is then replayed to replica B.
What prevents B from accepting it?
Nothing—unless the replay state is shared, replicated, partitioned using a safe ownership model, or the routing guarantee itself makes cross-replica replay impossible.
We found this class of defect during pre-production qualification of our Traffic runtime.
The missing invariant was application-level:
Once a nonce has been successfully consumed, no live replica may accept it again inside its replay window.
The same issue appears with:
- idempotency;
- retries;
- deduplication;
- durable acknowledgements.
A Deployment with multiple replicas gives you multiple processes.
It does not automatically give you one correctness authority.
6. HTTP 200: acknowledgement is a contract, not a status code
Consider this implementation:
receive event
-> place event in memory
-> return HTTP 200
-> process dies
-> event disappears
Kubernetes may restart that process perfectly.
The event is still gone.
The infrastructure behaved correctly.
The application contract didn't.
So we had to define what successful acknowledgement actually means.
For a delivery system, a useful invariant is:
If we return success, the event has crossed the declared durable acceptance boundary.
The exact durability class can vary by topology.
A single-node runtime can reasonably have a narrower failure model than a replicated system.
But the response must not imply durability the architecture does not actually provide.
This distinction sounds obvious until you start injecting crashes at inconvenient moments.
Infrastructure recovery and application correctness are related.
They are not interchangeable.
7. Desired state: reconciliation needs explicit ownership
Kubernetes made declarative desired state mainstream.
That's one of its most important contributions.
But real distributed systems frequently have more than one meaningful state:
desired state
what the authority wants
observed state
what the remote runtime reports
last-applied state
what an authorized controller knows it applied
Without explicit ownership, those fields quickly blur together.
Who owns generation 8?
Can the runtime advance it?
Can an old controller reconnect and write generation 7?
What does "drift" mean if desired, observed, and applied state live in different places?
We ended up using explicit generations, digests, connection state, and bounded reason codes to classify convergence.
The rule is simple:
Every authoritative field should have one owner.
Kubernetes reconciliation is a mechanism.
Application state authority is the contract layered on top.
8. NetworkPolicy: reachability is not identity
NetworkPolicy is extremely useful for controlling who can talk to whom.
But packet reachability does not tell the application who the caller is.
A stronger internal boundary can require several independent answers:
NetworkPolicy:
Is this network path allowed?
Workload identity / mTLS:
Which cryptographic principal is calling?
Authorization:
Is that principal allowed to perform this operation now?
Those controls complement each other.
They don't replace each other.
This becomes especially important during failover and rotation.
A network path can remain valid while a credential becomes stale.
A workload can possess a valid identity while being a follower.
A component can be reachable while being unauthorized for a tenant or operation.
Again, Kubernetes is doing its job.
The application supplies the semantic layer.
9. Container tags: deployability is not provenance
Kubernetes will happily deploy:
service:candidate
That's convenient during development.
It's weaker as qualification evidence.
Suppose we qualify service:candidate.
Later the same tag points to different bytes.
The label hasn't changed.
The artifact has.
For release qualification, the chain we actually care about looks more like:
source revision
->
built image
->
registry digest
->
deployment references digest
->
runtime image ID
The runtime should be able to prove that it is executing the artifact we actually qualified.
This is also why shortcuts such as preloading an image directly into cluster nodes can produce misleading release tests.
The system may appear healthy while bypassing the same registry and provenance path production will use.
Tags are excellent names. Digests are better evidence.
10. Some problems simply aren't Kubernetes problems
There's a trap at the other extreme.
Once you start identifying places where Kubernetes primitives are narrower than application guarantees, it becomes tempting to call every application problem a Kubernetes gap.
That would be wrong.
Kubernetes should not be expected to understand:
- our tenant authorization model;
- what a product revision means;
- whether a customer event is durably accepted;
- a provider-specific authentication contract;
- what evidence a customer needs to verify an outcome.
Those belong in product architecture.
The goal isn't to replace Kubernetes.
The goal is to know precisely where its responsibility ends.
The four questions we now ask
When we encounter an infrastructure primitive, we ask:
- What does this primitive actually guarantee?
- What stronger guarantee does our product require?
- Which component owns that stronger guarantee?
- How do we test the failure case, not just the happy path?
That discipline has influenced how we think about:
- garbage collection;
- leadership fencing;
- key custody;
- workload admission;
- replay protection;
- durable event acceptance;
- state reconciliation;
- artifact qualification.
Kubernetes remains the substrate underneath all of it.
The lesson is not that Kubernetes falls short.
The lesson is that a production system becomes easier to reason about when it stops asking infrastructure primitives to carry application semantics they were never designed to own.
This is the first article in **Beyond Kubernetes Defaults, a technical series based on engineering lessons from building Zen Mesh. The next article looks at a deceptively simple question: Kubernetes already has garbage collection—so when does another garbage collector actually make sense?
The canonical version of this article is published on Zen Mesh.
Top comments (0)