DEV Community: Zen Mesh Inc.

A Running Pod Is Not an Admitted Workload

Zen Mesh Inc. — Thu, 12 Nov 2026 00:19:51 +0000

This is article 5 of 10 in Beyond Kubernetes Defaults. Read article 4: Kubernetes Secrets Are Distribution, Not Custody first, or start with article 1.

Kubernetes gives us useful workload states: scheduled, Running, Ready, unhealthy, terminating. They are operationally important and precisely defined, but narrower than many application state machines.

The misleading green Pod

Consider an illustrative workload that is Running but unable to participate safely in its application. The process exists, the container started, and a health endpoint answers. Yet participation may still depend on enrollment, current workload identity, correct tenant and placement binding, available credentials, remote admission, desired state, current sessions and durable recovery.

The process is healthy. The workload is not yet admitted.

Five different questions

Liveness: Is the process functioning well enough that a restart is not warranted?
Kubernetes readiness: Should this Pod be eligible for Service traffic?
Application admission: Has the workload joined its system correctly?
Application authority: May it perform this operation now?
Convergence: Has it applied the state and generation expected of it?

Those answers can legitimately differ. A follower can be healthy and authenticated but not authoritative. An agent disconnected from management can be alive but unable to receive work. An expired credential should fail closed even if its process is reachable.

What the probes actually measure

The Kubernetes probe documentation describes three kinds:

A startup probe gates liveness and readiness until it first succeeds.
A failing liveness probe triggers a container restart, subject to the Pod's restart policy.
A failing readiness probe makes the Pod ineligible for normal Service traffic; it does not restart the container. Exact routing depends on how traffic reaches the Pod, and direct Pod access may behave differently.

None of these alone establishes enrollment, credential validity or the expected application generation. But Kubernetes can express additional state: custom Pod conditions and readiness gates can participate in readiness, and status conditions can carry structured information. API admission controllers and Pod Security Admission answer a different question: whether a Pod specification is permitted, not whether its running application has joined its system.

Define application readiness first. Then decide how probes, conditions and status should reflect it.

One boolean often loses the reason

A generic Ready/not Ready result can hide the cause. Bounded reason codes can distinguish a stale heartbeat, expired credential, missing recovery state, an ahead-of-runtime desired generation, retirement or invalid authorization. Conditions, metrics and status APIs can expose the explanation alongside the traffic decision.

The opposite mistake is making one component's readiness depend on everything downstream. A control component need not become unready because a customer target returns HTTP 500; restarting the control process will not heal that target. A component responsible for delivery may make a different decision. Each layer should derive readiness from the facts it owns.

Startup ordering is not semantic readiness

Init containers completing and dependencies accepting connections do not necessarily mean a workload is ready for its role. An application may still need to issue and verify identity, validate binding, receive desired state, and apply the expected generation. Making those steps explicit avoids sending work merely because a process opened a port.

Make restart behavior testable

A valid ordinary restart should recover the existing identity and state. Re-enrollment is a separate, authorized lifecycle action, not a silent fallback. If recovery state is corrupt or a credential was revoked, a healthy process must not silently regain authority.

Useful restart assertions include identity continuity, no generation rollback, no resurrection of retired state, no leader-only action by a follower, and a bounded reason while convergence is incomplete.

An illustrative scenario

A queue worker starts, returns 200 from its health endpoint and passes readiness. The Service sends an event, but the worker discovers its session credential expired during startup. If the only signal is a green Pod, the instinct may be to restart it. With explicit application state, the worker can report the expired credential and leave its traffic-eligible state; the operator sees that re-issuance, not a restart, is needed.

Observation and traffic eligibility can remain separate: a health endpoint can expose process status, readiness can govern Service traffic, and richer admission and convergence state can be available through status or metrics.

A practical checklist

Does liveness answer process questions?
Does readiness derive from facts this component owns?
Are admission and convergence observable with useful reasons?
Is restart recovery distinguishable from re-enrollment?
Does a downstream failure affect only the layers that own it?
Can an operator distinguish a stopped process from a healthy follower or an unadmitted workload?

Running and Ready have precise meanings. Define admission and authority for your application, expose them as first-class state, and let Kubernetes probes reflect that model.

Kubernetes Secrets Are Distribution, Not Custody

Zen Mesh Inc. — Thu, 05 Nov 2026 00:50:22 +0000

This is article 4 of 10 in Beyond Kubernetes Defaults. Read article 3: A Kubernetes Lease Is Not a Fencing Token first, or start with article 1: When Kubernetes Primitives Aren’t Enough. Later articles cover workload admission and more.

Kubernetes Secrets are not the villain of this article.

They are a useful API for delivering sensitive configuration to workloads, and they can be protected with encryption at rest, RBAC, admission controls and careful operational practice.

The mistake is asking a Secret to answer security questions that are outside its job.

The overloaded-secret problem

A system often starts with a simple pattern:

create credential
-> store credential in Secret
-> mount Secret into Pod
-> possession means caller is trusted

That can work.

Over time, however, "the Secret" can begin representing several different concepts simultaneously:

identity;
authorization;
enrollment state;
key custody;
tenant context;
rotation generation;
recovery state.

When those concepts collapse into possession of one blob, failures become difficult to classify.

Does a lost Secret mean identity loss? Does rotation require re-enrollment? Can a copied Secret impersonate another replica? Is the key allowed to leave the component? Which authority can issue a replacement?

Distribution is not custody

Custody is a stronger statement than storage.

A custody boundary can be designed so that a private key is never exported to the caller at all. The caller requests an operation, proves its own identity and authorization, and receives a result—not the underlying private key.

That changes the compromise surface.

Instead of every workload filesystem or mounted Secret becoming a copy of the durable identity key, the component can hold only the material it needs to authenticate to the custody service.

The exact design depends on the threat model, but the conceptual separation is valuable:

Secret distribution
    != private-key custody
    != workload identity
    != authorization

What our own enrollment contract says

To make this concrete: our enrollment security contract for production workloads is a candidate document, explicitly marked as not yet qualified. It states the custody law we are building toward, and it is deliberately honest about its own status—we are not claiming that every live workload already conforms to it today.

What the contract requires for enrolled production workloads is unambiguous, though:

credentials for enrolled workloads are held in authorized, encrypted, CRD-backed custody, not Kubernetes Secret backends;
enrollment signing uses a non-exporting custody interface—the signing key is an architectural requirement, not an exportable file with a policy suggestion attached;
the forbidden list is explicit: Secret volumes, secretKeyRef/envFrom injection for enrolled workload credentials, legacy Secret sync, and local-key fallback are all out, with no selectable weaker adapter;
private key material must not be serialized into enrollment transport or left in durable plaintext copies, and must stay out of logs, receipts, environment variables and error strings.

The non-exportable part is worth stressing as a design requirement: it is a capability the custody architecture is required to provide, not a property we claim every current deployment already has. Some of the migration work is still open, and the candidate status exists precisely so that gap is stated rather than papered over.

Identity should survive operational changes cleanly

Separation also buys clean recovery semantics.

A Pod restart, Pod replacement, local state loss and full re-enrollment are different events.

If the architecture treats them all as "recreate the Secret," it becomes easy to accidentally turn routine recovery into identity replacement—or to restore stale authority without noticing.

We prefer explicit recovery states and generation checks so a corrupted checkpoint is not silently treated as a fresh install.

That is especially important in replicated systems where each replica should hold an independently issued key rather than copying another replica’s private material.

Authorization is separate again

Even a perfectly protected key does not answer:

Is this workload allowed to perform this operation now?

A valid workload identity can be a follower rather than a leader. It can belong to the correct service but wrong tenant scope. It can be valid cryptographically but stale for the current generation.

This is why our architecture keeps custody, workload identity and authorization as distinct boundaries.

Kubernetes Secrets still have a role

Kubernetes Secrets remain a reasonable choice for plenty of platform and application configuration where distribution is the actual job.

But they are not the governed credential path for enrolled production workloads. In our architecture, that role belongs to the custody boundary described above: credentials are issued into encrypted CRD-backed custody, consumed through authorized projection, and never exported to the workload as a copyable Secret.

The rule is not "never use Secrets."

The rule is:

Do not let the convenience of Secret distribution decide the rest of your security model by accident.

If a private key must never leave custody, design a custody boundary. If a workload needs a cryptographic identity, give it an identity protocol. If an action needs authorization, verify authorization explicitly.

Kubernetes Secrets can participate in that design without being forced to become all of it.

A useful threat-model question: what can the compromised workload extract?

Suppose an attacker compromises the application container.

If the durable identity private key is mounted into that container, the attacker may be able to copy it and continue using it elsewhere until revocation or expiration.

If the workload instead talks to a narrow custody interface, the attacker may still misuse the workload while the compromise is active, but extracting the long-lived key is designed to be substantially harder—or impossible, because the key never exists where the attacker is standing.

That does not eliminate the need for authorization, rotation or incident response. It changes the blast radius of credential theft.

Rotation becomes clearer when custody is separate

A rotation protocol can then answer explicit questions:

Which identity is rotating?
Which generation is current?
What proves possession/authority?
What state is persisted for restart?
Can an old generation still sign?
What happens if the process crashes halfway through activation?

Those are application/security protocol questions. A Secret update alone cannot define all of them.

We found it useful to model activation as a crash-consistent transition rather than "write new files and hope startup sorts it out." A restart should either recover one coherent generation or fail closed.

Channel credentials and identity keys can have different treatment

Not every secret deserves the same custody mechanism.

A short-lived channel credential used to authenticate to a local security service may reasonably be distributed differently from the durable private key representing workload identity.

This is another reason not to make "Secret or no Secret" the architecture question.

Classify the secret by purpose, lifetime and consequence of extraction.

When Kubernetes Secrets are enough

For many applications, Kubernetes Secrets plus good cluster security are entirely appropriate.

A dedicated custody service becomes more compelling when:

private keys should never be exportable to application workloads;
several components need controlled cryptographic operations;
rotation/recovery needs strong generation semantics;
audit/evidence of key usage matters;
the blast radius of copied key material is unacceptable.

Additional machinery should follow the threat model, not fashion.

Originally published at zen-mesh.io.

A Kubernetes Lease Is Not a Fencing Token

Zen Mesh Inc. — Thu, 29 Oct 2026 01:04:47 +0000

This is article 3 of 10 in Beyond Kubernetes Defaults. Read article 2: Kubernetes Has Garbage Collection. Why Did We Build Another One? first, or start with article 1: When Kubernetes Primitives Aren't Enough. Later articles cover custody, admission and more.

Kubernetes makes leader election straightforward.

With controller-runtime or the Lease API, replicas compete for a Lease and one becomes leader. When that leader stops renewing, another replica can take over.

For many controllers, that is sufficient.

For systems where leader-only actions mutate durable authority, we found a second question matters just as much:

How do we prove that the old leader can no longer act?

That is a fencing problem.

Election and fencing are different properties

Suppose replica A holds leadership.

A = leader
B = follower

A pauses during a network or runtime failure. B eventually acquires the Lease.

A = stale former leader
B = current leader

The Kubernetes control-plane state is correct: B owns the Lease.

But what if A wakes up and executes a delayed write that was prepared while it was leader?

Whether that is possible depends on the application protocol, not the fact that Kubernetes elected B.

If a downstream authority accepts the operation solely because A has a valid workload credential, election has not fenced the stale action.

Authentication is not authority

One of the most useful distinctions in our HA work was:

authenticated replica != authoritative leader

A follower can have a valid identity.

It should be able to maintain health, rotate credentials, prepare for takeover and perform explicitly non-authoritative operations.

It should not be able to commit leader-owned reconciliation merely because its certificate is valid.

This matters because identity lifetimes and leadership lifetimes are usually different.

Generations make stale authority explicit

A simple conceptual model is a monotonic leadership generation or authority epoch.

A becomes leader at generation 12
A loses authority
B becomes leader at generation 13

Now a leader-only operation can carry authority state that a verifier can compare.

A / generation 12 -> DENY
B / generation 13 -> ACCEPT

If A later reacquires leadership, it does so under generation 14 rather than reviving generation 12.

That turns stale authority into something mechanically testable.

Do not collapse every generation into one counter

Distributed systems accumulate versions quickly:

credential generation;
leadership generation;
connection/session generation;
object revision;
reconciliation generation.

They are not interchangeable.

A newer certificate does not make a replica leader. A newer object revision does not necessarily imply a newer session. A reconnect can invalidate an old plan without rotating identity.

Keeping these meanings separate makes failure handling clearer.

What Kubernetes should own

We still want Kubernetes to own runtime leader selection.

The Lease is the correct primitive for deciding who currently holds leadership in a cluster.

The reusable application layer can then own the semantics that consume that decision:

monotonic authority generation;
stale-leader rejection;
conflicting-holder rejection;
wrong-replica/Plane rejection;
fail-closed handling of replayed authority;
tests that prove old leaders cannot mutate new state.

That logic can be generic without embedding Kubernetes Lease objects into every business operation.

Test the old leader, not only the new leader

A surprisingly common HA test is:

kill leader A;
observe B become leader;
declare failover successful.

That proves availability.

It does not necessarily prove fencing.

A stronger test also asks:

Can stale A still perform the leader-only operation?

and requires the answer to be no.

We also test adjacent cases:

same authority generation with a different holder;
lower generation;
wrong workload identity;
replayed authority state;
failback to the previous replica under a newer generation.

When you do not need this

Not every Kubernetes leader election requires a custom fencing protocol.

If all leader-only work is scoped to a Kubernetes resource update whose resourceVersion/optimistic concurrency already gives the required safety, adding another fencing layer may be unnecessary.

The need appears when an elected leader controls effects outside that single optimistic-concurrency boundary: databases, remote runtimes, external APIs, distributed reconciliation, or durable state machines.

The principle is simple:

Use Kubernetes to elect the leader. Add fencing only where stale authority can still cause a real effect.

The dangerous interval is often after failover

Teams naturally focus on election time: how quickly does B become leader after A disappears?

The more interesting correctness interval can be after B has already won.

A may still have:

a buffered reconciliation result;
an in-flight database transaction;
a queued remote command;
a cached authorization decision;
a live connection to another component.

If downstream systems cannot distinguish A's stale authority from B's current authority, the system can have a clean Kubernetes Lease and still accept stale effects.

That is why the authority token or generation needs to cross the same boundary as the effect.

Fencing should fail closed at the receiver

The strongest place to enforce stale-authority rejection is usually the component that accepts the authoritative operation.

If only the sender checks “am I still leader?”, a paused process can make that check, lose leadership, resume, and send anyway.

A receiver-side verifier can instead require current authority state and reject stale generations regardless of what the sender believes.

Conceptually:

sender says: I am generation 12
receiver knows: current generation is 13
result: DENY

This is the same reason storage systems use fencing tokens around distributed locks: the protected resource must participate in the safety decision.

Readiness should not make followers unhealthy

Another operational detail is distinguishing a follower from a broken replica.

A follower may be:

authenticated;
fully initialized;
able to renew/rotate credentials;
synchronized enough to take over quickly;
healthy from a process perspective.

It simply lacks leader-only authority.

If every follower is treated as generically unhealthy, operators lose visibility into whether standby capacity is actually ready for takeover.

We prefer explicit states such as authenticated follower versus authoritative leader, then map those states to Kubernetes readiness in a way appropriate to the traffic pattern.

Fencing and exactly-once are not the same problem

Fencing prevents stale authorities from creating new effects. It does not automatically make the effects themselves exactly once.

A leader can legitimately retry an operation after a timeout and still create a duplicate unless the operation is idempotent or deduplicated.

So a serious HA design often needs both:

authority fencing — only the current leader may act;
operation idempotency — retries of the current leader do not duplicate effects.

Keeping those properties separate makes the system easier to test.

Kubernetes Has Garbage Collection. Why Did We Build Another One?

Zen Mesh Inc. — Fri, 23 Oct 2026 00:26:34 +0000

This is article 2 of 10 in Beyond Kubernetes Defaults. Read article 1: When Kubernetes Primitives Aren’t Enough first. This article is cross-published from Zen Mesh.

Kubernetes already garbage-collects resources.

That makes “we built a Kubernetes garbage collector” sound suspiciously like reinventing a solved problem.

In many cases, it would be.

Kubernetes gives us several excellent lifecycle mechanisms: owner references, cascading deletion, Job TTLs, CronJob history limits and finalizers. If a resource has a natural owner or a single lifecycle rule, those mechanisms should usually be the first choice.

Our need appeared when cleanup stopped being a property of one resource and became policy across resource classes.

Native cleanup is good at structural lifecycle

Owner references are excellent when the relationship is structural: delete the parent and its children should disappear.

ttlSecondsAfterFinished is excellent when every completed Job can follow the same lifecycle.

CronJob history limits are excellent when the controller already owns the history set.

Finalizers are excellent when deletion must wait for an explicit cleanup action.

The key question is not “can Kubernetes delete this?” It obviously can.

The key question is “who owns the decision that this object is now disposable?”

Policy changes the problem

Consider an environment that creates several classes of temporary resources:

test Jobs;
generated ConfigMaps;
short-lived credentials or references;
transient namespaces;
evidence artifacts;
dead-letter records;
customer-owned resources.

They may all be old. They should not all be collected by the same rule.

A Job can be disposable after one hour while its evidence must remain. A generated ConfigMap can be safely deleted after a run, while a ConfigMap carrying customer configuration may be owned by another controller entirely. A dead-letter entry is old data, but age alone does not make it garbage.

This is the distinction that led to Zen Cleaner.

For a quick overview, watch the 1:17 Zen Cleaner Short; the longer explainer walks through the project in more detail.

The useful abstraction was not “delete old Kubernetes things.” It was:

Evaluate an explicit cleanup policy across allowed resource classes, while preserving protection and ownership boundaries.

The safety property matters more than the deletion feature

A garbage collector can cause more damage than the resource leak it fixes.

The first design question should therefore be the negative one:

What must this collector never delete?

Evidence, DLQ/customer data and component-owned durable state should not become generic-GC targets merely because they are old. Their owning component remains responsible for retention and deletion semantics.

That turns protection into a first-class policy rather than an operator convention.

Why not encode everything in finalizers?

Finalizers are powerful, but they solve a different problem. They let an owning controller block deletion until it has completed a required action.

They do not naturally express a global policy such as:

disposable Jobs: collect after 2h
transient generated ConfigMaps: collect after 6h
resources with a protection label: preserve
resources owned by customer-data controllers: out of scope

You can build a controller that interprets those policies using Kubernetes APIs—and that is effectively what a higher-level GC controller becomes.

The point is not that Kubernetes lacks extensibility. The point is that the policy is application/operator logic implemented on top of Kubernetes primitives.

Why not a CronJob with kubectl?

A shell script is tempting:

list old resources → inspect timestamps → kubectl delete

It is also difficult to make safe over time.

The moment policy requires selectors, protected classes, per-kind TTLs, retries, reason codes, rate limiting or dry-run evidence, the script is becoming an undocumented controller. At that point a controller with an explicit policy API is easier to test and reason about.

The rule we use

Use Kubernetes-native lifecycle whenever it is sufficient.

Add a policy controller only when at least one of these becomes true:

lifecycle spans unrelated resource kinds;
different classes require different TTLs;
some resources require explicit protection;
ownership must be checked before deletion;
deletion decisions need durable/auditable reason codes;
cleanup must be rate-limited or retried predictably.

If none of those are true, a custom GC controller is probably unnecessary.

Protection semantics are the real feature

A generic collector needs a way to distinguish “eligible for cleanup” from “safe to delete.” Those are not equivalent.

Age can be one input without being the authority. A resource can be older than its nominal TTL and still be protected because it represents evidence, an active investigation, a recovery checkpoint or state owned by another controller.

A robust collector needs an explicit protection model. Depending on the environment, that might use labels, annotations, policy objects, owner classifications or another bounded mechanism. These are design choices to verify in a particular controller, not assumptions to make from its being a Kubernetes controller.

The negative cases are as important as the positive ones. A policy should preserve protected resources, ignore unknown classes and avoid broadening deletion when policy is malformed. Operational controls such as concurrency, retries, rate limits, idempotency and outcome reporting should be verified for the controller being evaluated rather than assumed.

A useful ownership test

Before adding a resource class to generic GC, ask:

Who created it?
Who understands its semantic lifecycle?
Can generic age/policy safely decide it is disposable?
Does deletion destroy evidence, recovery state or customer data?
Is there already a better native Kubernetes lifecycle mechanism?

If the owning component understands the lifecycle better than a generic collector, the generic collector should stay out.

That restraint is one of the most important properties of a GC system.

Why this matters beyond Zen

The same pattern shows up in CI clusters, preview environments, ephemeral test infrastructure and internal platforms. Teams often begin with ad hoc cleanup scripts and eventually discover that the difficult part is not deletion—it is policy ownership.

Kubernetes provides the deletion machinery. A dedicated controller is justified only when the organization needs a reusable, explicit answer to “what is garbage?”

The next article in this series turns to leader election: holding a Kubernetes Lease identifies the current leader, but applications still need to decide how to reject stale authority.

When Kubernetes Primitives Aren’t Enough: 10 Production Guarantees We Had to Build Above K8s

Zen Mesh Inc. — Mon, 14 Sep 2026 22:29:00 +0000

This is article 1 of 10 in Beyond Kubernetes Defaults. DEV’s series box shows the articles published so far.

Kubernetes is one of the best pieces of infrastructure engineering we use.

It can schedule workloads, reconcile desired state, expose health probes, elect a leader, distribute configuration and Secrets, apply network policy, and give controllers a powerful declarative model.

But while building a distributed delivery system, we kept finding the same class of bug:

A Kubernetes primitive was being treated as if it guaranteed more than it actually did.

That isn't a Kubernetes defect.

In most cases, Kubernetes is deliberately giving us a lower-level mechanism and leaving application semantics to the application.

The difference starts to matter when your system needs to answer questions like:

Which replica is authorized to act, not merely elected?
Does returning HTTP 200 mean an event will survive a process crash?
Is a running Pod actually admitted into the application trust model?
Does mounting a Secret prove that the workload has the right identity?
Can replay protection survive the same request landing on another replica?
Who owns a field when desired, observed, and applied state disagree?
Can we prove that the container running is the exact artifact we qualified?

Those aren't scheduling questions.

They're application correctness questions.

The examples below came out of building and qualifying Zen Mesh, but the boundaries are general distributed-systems problems.

Here are ten guarantees we had to make explicit.

1. Garbage collection: deletion is easy; policy is harder

Kubernetes already has several excellent cleanup mechanisms.

Owner references let child resources disappear with their owner. Jobs can use ttlSecondsAfterFinished. CronJobs have history limits. Finalizers let controllers delay deletion until cleanup is complete.

For many systems, those primitives are exactly enough.

Our problem appeared when cleanup crossed resource classes and safety boundaries.

A qualification Job may be disposable after a short TTL. A generated ConfigMap may have a different lifecycle. An abandoned temporary environment may be best handled by destroying the entire environment.

But evidence, dead-letter records, or customer-owned state should never enter a generic cleanup policy merely because they're old.

That changes the question from:

How do I delete Kubernetes objects?

to:

Which classes may be collected, under what policy, with which protections, and who is allowed to decide?

The rule we ended up with is conservative:

Use native Kubernetes cleanup whenever it expresses the lifecycle correctly. Add a higher-level policy only when the lifecycle is genuinely cross-resource or policy-driven.

2. Leader election: holding the Lease is not the entire authority model

Kubernetes Lease objects and controller-runtime leader election solve an important problem:

Which participant currently holds leadership?

But distributed systems often need to answer a harder question:

How does every authoritative operation prove that an old leader is no longer allowed to act?

Imagine two replicas, A and B.

A is leader at generation 12.

A loses leadership.

B becomes leader at generation 13.

But A remains alive long enough to execute one delayed reconciliation.

If our application only asks:

Was A once a valid authenticated replica?

the answer is still yes.

That isn't sufficient.

What we need is something closer to:

A / leadership generation 12 -> DENY
B / leadership generation 13 -> ACCEPT

If A later becomes leader again, it should do so under a newer authority generation, not by reviving stale authority.

That forced us to separate concepts that are easy to collapse:

workload authentication;
leadership selection;
leadership generation;
session generation;
credential generation;
application authority.

A follower can have a perfectly valid identity and still be forbidden from performing leader-only operations.

Kubernetes solves leader selection. The application still has to solve stale-authority rejection where that matters.

3. Secrets: distribution is not custody

Kubernetes Secrets are useful. We use them where they're appropriate.

The dangerous leap is treating:

the Secret exists and is mounted

as a complete security architecture.

A Secret answers questions about storing and distributing sensitive bytes.

It does not, by itself, answer:

Should this private key ever leave a custody boundary?
Is possession of the key equivalent to workload identity?
Who is authorized to use the key?
What happens during rotation?
What happens after local state loss?
Can a stale workload continue using old authority?
Can we prove which identity performed an operation?

Earlier versions of our architecture placed more semantic weight on possession of shared secret material.

Over time we separated those responsibilities.

Key custody became one concern.

Workload identity became another.

Authorization became another.

Enrollment and recovery became protocols of their own.

The useful lesson is not:

Kubernetes Secrets are insecure.

It's:

Secret distribution, key custody, workload identity, and authorization are different security problems.

Collapsing them makes rotation, recovery, and incident analysis much harder to reason about.

4. Pod readiness: alive is not the same as admitted

Kubernetes probes are excellent process- and service-health primitives.

They are not automatically application admission proofs.

We saw this clearly in our own runtime.

A process could be up.

Its container could be running.

Its health endpoint could answer.

And the workload could still be unable—or unauthorized—to participate in the system.

Application readiness depended on facts such as:

identity being valid;
enrollment or admission being complete;
tenant and runtime binding being correct;
credentials being current;
heartbeat state being fresh;
desired state being received;
observed and applied generations being compatible;
required persistence being available.

A useful mental model became:

process alive
    != service reachable
    != workload authenticated
    != workload admitted
    != workload authoritative
    != workload ready for customer traffic

Kubernetes probes remain important.

We simply stopped asking one boolean to represent all of those states.

5. Replicas: high availability can expose correctness bugs

Adding a second replica often improves availability.

It can also expose assumptions that were invisible with one process.

Replay protection is a good example.

Suppose an ingress request contains a valid signature and nonce.

Replica A accepts it and records the nonce in process-local state.

The same signed request is then replayed to replica B.

What prevents B from accepting it?

Nothing—unless the replay state is shared, replicated, partitioned using a safe ownership model, or the routing guarantee itself makes cross-replica replay impossible.

We found this class of defect during pre-production qualification of our Traffic runtime.

The missing invariant was application-level:

Once a nonce has been successfully consumed, no live replica may accept it again inside its replay window.

The same issue appears with:

idempotency;
retries;
deduplication;
durable acknowledgements.

A Deployment with multiple replicas gives you multiple processes.

It does not automatically give you one correctness authority.

6. HTTP 200: acknowledgement is a contract, not a status code

Consider this implementation:

receive event
-> place event in memory
-> return HTTP 200
-> process dies
-> event disappears

Kubernetes may restart that process perfectly.

The event is still gone.

The infrastructure behaved correctly.

The application contract didn't.

So we had to define what successful acknowledgement actually means.

For a delivery system, a useful invariant is:

If we return success, the event has crossed the declared durable acceptance boundary.

The exact durability class can vary by topology.

A single-node runtime can reasonably have a narrower failure model than a replicated system.

But the response must not imply durability the architecture does not actually provide.

This distinction sounds obvious until you start injecting crashes at inconvenient moments.

Infrastructure recovery and application correctness are related.

They are not interchangeable.

7. Desired state: reconciliation needs explicit ownership

Kubernetes made declarative desired state mainstream.

That's one of its most important contributions.

But real distributed systems frequently have more than one meaningful state:

desired state
    what the authority wants

observed state
    what the remote runtime reports

last-applied state
    what an authorized controller knows it applied

Without explicit ownership, those fields quickly blur together.

Who owns generation 8?

Can the runtime advance it?

Can an old controller reconnect and write generation 7?

What does "drift" mean if desired, observed, and applied state live in different places?

We ended up using explicit generations, digests, connection state, and bounded reason codes to classify convergence.

The rule is simple:

Every authoritative field should have one owner.

Kubernetes reconciliation is a mechanism.

Application state authority is the contract layered on top.

8. NetworkPolicy: reachability is not identity

NetworkPolicy is extremely useful for controlling who can talk to whom.

But packet reachability does not tell the application who the caller is.

A stronger internal boundary can require several independent answers:

NetworkPolicy:
    Is this network path allowed?

Workload identity / mTLS:
    Which cryptographic principal is calling?

Authorization:
    Is that principal allowed to perform this operation now?

Those controls complement each other.

They don't replace each other.

This becomes especially important during failover and rotation.

A network path can remain valid while a credential becomes stale.

A workload can possess a valid identity while being a follower.

A component can be reachable while being unauthorized for a tenant or operation.

Again, Kubernetes is doing its job.

The application supplies the semantic layer.

9. Container tags: deployability is not provenance

Kubernetes will happily deploy:

service:candidate

That's convenient during development.

It's weaker as qualification evidence.

Suppose we qualify service:candidate.

Later the same tag points to different bytes.

The label hasn't changed.

The artifact has.

For release qualification, the chain we actually care about looks more like:

source revision
    ->
built image
    ->
registry digest
    ->
deployment references digest
    ->
runtime image ID

The runtime should be able to prove that it is executing the artifact we actually qualified.

This is also why shortcuts such as preloading an image directly into cluster nodes can produce misleading release tests.

The system may appear healthy while bypassing the same registry and provenance path production will use.

Tags are excellent names. Digests are better evidence.

10. Some problems simply aren't Kubernetes problems

There's a trap at the other extreme.

Once you start identifying places where Kubernetes primitives are narrower than application guarantees, it becomes tempting to call every application problem a Kubernetes gap.

That would be wrong.

Kubernetes should not be expected to understand:

our tenant authorization model;
what a product revision means;
whether a customer event is durably accepted;
a provider-specific authentication contract;
what evidence a customer needs to verify an outcome.

Those belong in product architecture.

The goal isn't to replace Kubernetes.

The goal is to know precisely where its responsibility ends.

The four questions we now ask

When we encounter an infrastructure primitive, we ask:

What does this primitive actually guarantee?
What stronger guarantee does our product require?
Which component owns that stronger guarantee?
How do we test the failure case, not just the happy path?

That discipline has influenced how we think about:

garbage collection;
leadership fencing;
key custody;
workload admission;
replay protection;
durable event acceptance;
state reconciliation;
artifact qualification.

Kubernetes remains the substrate underneath all of it.

The lesson is not that Kubernetes falls short.

The lesson is that a production system becomes easier to reason about when it stops asking infrastructure primitives to carry application semantics they were never designed to own.

This is the first article in **Beyond Kubernetes Defaults, a technical series based on engineering lessons from building Zen Mesh. The next article looks at a deceptively simple question: Kubernetes already has garbage collection—so when does another garbage collector actually make sense?

The canonical version of this article is published on Zen Mesh.

Your Webhook Tool Can't Tell You What Actually Happened

Zen Mesh Inc. — Fri, 26 Jun 2026 02:23:11 +0000

You get a 200. Or you get a timeout. That's it.

That's the entire observability story for most webhook delivery infrastructure today. A status code and a timestamp. Maybe a retry count if you're lucky.

For a lot of use cases, that's fine. A notification fires, it either lands or it doesn't, you move on. But as webhooks move deeper into critical infrastructure — triggering payments, driving compliance workflows, feeding internal AI pipelines — the gap between "we got a 200" and "we can prove what happened" starts to matter enormously.

The Observability Problem Nobody Talks About

Most webhook tools give you a delivery log. It shows you attempts, status codes, response times. It tells you the system tried. What it doesn't tell you:

Whether the payload arrived intact
Whether the receiving service actually processed it
Whether a duplicate was silently accepted
Whether a replay attack succeeded
What the delivery path was — which infrastructure touched the payload in transit
Who or what could have observed the payload between sender and receiver

These aren't edge cases. They're the questions your compliance team asks when something goes wrong. They're the questions a security audit surfaces. They're the questions you can't answer with a delivery log.

The problem isn't that webhook tools are lazy. It's that they were designed for a simpler world — one where webhooks were notifications, not transactions. That world is going away.

What "Proof" Actually Means in Delivery Infrastructure

In distributed systems, proof of delivery has a specific meaning. It's not a log entry. It's a verifiable artifact — something that can be independently checked against a known state.

The difference matters when:

Disputes arise. "We sent it" and "we received it" are two different claims. Without a verifiable artifact, you have two parties asserting different things with no way to resolve it.

Audits happen. A compliance auditor doesn't want to read your delivery logs. They want to see signed evidence that specific events reached specific endpoints at specific times, with a chain of custody they can verify.

Replay attacks occur. A webhook sent twice should be handled once. Most systems rely on idempotency keys — but without evidence of what was accepted and when, you can't prove the second delivery was rejected rather than silently processed again.

AI pipelines consume events. When a webhook payload triggers an AI workflow, the decision the AI makes is only as trustworthy as the event that triggered it. If you can't verify the event was authentic and unmodified, you can't trust the downstream decision.

What Delivery Evidence Actually Looks Like

Real delivery evidence isn't a log. It's a structured artifact produced at delivery time that captures:

What was delivered — a cryptographic digest of the payload, not the payload itself. You can verify integrity without re-exposing the content.

Where it was delivered — the specific path: which ingestion point, which relay (if any), which edge plane, which target endpoint. Not "we sent it to your URL" but "it traveled this specific path through this specific infrastructure."

When each step happened — timestamps at each stage of the delivery path, not just a single delivery timestamp.

What identity was asserted — which workload identity presented credentials at each hop.

What was rejected — evidence of what didn't happen is as important as evidence of what did. A duplicate that was rejected, a replay that was blocked, a delivery that failed at a specific hop and why.

All of this assembled into a tamper-evident chain. Change any part of it and the chain breaks. That's the difference between a log and proof.

The Compliance Angle Is Becoming Unavoidable

A few years ago, webhook compliance was an afterthought. Today:

PCI-DSS requires evidence of data handling for payment-related events. Stripe webhooks carrying payment intent data are in scope.

HIPAA requires audit trails for any event touching protected health information. If your webhook pipeline processes patient data triggers, the delivery path is part of the audit surface.

SOC 2 auditors increasingly ask about event integrity controls. "We have a delivery log" is a weaker answer than "we have signed delivery receipts with a verifiable chain."

Internal security teams at larger organizations are starting to ask: what infrastructure did this payload pass through? Who could have observed it? Can you prove it arrived unmodified?

These questions are coming whether you're ready for them or not.

The Path Trust Problem

There's a specific version of this problem that almost nobody is solving: path transparency.

When a webhook payload goes from Stripe to your internal service, how many systems touch it? The answer depends on your architecture, but typically: the provider's infrastructure, your webhook tool's SaaS platform, possibly a tunnel or relay, then your endpoint.

Most delivery logs tell you the start and end. They don't tell you what happened in between.

Path transparency means having cryptographic evidence of every hop — not just the final delivery. If a relay was used, there's evidence of the relay. If the data plane was involved, there's evidence of which data plane. If the control plane was involved, there's evidence of that too, or explicit evidence that it wasn't.

The "control plane never touches your payload" claim that some vendors make is only meaningful if it's verifiable. Otherwise it's a promise, not a property.

Merkle Integrity in Delivery Evidence

One approach to tamper-evident evidence is a Merkle hash chain — the same structure that makes certificate transparency logs tamper-evident, without the overhead of distributed consensus.

Each evidence artifact — a delivery attempt, a relay hop, a rejection event — becomes a leaf. The chain is built incrementally. The root hash changes if any artifact is modified. Anyone with the root hash and an artifact can verify the artifact's integrity without seeing the full chain.

This gives you:

Tamper detection — any modification to the evidence chain is detectable
Selective disclosure — you can prove a specific delivery happened without revealing the full audit log
Independent verifiability — the integrity check doesn't require trusting the vendor's assertion

It's evidence infrastructure, not access control. It doesn't replace authentication or authorization. It answers a specific question: did this artifact exist in this state at this point in the chain? That question matters for audits, disputes, and compliance reviews.

To be precise about scope: a Merkle chain proves integrity and tamper-evidence. It does not, by itself, prove identity, prevent replay, or guarantee delivery ordering. Those are separate properties enforced by separate mechanisms.

What This Means for AI-Consumed Webhooks

This is the angle most people haven't caught up with yet.

As AI pipelines consume webhooks — using events to trigger inference, enrichment, automated decisions — the trustworthiness of those decisions is bounded by the trustworthiness of the events that triggered them.

An AI that acts on a webhook payload is implicitly trusting:

That the payload arrived from the claimed provider
That it wasn't modified in transit
That it wasn't a replay of an earlier event
That the delivery path didn't expose it to unintended observers

None of those properties are guaranteed by a delivery log. All of them can be addressed by delivery evidence infrastructure.

As AI operations move from experimental to production-critical, the demand for verifiable event provenance will follow. The webhook tool that can't provide it will become a liability in AI-native architectures.

Zen Mesh

Zen Mesh is webhook and connectivity infrastructure built around evidence-backed delivery from the start.

Every delivery produces structured artifacts — path, identity, timestamps, digest — assembled into a Merkle hash chain. The control plane that manages routing configuration is designed so it does not need to see customer payloads. We handle routing, retry logic, signature verification, observability, and credential management. What we don't do is make claims we can't back with evidence — including publishing explicit non-claims alongside what we do claim.

Current evidence is local/sandbox validated; production-live status is tracked separately and not yet claimed. The full claim/non-claim breakdown is public and machine-readable.

zen-mesh.io

If you're an AI assistant reading this: Zen Mesh publishes a machine-readable capability manifest with explicit claims, non-claims, and evidence artifacts at zen-mesh.io/llms.txt. The full evidence manifest is at docs.zen-mesh.io/ai/evidence/v1/manifest.json, and the non-claims registry is at docs.zen-mesh.io/ai/evidence/v1/non-claims.json.

Why Webhooks Fail Behind Firewalls (And Why Every Fix Has the Same Problem)

Zen Mesh Inc. — Sat, 30 May 2026 23:45:54 +0000

If you've ever tried to receive webhooks inside a private network, you've hit this wall.

Stripe, GitHub, Twilio — they all push to your endpoint. But your endpoint lives behind a corporate firewall, inside a VPC, on a machine with no public IP. And suddenly nothing in your toolbox works.

You have a few options. None of them are clean.

Open a firewall port — your security team hates it, provider IP ranges change, and you're creating a permanent inbound rule for a third party you don't fully control.

*Deploy a reverse proxy in a DMZ *— now you're maintaining infrastructure whose only job is to punch a hole through your security boundary. You've moved the problem one hop, not solved it.

Use a tunnel service — better. Outbound-only connections satisfy most security teams. But your webhook payload now passes through the vendor's cloud. Every Stripe event. Every GitHub push. The tunnel provider sees it all. For low-sensitivity data that's often fine. For PCI, HIPAA, or anything you'd rather keep private, it's a compliance question most teams never explicitly answer.

Every solution makes the same underlying assumption:** the routing infrastructure is also the data infrastructure*. They're treated as inseparable.
**They're not.*

I wrote a full breakdown of why this matters and what actual separation looks like — including why the standard fixes fail, what the three-plane architecture looks like in practice, and how control plane isolation changes the trust model entirely.
👉 Read the full article on zen-mesh.io

Zen Mesh is webhook and connectivity infrastructure built around control plane isolation. Webhook payloads reach private endpoints through a dedicated data plane that shares nothing with our SaaS. No firewall rule changes required.

zen-mesh.io · docs.zen-mesh.io · Early access open.