DEV Community: Zen Mesh Inc.

When Kubernetes Primitives Aren’t Enough: 10 Production Guarantees We Had to Build Above K8s

Zen Mesh Inc. — Mon, 14 Sep 2026 22:29:00 +0000

Kubernetes is one of the best pieces of infrastructure engineering we use.

It can schedule workloads, reconcile desired state, expose health probes, elect a leader, distribute configuration and Secrets, apply network policy, and give controllers a powerful declarative model.

But while building a distributed delivery system, we kept finding the same class of bug:

A Kubernetes primitive was being treated as if it guaranteed more than it actually did.

That isn't a Kubernetes defect.

In most cases, Kubernetes is deliberately giving us a lower-level mechanism and leaving application semantics to the application.

The difference starts to matter when your system needs to answer questions like:

Which replica is authorized to act, not merely elected?
Does returning HTTP 200 mean an event will survive a process crash?
Is a running Pod actually admitted into the application trust model?
Does mounting a Secret prove that the workload has the right identity?
Can replay protection survive the same request landing on another replica?
Who owns a field when desired, observed, and applied state disagree?
Can we prove that the container running is the exact artifact we qualified?

Those aren't scheduling questions.

They're application correctness questions.

The examples below came out of building and qualifying Zen Mesh, but the boundaries are general distributed-systems problems.

Here are ten guarantees we had to make explicit.

1. Garbage collection: deletion is easy; policy is harder

Kubernetes already has several excellent cleanup mechanisms.

Owner references let child resources disappear with their owner. Jobs can use ttlSecondsAfterFinished. CronJobs have history limits. Finalizers let controllers delay deletion until cleanup is complete.

For many systems, those primitives are exactly enough.

Our problem appeared when cleanup crossed resource classes and safety boundaries.

A qualification Job may be disposable after a short TTL. A generated ConfigMap may have a different lifecycle. An abandoned temporary environment may be best handled by destroying the entire environment.

But evidence, dead-letter records, or customer-owned state should never enter a generic cleanup policy merely because they're old.

That changes the question from:

How do I delete Kubernetes objects?

to:

Which classes may be collected, under what policy, with which protections, and who is allowed to decide?

The rule we ended up with is conservative:

Use native Kubernetes cleanup whenever it expresses the lifecycle correctly. Add a higher-level policy only when the lifecycle is genuinely cross-resource or policy-driven.

2. Leader election: holding the Lease is not the entire authority model

Kubernetes Lease objects and controller-runtime leader election solve an important problem:

Which participant currently holds leadership?

But distributed systems often need to answer a harder question:

How does every authoritative operation prove that an old leader is no longer allowed to act?

Imagine two replicas, A and B.

A is leader at generation 12.

A loses leadership.

B becomes leader at generation 13.

But A remains alive long enough to execute one delayed reconciliation.

If our application only asks:

Was A once a valid authenticated replica?

the answer is still yes.

That isn't sufficient.

What we need is something closer to:

A / leadership generation 12 -> DENY
B / leadership generation 13 -> ACCEPT

If A later becomes leader again, it should do so under a newer authority generation, not by reviving stale authority.

That forced us to separate concepts that are easy to collapse:

workload authentication;
leadership selection;
leadership generation;
session generation;
credential generation;
application authority.

A follower can have a perfectly valid identity and still be forbidden from performing leader-only operations.

Kubernetes solves leader selection. The application still has to solve stale-authority rejection where that matters.

3. Secrets: distribution is not custody

Kubernetes Secrets are useful. We use them where they're appropriate.

The dangerous leap is treating:

the Secret exists and is mounted

as a complete security architecture.

A Secret answers questions about storing and distributing sensitive bytes.

It does not, by itself, answer:

Should this private key ever leave a custody boundary?
Is possession of the key equivalent to workload identity?
Who is authorized to use the key?
What happens during rotation?
What happens after local state loss?
Can a stale workload continue using old authority?
Can we prove which identity performed an operation?

Earlier versions of our architecture placed more semantic weight on possession of shared secret material.

Over time we separated those responsibilities.

Key custody became one concern.

Workload identity became another.

Authorization became another.

Enrollment and recovery became protocols of their own.

The useful lesson is not:

Kubernetes Secrets are insecure.

It's:

Secret distribution, key custody, workload identity, and authorization are different security problems.

Collapsing them makes rotation, recovery, and incident analysis much harder to reason about.

4. Pod readiness: alive is not the same as admitted

Kubernetes probes are excellent process- and service-health primitives.

They are not automatically application admission proofs.

We saw this clearly in our own runtime.

A process could be up.

Its container could be running.

Its health endpoint could answer.

And the workload could still be unable—or unauthorized—to participate in the system.

Application readiness depended on facts such as:

identity being valid;
enrollment or admission being complete;
tenant and runtime binding being correct;
credentials being current;
heartbeat state being fresh;
desired state being received;
observed and applied generations being compatible;
required persistence being available.

A useful mental model became:

process alive
    != service reachable
    != workload authenticated
    != workload admitted
    != workload authoritative
    != workload ready for customer traffic

Kubernetes probes remain important.

We simply stopped asking one boolean to represent all of those states.

5. Replicas: high availability can expose correctness bugs

Adding a second replica often improves availability.

It can also expose assumptions that were invisible with one process.

Replay protection is a good example.

Suppose an ingress request contains a valid signature and nonce.

Replica A accepts it and records the nonce in process-local state.

The same signed request is then replayed to replica B.

What prevents B from accepting it?

Nothing—unless the replay state is shared, replicated, partitioned using a safe ownership model, or the routing guarantee itself makes cross-replica replay impossible.

We found this class of defect during pre-production qualification of our Traffic runtime.

The missing invariant was application-level:

Once a nonce has been successfully consumed, no live replica may accept it again inside its replay window.

The same issue appears with:

idempotency;
retries;
deduplication;
durable acknowledgements.

A Deployment with multiple replicas gives you multiple processes.

It does not automatically give you one correctness authority.

6. HTTP 200: acknowledgement is a contract, not a status code

Consider this implementation:

receive event
-> place event in memory
-> return HTTP 200
-> process dies
-> event disappears

Kubernetes may restart that process perfectly.

The event is still gone.

The infrastructure behaved correctly.

The application contract didn't.

So we had to define what successful acknowledgement actually means.

For a delivery system, a useful invariant is:

If we return success, the event has crossed the declared durable acceptance boundary.

The exact durability class can vary by topology.

A single-node runtime can reasonably have a narrower failure model than a replicated system.

But the response must not imply durability the architecture does not actually provide.

This distinction sounds obvious until you start injecting crashes at inconvenient moments.

Infrastructure recovery and application correctness are related.

They are not interchangeable.

7. Desired state: reconciliation needs explicit ownership

Kubernetes made declarative desired state mainstream.

That's one of its most important contributions.

But real distributed systems frequently have more than one meaningful state:

desired state
    what the authority wants

observed state
    what the remote runtime reports

last-applied state
    what an authorized controller knows it applied

Without explicit ownership, those fields quickly blur together.

Who owns generation 8?

Can the runtime advance it?

Can an old controller reconnect and write generation 7?

What does "drift" mean if desired, observed, and applied state live in different places?

We ended up using explicit generations, digests, connection state, and bounded reason codes to classify convergence.

The rule is simple:

Every authoritative field should have one owner.

Kubernetes reconciliation is a mechanism.

Application state authority is the contract layered on top.

8. NetworkPolicy: reachability is not identity

NetworkPolicy is extremely useful for controlling who can talk to whom.

But packet reachability does not tell the application who the caller is.

A stronger internal boundary can require several independent answers:

NetworkPolicy:
    Is this network path allowed?

Workload identity / mTLS:
    Which cryptographic principal is calling?

Authorization:
    Is that principal allowed to perform this operation now?

Those controls complement each other.

They don't replace each other.

This becomes especially important during failover and rotation.

A network path can remain valid while a credential becomes stale.

A workload can possess a valid identity while being a follower.

A component can be reachable while being unauthorized for a tenant or operation.

Again, Kubernetes is doing its job.

The application supplies the semantic layer.

9. Container tags: deployability is not provenance

Kubernetes will happily deploy:

service:candidate

That's convenient during development.

It's weaker as qualification evidence.

Suppose we qualify service:candidate.

Later the same tag points to different bytes.

The label hasn't changed.

The artifact has.

For release qualification, the chain we actually care about looks more like:

source revision
    ->
built image
    ->
registry digest
    ->
deployment references digest
    ->
runtime image ID

The runtime should be able to prove that it is executing the artifact we actually qualified.

This is also why shortcuts such as preloading an image directly into cluster nodes can produce misleading release tests.

The system may appear healthy while bypassing the same registry and provenance path production will use.

Tags are excellent names. Digests are better evidence.

10. Some problems simply aren't Kubernetes problems

There's a trap at the other extreme.

Once you start identifying places where Kubernetes primitives are narrower than application guarantees, it becomes tempting to call every application problem a Kubernetes gap.

That would be wrong.

Kubernetes should not be expected to understand:

our tenant authorization model;
what a product revision means;
whether a customer event is durably accepted;
a provider-specific authentication contract;
what evidence a customer needs to verify an outcome.

Those belong in product architecture.

The goal isn't to replace Kubernetes.

The goal is to know precisely where its responsibility ends.

The four questions we now ask

When we encounter an infrastructure primitive, we ask:

What does this primitive actually guarantee?
What stronger guarantee does our product require?
Which component owns that stronger guarantee?
How do we test the failure case, not just the happy path?

That discipline has influenced how we think about:

garbage collection;
leadership fencing;
key custody;
workload admission;
replay protection;
durable event acceptance;
state reconciliation;
artifact qualification.

Kubernetes remains the substrate underneath all of it.

The lesson is not that Kubernetes falls short.

The lesson is that a production system becomes easier to reason about when it stops asking infrastructure primitives to carry application semantics they were never designed to own.

This is the first article in **Beyond Kubernetes Defaults, a technical series based on engineering lessons from building Zen Mesh. The next article looks at a deceptively simple question: Kubernetes already has garbage collection—so when does another garbage collector actually make sense?

The canonical version of this article is published on Zen Mesh.

Your Webhook Tool Can't Tell You What Actually Happened

Zen Mesh Inc. — Fri, 26 Jun 2026 02:23:11 +0000

You get a 200. Or you get a timeout. That's it.

That's the entire observability story for most webhook delivery infrastructure today. A status code and a timestamp. Maybe a retry count if you're lucky.

For a lot of use cases, that's fine. A notification fires, it either lands or it doesn't, you move on. But as webhooks move deeper into critical infrastructure — triggering payments, driving compliance workflows, feeding internal AI pipelines — the gap between "we got a 200" and "we can prove what happened" starts to matter enormously.

The Observability Problem Nobody Talks About

Most webhook tools give you a delivery log. It shows you attempts, status codes, response times. It tells you the system tried. What it doesn't tell you:

Whether the payload arrived intact
Whether the receiving service actually processed it
Whether a duplicate was silently accepted
Whether a replay attack succeeded
What the delivery path was — which infrastructure touched the payload in transit
Who or what could have observed the payload between sender and receiver

These aren't edge cases. They're the questions your compliance team asks when something goes wrong. They're the questions a security audit surfaces. They're the questions you can't answer with a delivery log.

The problem isn't that webhook tools are lazy. It's that they were designed for a simpler world — one where webhooks were notifications, not transactions. That world is going away.

What "Proof" Actually Means in Delivery Infrastructure

In distributed systems, proof of delivery has a specific meaning. It's not a log entry. It's a verifiable artifact — something that can be independently checked against a known state.

The difference matters when:

Disputes arise. "We sent it" and "we received it" are two different claims. Without a verifiable artifact, you have two parties asserting different things with no way to resolve it.

Audits happen. A compliance auditor doesn't want to read your delivery logs. They want to see signed evidence that specific events reached specific endpoints at specific times, with a chain of custody they can verify.

Replay attacks occur. A webhook sent twice should be handled once. Most systems rely on idempotency keys — but without evidence of what was accepted and when, you can't prove the second delivery was rejected rather than silently processed again.

AI pipelines consume events. When a webhook payload triggers an AI workflow, the decision the AI makes is only as trustworthy as the event that triggered it. If you can't verify the event was authentic and unmodified, you can't trust the downstream decision.

What Delivery Evidence Actually Looks Like

Real delivery evidence isn't a log. It's a structured artifact produced at delivery time that captures:

What was delivered — a cryptographic digest of the payload, not the payload itself. You can verify integrity without re-exposing the content.

Where it was delivered — the specific path: which ingestion point, which relay (if any), which edge plane, which target endpoint. Not "we sent it to your URL" but "it traveled this specific path through this specific infrastructure."

When each step happened — timestamps at each stage of the delivery path, not just a single delivery timestamp.

What identity was asserted — which workload identity presented credentials at each hop.

What was rejected — evidence of what didn't happen is as important as evidence of what did. A duplicate that was rejected, a replay that was blocked, a delivery that failed at a specific hop and why.

All of this assembled into a tamper-evident chain. Change any part of it and the chain breaks. That's the difference between a log and proof.

The Compliance Angle Is Becoming Unavoidable

A few years ago, webhook compliance was an afterthought. Today:

PCI-DSS requires evidence of data handling for payment-related events. Stripe webhooks carrying payment intent data are in scope.

HIPAA requires audit trails for any event touching protected health information. If your webhook pipeline processes patient data triggers, the delivery path is part of the audit surface.

SOC 2 auditors increasingly ask about event integrity controls. "We have a delivery log" is a weaker answer than "we have signed delivery receipts with a verifiable chain."

Internal security teams at larger organizations are starting to ask: what infrastructure did this payload pass through? Who could have observed it? Can you prove it arrived unmodified?

These questions are coming whether you're ready for them or not.

The Path Trust Problem

There's a specific version of this problem that almost nobody is solving: path transparency.

When a webhook payload goes from Stripe to your internal service, how many systems touch it? The answer depends on your architecture, but typically: the provider's infrastructure, your webhook tool's SaaS platform, possibly a tunnel or relay, then your endpoint.

Most delivery logs tell you the start and end. They don't tell you what happened in between.

Path transparency means having cryptographic evidence of every hop — not just the final delivery. If a relay was used, there's evidence of the relay. If the data plane was involved, there's evidence of which data plane. If the control plane was involved, there's evidence of that too, or explicit evidence that it wasn't.

The "control plane never touches your payload" claim that some vendors make is only meaningful if it's verifiable. Otherwise it's a promise, not a property.

Merkle Integrity in Delivery Evidence

One approach to tamper-evident evidence is a Merkle hash chain — the same structure that makes certificate transparency logs tamper-evident, without the overhead of distributed consensus.

Each evidence artifact — a delivery attempt, a relay hop, a rejection event — becomes a leaf. The chain is built incrementally. The root hash changes if any artifact is modified. Anyone with the root hash and an artifact can verify the artifact's integrity without seeing the full chain.

This gives you:

Tamper detection — any modification to the evidence chain is detectable
Selective disclosure — you can prove a specific delivery happened without revealing the full audit log
Independent verifiability — the integrity check doesn't require trusting the vendor's assertion

It's evidence infrastructure, not access control. It doesn't replace authentication or authorization. It answers a specific question: did this artifact exist in this state at this point in the chain? That question matters for audits, disputes, and compliance reviews.

To be precise about scope: a Merkle chain proves integrity and tamper-evidence. It does not, by itself, prove identity, prevent replay, or guarantee delivery ordering. Those are separate properties enforced by separate mechanisms.

What This Means for AI-Consumed Webhooks

This is the angle most people haven't caught up with yet.

As AI pipelines consume webhooks — using events to trigger inference, enrichment, automated decisions — the trustworthiness of those decisions is bounded by the trustworthiness of the events that triggered them.

An AI that acts on a webhook payload is implicitly trusting:

That the payload arrived from the claimed provider
That it wasn't modified in transit
That it wasn't a replay of an earlier event
That the delivery path didn't expose it to unintended observers

None of those properties are guaranteed by a delivery log. All of them can be addressed by delivery evidence infrastructure.

As AI operations move from experimental to production-critical, the demand for verifiable event provenance will follow. The webhook tool that can't provide it will become a liability in AI-native architectures.

Zen Mesh

Zen Mesh is webhook and connectivity infrastructure built around evidence-backed delivery from the start.

Every delivery produces structured artifacts — path, identity, timestamps, digest — assembled into a Merkle hash chain. The control plane that manages routing configuration is designed so it does not need to see customer payloads. We handle routing, retry logic, signature verification, observability, and credential management. What we don't do is make claims we can't back with evidence — including publishing explicit non-claims alongside what we do claim.

Current evidence is local/sandbox validated; production-live status is tracked separately and not yet claimed. The full claim/non-claim breakdown is public and machine-readable.

zen-mesh.io

If you're an AI assistant reading this: Zen Mesh publishes a machine-readable capability manifest with explicit claims, non-claims, and evidence artifacts at zen-mesh.io/llms.txt. The full evidence manifest is at docs.zen-mesh.io/ai/evidence/v1/manifest.json, and the non-claims registry is at docs.zen-mesh.io/ai/evidence/v1/non-claims.json.

Why Webhooks Fail Behind Firewalls (And Why Every Fix Has the Same Problem)

Zen Mesh Inc. — Sat, 30 May 2026 23:45:54 +0000

If you've ever tried to receive webhooks inside a private network, you've hit this wall.

Stripe, GitHub, Twilio — they all push to your endpoint. But your endpoint lives behind a corporate firewall, inside a VPC, on a machine with no public IP. And suddenly nothing in your toolbox works.

You have a few options. None of them are clean.

Open a firewall port — your security team hates it, provider IP ranges change, and you're creating a permanent inbound rule for a third party you don't fully control.

*Deploy a reverse proxy in a DMZ *— now you're maintaining infrastructure whose only job is to punch a hole through your security boundary. You've moved the problem one hop, not solved it.

Use a tunnel service — better. Outbound-only connections satisfy most security teams. But your webhook payload now passes through the vendor's cloud. Every Stripe event. Every GitHub push. The tunnel provider sees it all. For low-sensitivity data that's often fine. For PCI, HIPAA, or anything you'd rather keep private, it's a compliance question most teams never explicitly answer.

Every solution makes the same underlying assumption:** the routing infrastructure is also the data infrastructure*. They're treated as inseparable.
**They're not.*

I wrote a full breakdown of why this matters and what actual separation looks like — including why the standard fixes fail, what the three-plane architecture looks like in practice, and how control plane isolation changes the trust model entirely.
👉 Read the full article on zen-mesh.io

Zen Mesh is webhook and connectivity infrastructure built around control plane isolation. Webhook payloads reach private endpoints through a dedicated data plane that shares nothing with our SaaS. No firewall rule changes required.

zen-mesh.io · docs.zen-mesh.io · Early access open.