This is article 6 of 10 in Beyond Kubernetes Defaults. Read article 5: A Running Pod Is Not an Admitted Workload first.
Single-process correctness can hide distributed-system bugs. Replay protection is a particularly clear example.
One replica, then two
A signed request arrives with a timestamp and a nonce. The application verifies the signature and records the nonce so it cannot be accepted twice. With one process and local state, this can look correct.
Now run two replicas behind a Kubernetes Service. The first request lands on replica A, which records nonce N locally. A replay lands on B, which has never seen N. B accepts the same signed request.
A Deployment or Service does not turn local security state into shared state. Scaling is not state synchronization.
Sticky routing is not a security boundary
Consistent hashing or sticky sessions can reduce the chance of a replay reaching another replica. But Pods restart, pools resize and endpoints change. If membership movement makes a consumed nonce fresh again, routing affinity reduces probability; it does not enforce the guarantee.
Define the invariant
After the first successful use of a nonce, every later use inside the replay window must be denied, regardless of which live replica receives it.
A shared store, safely replicated state, or correctly managed partition ownership may help, but each design needs explicit answers about:
- Scope: include the appropriate tenant or source in the replay key.
- Atomicity: checking and recording must be one atomic operation; two replicas must not both accept the same nonce.
- Lifetime: retain the record for the full replay window, accounting for clock skew and membership changes.
- Failure: decide what happens when the replay authority is unavailable. For security-sensitive signed ingress, a retryable refusal is often safer than silently accepting without the check.
Acknowledgement has a similar boundary
If a service returns HTTP 200 after putting an event only in process memory, a crash can lose the event after the sender believes it was accepted. A durable acknowledgement should correspond to the durability guarantee the service actually offers.
Test the seams
A useful HA test is not just killing a Pod and checking that another stays up:
- Send a signed request with nonce N and confirm acceptance.
- Replay N to a different replica and confirm rejection.
- Replace a replica and replay N again.
- Scale the pool up and down and repeat within the replay window.
- Make the replay authority unavailable and verify the documented failure behavior.
Also test the difference between recording durable acceptance and marking a downstream effect complete: suppressing a retry before an effect succeeds can lose an event.
This is not exactly-once delivery
Fencing, replay protection and idempotency protect different boundaries. At-least-once transport with correctly scoped idempotent effects can be useful, but exactly-once claims need precise boundaries and evidence.
A second replica is an availability feature and a correctness question: do your guarantees survive when state is distributed?
Originally published on the Zen Mesh blog.
Top comments (0)