Your Distributed Lock Can Expire Correctly and Still Corrupt Data

A worker acquires a distributed lock, starts updating a document, and freezes for 12 seconds.

Its 10-second lock expires. Another worker acquires the same lock, finishes the update—and then the first worker wakes up and overwrites it.

Nothing in that sequence requires the lock service to malfunction.

That is the trap: most “distributed locks” are leases. A lease can tell everyone who owns a resource for a limited window. It cannot recall code after that window or pull a delayed write out of the network.

The fix is not a longer TTL. It is a fencing token enforced by the resource being protected.

The lock expired correctly—and data still lost

Imagine two workers and a lock with a 10-second TTL:

worker A: acquire #41 ───── paused ─────────────────── write #41 ✗
lease:                         expires
worker B:                         acquire #42 ─ write #42 ✓
resource:                                      remembers 42

Worker A may pause for garbage collection, CPU starvation, a page fault, a frozen container, or SIGSTOP. Its write may also sit in a network queue after the process sent it.

Those delays are not theoretical. During a 2012 GitHub outage, network traffic between access switches was blocked for roughly 90 seconds. Active/passive file-server pairs missed heartbeats, fencing commands were not delivered, and multiple pairs later believed they were active for the same resource. GitHub kept the site in maintenance mode while recovering its file-storage infrastructure; recovery took more than five hours.

GitHub’s architecture was not our toy worker, but the lesson transfers directly: a timeout is evidence about time, not proof that the old owner has stopped acting.

A TTL turns your mutex into a lease

The usual code looks comforting:

const lease = await locks.acquire(resourceId, { ttlMs: 10_000 });

try {
  const current = await load(resourceId);
  await save(resourceId, update(current));
} finally {
  await lease.release();
}

The hole is the save, not only the release.

Redis’s official locking pattern uses SET key random-value NX PX ttl. The random value matters: release must delete the key only if its value still matches. Otherwise worker A can wake after expiry and delete worker B’s newer lock. Redis 8.4 added DELEX ... IFEQ; older versions use a compare-and-delete Lua script.

But that random value answers only:

“Is this still the lock instance I acquired?”

It does not answer:

“Is this operation newer than every operation previously accepted?”

A UUID has identity but no order. It prevents an unsafe unlock; it cannot reject a stale write.

What if A checks the lock immediately before save? A can pause between the check and the write. What if the process uses a monotonic clock and aborts after its deadline? The packet can be delayed after the process sends it.

There is always a final gap unless the protected resource participates in the decision.

Fencing gives the resource the final say

On each successful acquisition, the coordinator returns a strictly increasing generation:

first owner  → token 41
next owner   → token 42
next owner   → token 43

Every protected write carries that token. The resource remembers the greatest generation it has accepted and rejects anything older.

For a PostgreSQL row, the core can be one atomic statement:

UPDATE documents
SET body = $1,
    fencing_token = $2
WHERE id = $3
  AND fencing_token < $2;

If token 42 has already landed, the later-arriving token 41 updates zero rows. Worker A is stale, regardless of what it believes about its lease.

The comparison and mutation must be atomic. This is subtly broken:

const seen = await db.getFence(id);
if (token > seen) {
  await db.updateDocument(id, body, token);
}

Two workers can both pass the check before either update commits. Use one conditional update, or lock the fence row and perform the fence update plus all business mutations in one database transaction.

For inserts, store the accepted generation beside the logical resource. For a multi-row operation, keep one fence record and the business changes inside the same transaction. For an object store, use its conditional-write primitive if it exposes one.

This pattern has history. Google’s Chubby paper describes a monotonically increasing 64-bit lock generation. A Chubby “sequencer” contains the lock name, mode, and generation; clients pass it to file servers and other protected services, which validate it or compare it with the latest sequencer observed.

The lock service grants an opinion. The resource enforces the order.

Why INCR is not automatically token 42

Fencing needs a token source whose order matches lock acquisition order, including failover.

Adding INCR lock:counter somewhere beside a lease is safe only if the combined grant path guarantees one total order. If the lock is granted but token allocation fails, what happens? If a primary fails after issuing a value that its replacement never saw, can a later owner receive a lower value? If acquisition and increment happen on unrelated systems, which one defines ownership?

This is why the distinction between an owner UUID and a fencing generation matters. Redis’s current distributed-lock documentation explicitly recommends fencing tokens for correctness-sensitive work and warns not to assume a lock remains held as long as its process is alive.

Consensus-backed coordinators already expose useful order. Chubby has lock generations. ZooKeeper’s lock recipe creates ephemeral sequential nodes; the smallest sequence owns the lock, and each waiter watches only its immediate predecessor, avoiding a thundering herd.

Do not invent a distributed counter because the SQL example looks easy. The easy part is comparing generations. The hard part is issuing them in an order you can defend during partitions and failover.

Production-grade means observing lease loss

Fencing handles stale writes, but it does not make leases operationally free.

AWS’s DynamoDB Lock Client guide shows a 10-second lease with a 3-second heartbeat. It also names the costs developers skip in diagrams: extra table capacity, clock-skew sensitivity for short leases, and consistent acquisition ordering when an operation needs multiple locks.

In production, capture at least:

• acquisition wait time
• time the lease is held
• successful-renewal slack before expiry
• renewal failures and lost-lease events
• rejected stale generations

fence_rejected_total should be almost boring—and never ignored. One increment means an old owner attempted a write that would have been accepted without fencing. That metric is evidence that the safety mechanism just prevented corruption.

Renewal is still useful. It reduces unnecessary takeovers during healthy long work. It is not the correctness proof: after the last successful heartbeat, a process or packet can still be delayed past expiry.

Longer TTLs make that overlap less likely but make crash recovery slower. Shorter TTLs improve takeover time but increase false lease loss. Fencing separates that availability tradeoff from data safety.

Sometimes the right lock is no distributed lock

Before adding a coordinator, ask where the protected truth already lives.

Situation	Prefer
One relational database owns all state	Row lock, unique constraint, advisory lock, or conditional update in a transaction
Work is naturally a durable job	Queue claim/ack with retry semantics
Duplicate request can be replayed safely	Idempotency key
Several processes coordinate an external mutable resource	Lease plus a fence the resource enforces
Duplicate cache refresh is only wasted work	A simple best-effort lease may be enough

Fencing also has a hard boundary: the recipient must understand the token. You cannot fence an email after it was sent or ask an arbitrary payment API to compare your generation. For irreversible external effects, route the effect through one durable owner, use the provider’s idempotency facility, or record an outbox entry transactionally and let a controlled dispatcher deliver it.

The lock is not a magical “exactly once” wrapper.

If you remember one thing

A distributed lease can expire while its former owner is still capable of acting. Therefore, ownership alone cannot protect correctness.

Make every sensitive write prove that its generation is newer than the last accepted one—or choose a primitive that keeps coordination and state in the same transaction.

Where are you currently using a TTL lock, and can the resource it protects actually reject a stale owner?