Imagine you are in a bathroom with a 5 minute timer.
You go in but crash at minute 3. The lock is still on for 2 more minutes.
Your friend arrives at minute 3 and just stands outside waiting even though the bathroom is empty.
That wasted time is exactly what happens in software when a process dies but the Redis lock stays alive.
Now imagine three servers behind a load balancer. Every 5 minutes a cron job fires to generate a daily report and all three try to grab it. You use Redis SETNX with a 5 minute TTL. Instance A wins and thinks it owns the bathroom. But then it gets OOM killed at minute 3. The lock remains. No report runs for 2 more minutes. Then another instance grabs it. But wait, the first instance restarted and also thinks it still owns the bathroom. Now both are writing to the same output. Data corruption. 💥
This is the split brain problem. Redis cannot tell if a client is working or just frozen.
So what is the right pattern here?
A) Redis SETNX plus short TTL plus fencing token
B) Redlock across 5 independent Redis nodes
C) Database pessimistic lock using SELECT FOR UPDATE
D) Optimistic concurrency with unique key inserts
All four are real patterns engineers ship. But only one survives this exact failure mode. One of them is what Martin Kleppmann publicly tore apart 8 years ago and people still argue about it today.
Firstly, What would you pick? Drop your answer below. 👇
Let us break it down simply.
Option B, Redlock
This sounds bulletproof. Acquire locks on 5 separate Redis nodes and proceed only if majority agree. But Martin Kleppmann proved that if system clocks drift even slightly, two instances can get the lock simultaneously. It is complex, fragile, and usually over-engineering for most teams. This is the trap.
Option C, Database Pessimistic Lock
You lock the row in your jobs table until the transaction completes. If Instance B dies, the connection drops and the lock releases automatically. Safe? Yes. But heavy. Holding a database connection open for a long running report job is bad practice. Your database becomes the bottleneck instead of your compute.
Option D, Optimistic Concurrency
No locks at all. Each instance generates the full report and tries to insert it with a unique key. Database rejects duplicates. Sounds elegant until you realize both instances just wasted 10 minutes of compute generating a report that gets thrown away. For heavy jobs, this is pure waste.
Option A, Fencing Token
This is your winner. Think of it like a ticket number at a government office. Every time someone gets the lock, they receive a monotonically increasing number. Token 101, then 102, then 103.
When Instance B comes back from its GC pause with Token 101 and tries to write, the storage layer checks: “Current valid token is 103. You have 101. Your lock is stale. Rejected.”
Instance C with Token 103 writes safely. No corruption. No wasted compute. No clock dependency.
The fencing token makes the downstream resource the source of truth, not Redis. Redis just hands out tickets. The storage layer enforces order.
The lesson here is simple. Distributed locks are not about preventing two workers from starting. They are about preventing a zombie worker from writing after it should be dead. TTL alone cannot do that. You need a monotonic guarantee.
If your team has ever argued about Redlock versus SETNX at 3am, share this with them. The debate itself is the learning.
Top comments (0)