A distributed lock looks deceptively simple.
You have multiple instances of a service, but only one of them should perform a particular operation at a time.
So you put a key in Redis:
lock:invoice:123 = worker-1
If the key exists, another worker waits.
If it does not exist, the worker creates it and starts processing.
Problem solved.
Except it isn't.
The difficult part of distributed locking is not acquiring the lock.
The difficult part is answering questions like:
- What happens if the process crashes?
- What happens if the lock expires while the process is still running?
- What happens if another worker acquires the lock?
- What happens if the original worker continues executing?
- What happens if the network connection disappears temporarily?
- Can a worker accidentally release another worker's lock?
- What happens when the protected operation takes longer than the lease?
- Can the system prevent a stale worker from modifying the resource?
These are not edge cases.
They are the reason distributed locking is fundamentally different from using sync.Mutex.
A Distributed Lock Is Not a sync.Mutex in Redis
A Go mutex protects shared memory inside one process.
var mu sync.Mutex
func update() {
mu.Lock()
defer mu.Unlock()
// critical section
}
Every goroutine using that mutex is part of the same process and shares the same memory.
A distributed lock operates across:
┌───────────────┐
│ Redis │
└───────┬───────┘
│
┌─────────┴─────────┐
│ │
Worker A Worker B
Instance 1 Instance 2
The workers do not share memory.
They communicate through a remote system.
That introduces failure modes that a local mutex does not have:
process crash
network partition
packet delay
Redis failover
clock differences
lease expiration
long-running operations
stale clients
Redis itself documents the basic single-instance pattern as an atomic SET with NX and an expiration, using a unique random value for ownership.
That distinction — ownership over a remote lease rather than a local mutex — is the foundation for everything that follows.
The Basic Redis Lock
The simplest useful primitive looks like this:
SET resource_name random_token NX PX 30000
The important parts are:
-
NX: only create the key if it does not exist -
PX 30000: give it a 30-second expiration -
random_token: uniquely identifies the owner
Redis supports combining the conditional set and expiration in a single SET command. SETNX itself is considered deprecated for new locking patterns in favor of SET ... NX.
In Go, using go-redis:
type Lock struct {
client *redis.Client
key string
ttl time.Duration
}
func (l *Lock) Acquire(ctx context.Context, token string) error {
ok, err := l.client.SetNX(
ctx,
l.key,
token,
l.ttl,
).Result()
if err != nil {
return err
}
if !ok {
return ErrLockNotAcquired
}
return nil
}
Usage:
token := uuid.NewString()
if err := lock.Acquire(ctx, token); err != nil {
return err
}
defer func() {
releaseCtx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
if err := lock.Release(releaseCtx, token); err != nil {
log.Printf("lock release failed: %v", err)
}
}()
return processInvoice(ctx)
Use a bounded context for release and report failures rather than silently
discarding them. The TTL remains the recovery mechanism if release cannot
complete.
There is also an ambiguous-outcome case: Redis may apply SET while the reply
is lost or times out. An acquisition error therefore does not always prove
that no lock was created. Do not blindly retry with a new ownership token; that
can leave the first lease held until it expires. Generate the token before the
command and retain it. If recovering from an ambiguous result, an owner check
must also account for the remaining lease time; a matching GET alone cannot
guarantee ownership after the check. Otherwise treat the result as unknown and
rely on the lease to expire.
At first glance this looks good.
But there is already a subtle bug waiting for us.
Never Release a Lock You Don't Own
Imagine this timeline.
Worker A:
acquire lock
token = "aaa"
TTL = 10 seconds
Worker A:
starts processing
10 seconds later:
lock expires
Worker B:
acquires lock
token = "bbb"
Worker A:
finishes processing
Worker A:
DEL lock
That final DEL is dangerous.
Worker A no longer owns the lock.
Worker B does.
If Worker A blindly deletes the key, it just deleted Worker B's lock.
This is why the lock value needs to identify the owner.
The release operation must effectively mean:
if current_owner == my_token:
delete lock
And that check must be atomic.
Not:
owner, _ := rdb.Get(ctx, key).Result()
if owner == token {
rdb.Del(ctx, key)
}
Because another worker can acquire the lock between those two operations.
You need one atomic operation.
A Lua script is a straightforward way to express it:
if redis.call("GET", KEYS[1]) == ARGV[1] then
return redis.call("DEL", KEYS[1])
end
return 0
In Go:
var releaseScript = redis.NewScript(`
if redis.call("GET", KEYS[1]) == ARGV[1] then
return redis.call("DEL", KEYS[1])
end
return 0
`)
func (l *Lock) Release(ctx context.Context, token string) error {
result, err := releaseScript.Run(
ctx,
l.client,
[]string{l.key},
token,
).Int()
if err != nil {
return err
}
if result == 0 {
return ErrLockNotOwned
}
return nil
}
This gives us an important invariant:
A worker may release the lock only if the lock still contains its ownership token.
Redis documents this owner-checking release pattern explicitly.
That solves one class of race.
It does not solve the hardest one.
The Dangerous Part: Your Work Can Outlive Your Lock
Consider a job that normally takes around three seconds.
You configure:
lock TTL = 10 seconds
That sounds reasonable.
Until one day the downstream database becomes slow.
The job takes 15 seconds.
Now:
T=0s
Worker A
acquire lock
└── TTL = 10s
T=8s
Worker A
still processing
T=10s
Redis
lock expires
T=11s
Worker B
acquires lock
T=12s
Worker A
still processing
T=15s
Worker A
writes result
We now have two workers performing the supposedly exclusive operation.
The lock did exactly what we asked.
It prevented another worker from acquiring the lock while the lease was valid.
The mistake was assuming that the lock guaranteed ownership for the entire lifetime of the operation.
It doesn't.
A TTL-based distributed lock is better understood as a lease.
The ownership is valid only for a bounded period.
Why TTLs Are Necessary
You might ask:
Why not just remove the expiration?
Because then a crashed process can leave the system permanently locked.
Worker A
acquire lock
Worker A
crash 💥
Redis
lock remains forever
Every other worker now sees:
lock exists
and waits forever.
The TTL solves the dead-owner problem:
Worker A
acquire lock
Worker A
crash 💥
TTL expires
Worker B
acquire lock
But TTL introduces another problem:
The lease can expire while the owner is still alive.
Distributed locking is therefore a trade-off between two failure modes:
No TTL
↓
dead worker can block everyone forever
TTL
↓
live but slow worker can become stale
This is why choosing a TTL is not simply:
"How long does this job normally take?"
It is a correctness decision.
Lock Renewal
For long-running work, one option is to renew the lease.
For example:
TTL = 30 seconds
renew every 10 seconds
The worker periodically verifies that it still owns the lock and extends the expiration.
Conceptually:
if redis.call("GET", KEYS[1]) == ARGV[1] then
return redis.call("PEXPIRE", KEYS[1], ARGV[2])
end
return 0
The important detail is the ownership check.
Never do:
PEXPIRE lock 30000
without verifying ownership.
Otherwise a stale worker could extend a lock that now belongs to another worker.
A renewal loop might look like:
func (l *Lock) RenewLoop(
ctx context.Context,
token string,
) <-chan error {
errors := make(chan error, 1)
go func() {
defer close(errors)
ticker := time.NewTicker(l.ttl / 3)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
ok, err := l.Renew(ctx, token)
if err != nil {
errors <- err
return
}
if !ok {
errors <- ErrLockLost
return
}
}
}
}()
return errors
}
But renewal doesn't magically make the system safe.
You still have to define what happens when renewal fails.
For example:
Worker A
lock acquired
↓
processing
↓
renewal fails
↓
lock may expire
↓
Worker B acquires lock
What should Worker A do?
The safe answer is generally:
Stop treating yourself as the owner.
This is where context.Context becomes useful. Go's context model is specifically designed to propagate cancellation and deadlines across goroutines and API boundaries.
For example:
workCtx, cancel := context.WithCancel(ctx)
defer cancel()
renewErrors := lock.RenewLoop(workCtx, token)
workDone := make(chan error, 1)
go func() {
workDone <- processInvoice(workCtx)
}()
select {
case workErr := <-workDone:
cancel()
return workErr
case err, ok := <-renewErrors:
cancel()
if !ok {
return errors.New("lock renewal stopped unexpectedly")
}
if err != nil {
return fmt.Errorf("lock lost: %w", err)
}
return errors.New("lock renewal stopped unexpectedly")
case <-ctx.Done():
cancel()
return ctx.Err()
}
This assumes processInvoice observes cancellation. If it cannot stop promptly,
downstream writes still need fencing.
The exact implementation will depend on how the work is structured, but the principle is more important:
Losing the lease should become a first-class state in the worker lifecycle.
But What If the Worker Cannot Stop Immediately?
This is the question that takes distributed locking from "Redis tutorial" to distributed-systems engineering.
Imagine:
Worker A
token = 41
lock expires
Worker B
token = 42
acquires lock
Worker A
continues running
Worker A
writes to database
Even if Worker A knows it lost the lock, there may be no way to instantly stop every operation.
Maybe it is blocked inside:
db.ExecContext(...)
Maybe a network request is already in flight.
Maybe the process is paused.
Maybe the runtime is under heavy CPU pressure.
Maybe the operation cannot be interrupted.
So cancellation alone isn't enough.
We need another mechanism.
Fencing Tokens
This is where fencing tokens become extremely useful.
Instead of treating the lock token only as an ownership identifier, every successful acquisition receives a monotonically increasing number.
For example:
Worker A → fencing token 41
Worker B → fencing token 42
Now the downstream resource can reject stale operations.
Suppose Worker A loses its lease:
Worker A
token = 41
Worker B
token = 42
Worker A tries:
WRITE resource WITH token 41
The resource knows:
latest token = 42
Therefore:
41 < 42
Reject it.
Worker B's operation:
WRITE resource WITH token 42
is accepted.
The key idea is:
The lock determines who may start the work. The fencing token determines whether an operation is still authorized to affect the resource.
This distinction is critical for long-running distributed operations. Redis's own distributed-lock documentation explicitly calls out fencing tokens as important for processes whose work can take significant time.
Generating Fencing Tokens
A monotonically increasing token can be generated with Redis INCR, but it
must be tied atomically to successful lock acquisition. Incrementing in a
separate command can reverse the intended order: an old worker can pause after
acquiring its lease, lose it, then increment after the new owner and receive a
higher token.
For a single Redis primary, one Lua script can perform both operations:
local acquired = redis.call(
"SET", KEYS[1], ARGV[1], "NX", "PX", ARGV[2]
)
if not acquired then
return 0
end
return redis.call("INCR", KEYS[2])
The caller generates and retains the random ownership token in ARGV[1]; a
positive result is the fencing token, while 0 means the lease was not
acquired. The lock key and counter key must be distinct, and the counter must
remain numeric.
Atomic execution on one primary does not make Redis failover strongly
consistent. With asynchronous replication, a failover can lose a successful
lock or counter write, allowing overlapping owners or a token to move
backward. If stale writes would violate a critical invariant, use a coordinator
and downstream enforcement whose durability and ordering guarantees meet that
invariant; a Lua script alone does not provide that guarantee.
Then:
Worker A → 41
Worker B → 42
Worker C → 43
But generating the token is only half of the solution.
The downstream system must actually enforce it.
For example, suppose we store the latest fencing token with the protected resource:
UPDATE invoices
SET
status = $1,
fencing_token = $2
WHERE id = $3
AND fencing_token <= $2;
The column should be NOT NULL with an initial value below the first issued
token. The inclusive comparison permits multiple writes by the same lease;
the stored value still advances when a newer token arrives.
Now a stale worker cannot overwrite a newer state.
A more complete example:
func updateInvoice(
ctx context.Context,
db *sql.DB,
invoiceID string,
status string,
fenceToken int64,
) error {
result, err := db.ExecContext(ctx, `
UPDATE invoices
SET
status = $1,
fencing_token = $2
WHERE id = $3
AND fencing_token <= $2
`,
status,
fenceToken,
invoiceID,
)
if err != nil {
return err
}
rows, err := result.RowsAffected()
if err != nil {
return err
}
if rows == 0 {
return ErrStaleWorker
}
return nil
}
This permits more than one update by the same owner token, while rejecting an
older token after the resource has recorded a newer one. Fencing is not
instantaneous revocation: if the old worker's write reaches the resource before
any write carrying the new token, the resource has not yet observed the newer
token and cannot reject the old one on that basis alone.
Now consider:
Worker A → token 41
Worker B → token 42
Worker B updates first:
fencing_token = 42
Worker A eventually wakes up:
UPDATE ... WHERE fencing_token <= 41
That condition is false.
The stale operation is rejected.
This is fundamentally stronger than trusting the lock's TTL.
Lock Tokens and Fencing Tokens Are Different
It is useful to keep these concepts separate.
A lock ownership token answers:
"Does this client currently own this lease?"
A fencing token answers:
"Is this operation newer than the operations that came before it?"
For example:
Lock token:
"550e8400-e29b-41d4-a716-446655440000"
Fencing token:
42
The first is useful for safe release:
GET lock == my-token
The second is useful for downstream ordering:
incoming-token >= stored-token
They solve different problems.
A More Complete Lock Lifecycle
A production workflow might therefore look like this:
┌──────────────────┐
│ Acquire lease │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Get fence token │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Start work │
└────────┬─────────┘
│
┌────────────┴────────────┐
│ │
▼ ▼
renew succeeds renewal fails
│ │
│ ▼
│ cancel work
│ │
▼ ▼
continue work stop writes
│
▼
fenced downstream writes
│
▼
owner-checked release
This is much closer to a real distributed coordination mechanism than simply:
SET lock
doWork()
DEL lock
PostgreSQL Can Sometimes Be a Better Lock Manager
Redis is not the only option.
If the resource you're coordinating already lives in PostgreSQL, adding Redis solely for locking can introduce unnecessary infrastructure.
For example, database row locking can provide transactional coordination:
SELECT id
FROM jobs
WHERE status = 'pending'
ORDER BY id
FOR UPDATE SKIP LOCKED
LIMIT 1;
This can be extremely useful for worker queues backed by PostgreSQL.
The database already knows:
who owns the row
what transaction owns it
when the transaction ends
You may not need a distributed lock at all.
PostgreSQL advisory locks are another option when the coordination is conceptually tied to the database.
The broader design question should therefore be:
Where does ownership already live?
If your work is already represented as a database row, using a separate Redis lock may be unnecessary complexity.
Distributed Lock vs Queue Ownership
Sometimes the correct solution isn't a lock.
Suppose you have:
10 workers
100,000 jobs
You might initially think:
lock job
process job
unlock job
But a queue can model ownership directly.
For example:
Job 123 → Worker A
Job 124 → Worker B
Job 125 → Worker C
The system is not asking:
"Who currently owns this global lock?"
It is asking:
"Who owns this piece of work?"
That distinction often produces simpler systems.
Locks are especially attractive when multiple workers need to coordinate access to the same shared resource.
Queues are often more natural when the problem is work distribution.
Distributed Lock vs Leader Election
These are also different concepts.
A distributed lock might say:
Only one worker should process this resource right now.
Leader election says:
One instance should act as the active controller.
For example:
Worker A
Worker B
Worker C
might elect:
Worker B = leader
Worker B then performs:
scheduler
configuration refresh
control-plane reconciliation
This is not necessarily the same problem as:
lock:customer:123
Trying to solve both problems with the same primitive often creates unnecessary complexity.
Context-Aware Lock Acquisition
Lock acquisition should also respect the caller's deadline.
This is especially important for HTTP handlers and request-driven workflows.
Imagine:
HTTP timeout = 2 seconds
lock acquisition timeout = 30 seconds
A request might spend the entire 2 seconds waiting for a lock and then fail anyway.
Instead:
func (l *Lock) AcquireWithRetry(
ctx context.Context,
) (string, error) {
delay := 50 * time.Millisecond
const maxDelay = time.Second
for {
token := uuid.NewString()
err := l.Acquire(ctx, token)
if err == nil {
return token, nil
}
if !errors.Is(err, ErrLockNotAcquired) {
return token, err
}
jitter := time.Duration(rand.Int63n(int64(delay / 2)))
timer := time.NewTimer(delay + jitter)
select {
case <-timer.C:
if delay < maxDelay {
delay *= 2
if delay > maxDelay {
delay = maxDelay
}
}
case <-ctx.Done():
timer.Stop()
return "", ctx.Err()
}
}
}
The exact bounds are workload-dependent. Exponential backoff with jitter avoids
having contending workers retry in lockstep. The important part is that waiting
for ownership is part of the operation's deadline.
Go's context package is designed to propagate cancellation and deadlines through a chain of operations, including across goroutines and external calls.
Don't Create a Lock That Lives Forever
A tempting design is:
Acquire lock
Renew forever
This can be dangerous.
Imagine the renewal goroutine survives while the actual worker is stuck.
You can accidentally create a lock that remains alive even though the operation is no longer making progress.
The lifecycle should therefore be explicit:
worker starts
↓
lock acquired
↓
renewal starts
↓
work starts
↓
work finishes / fails / is canceled
↓
renewal stops
↓
lock released
The renewal mechanism should belong to the same lifecycle as the work it protects.
This is one of the reasons structured concurrency principles are useful even when implementing distributed coordination.
What About Redlock?
Redis also documents the Redlock algorithm for using multiple independent Redis masters.
The idea is roughly:
Redis A ──┐
Redis B ──┤
Redis C ──┼── acquire majority
Redis D ──┤
Redis E ──┘
The goal is to reduce dependence on a single Redis instance.
But Redlock is not a magic word that turns distributed coordination into a formally solved problem.
There has been substantial discussion around its safety assumptions, timing model, failure modes, and whether it provides the guarantees applications actually need.
Redis's own documentation notes that the analysis of Redlock should be considered carefully, including clock behavior and the use of fencing tokens.
The important engineering lesson is:
Don't choose Redlock because "distributed locks require Redlock."
Choose a coordination mechanism based on the consistency guarantees your application actually needs.
If stale work can corrupt data, you need to think about stale work independently of the lock algorithm.
Observability Matters
A distributed lock without metrics is difficult to operate.
At minimum, I would want to know:
lock_acquisition_attempts
lock_acquisition_failures
lock_acquisition_latency
lock_hold_duration
lock_renewal_failures
lock_release_failures
lock_lost_events
stale_operation_rejections
For example:
lock_acquisition_latency:
p50 = 3ms
p95 = 42ms
p99 = 310ms
A sudden increase might indicate:
higher contention
slow Redis
long-running workers
stuck jobs
incorrect TTL
worker overload
You should also log ownership transitions with enough context to debug them:
lock acquired
resource=invoice:123
owner=worker-a
fence=41
lock renewal failed
resource=invoice:123
owner=worker-a
fence=41
stale operation rejected
resource=invoice:123
incoming_fence=41
current_fence=42
That is much more useful than:
lock failed
When You Should Not Use a Distributed Lock
This might be the most important section in the entire article.
Before introducing a distributed lock, ask whether the problem can be solved with:
Idempotency
Instead of preventing duplicate execution, make duplicate execution safe.
request A
request A again
request A again
If all three produce the same valid final state, a lock may not be necessary.
Database constraints
Sometimes:
UNIQUE(user_id, operation_id)
is enough to enforce the invariant.
Optimistic concurrency
For example:
UPDATE accounts
SET balance = $1,
version = version + 1
WHERE id = $2
AND version = $3;
No distributed lock required.
Queue ownership
Let the queue determine which worker processes which job.
Transactions
If the critical section exists entirely inside one database transaction, the database may already provide the coordination primitive you need.
Leader election
If the actual requirement is "one active controller", use a leader-election mechanism rather than creating dozens of unrelated distributed locks.
The best distributed lock is sometimes the lock you don't need.
Production Checklist
Before shipping a distributed lock, I would want to answer all of these:
□ What exactly is being protected?
□ Why does this require distributed coordination?
□ Can idempotency solve the problem instead?
□ Can a database transaction solve it?
□ Can queue ownership solve it?
□ Does the lock have a TTL?
□ What happens when the process crashes?
□ Does every acquisition have a unique ownership token?
□ Is release atomic and ownership-checked?
□ What happens if the TTL expires during work?
□ Can the worker renew the lease?
□ What happens when renewal fails?
□ Can stale workers still reach the downstream resource?
□ Do we need fencing tokens?
□ Does the downstream system enforce fencing?
□ Is lock acquisition bounded by context/deadline?
□ What happens during Redis failure?
□ What happens during network partitions?
□ What metrics tell us that contention is increasing?
□ How do we recover from stuck work?
If these questions don't have clear answers, the implementation isn't finished just because the Redis command works.
The Real Mental Model
The easiest mistake is to think about a distributed lock like this:
lock()
doWork()
unlock()
A more accurate mental model is:
acquire lease
│
▼
obtain ownership
│
▼
do work
│
┌──────┴──────┐
│ │
lease valid lease lost
│ │
▼ ▼
continue stop/cancel
│ │
▼ ▼
fenced writes reject stale work
│
▼
owner-checked release
The distinction matters because distributed systems don't give you the guarantees that local memory gives you.
Processes crash.
Networks delay messages.
Connections disappear.
Workers pause.
Operations take longer than expected.
And a worker can continue running after the system has stopped considering it the owner.
That last case is the one that breaks many otherwise reasonable distributed-lock implementations.
Final Takeaway
A distributed lock is not simply a mutex stored in Redis.
It is a lease over ownership in a system where processes can disappear, networks can fail, and work can outlive the lease that started it.
A production design therefore needs to think about more than acquisition:
ownership
↓
lease duration
↓
expiration
↓
renewal
↓
loss of ownership
↓
stale work
↓
fencing
↓
downstream enforcement
If your only question is:
"How do I prevent two workers from entering this critical section?"
a simple lock may be enough.
If your real question is:
"How do I guarantee that an old worker cannot corrupt shared state after it loses ownership?"
then you are no longer just designing a lock.
You are designing a distributed coordination protocol.
And that is where the interesting engineering begins.
Top comments (1)
Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support