22. The Best Distributed Lock Is Often No Lock at All
By now, we've explored distributed locks, leases, fencing tokens, leader election, and consensus.
Each of these mechanisms exists to solve a specific coordination problem in distributed systems, where multiple machines must agree on who is allowed to do what and when.
A natural conclusion from studying all of them might be that building reliable distributed systems requires increasingly sophisticated coordination mechanisms.
However, in practice, experienced engineers often arrive at the opposite conclusion after working with these systems at scale. Before introducing any form of distributed coordination, they tend to step back and ask a more fundamental question:
Can we re-design the system so that coordination isn't necessary in the first place?
This question is important because every coordination mechanism introduces real cost into the system.
It adds additional network communication between services. It increases the number of failure scenarios that must be handled correctly. It introduces operational complexity that must be monitored, debugged, and maintained. It often increases latency, since coordination typically requires waiting for agreement or confirmation.
Because of these costs, the most reliable distributed systems are not necessarily the ones with the strongest or most complex coordination mechanisms.
Instead, they are often the systems that minimize coordination as much as possible while still preserving correctness.
23. Let the Database Protect Its Own Data
Suppose an application allows users to register with an email address.
A straightforward but naive approach might introduce a distributed lock to prevent duplicate registrations. The flow would look like this:
Acquire Distributed Lock
|
v
Check Email Exists
|
v
Insert Customer
|
v
Release Lock
In this design, the distributed lock ensures that two application instances do not attempt to create the same customer at the same time.
While this approach works in principle, it is often unnecessary in practice. Relational databases already provide strong guarantees about data integrity, including uniqueness constraints. Instead of coordinating at the application level, we can express the business rule directly in the database schema.
A unique constraint makes the intent explicit and enforces it reliably:
CREATE UNIQUE INDEX uk_customer_email
ON customer(email);
With this constraint in place, every application instance can simply attempt the insert operation without any external coordination.
The flow becomes much simpler:
Insert Customer
|
|
Unique Constraint
|
Success/Reject
There is no distributed lock.
There is no coordination service.
There is no lease management.
The database already owns the data, and therefore it is the most appropriate place to enforce rules about that data.
Whenever correctness depends entirely on the state of a single database, delegating enforcement to the database itself is usually the simplest and most reliable solution.
24. Optimistic Concurrency Instead of Ownership
Some coordination problems arise because multiple application instances attempt to modify the same record at the same time.
At first glance, this seems to require a distributed lock to ensure exclusive access.
However, that assumption is not always correct.
Consider an inventory record:
Product
--------
Quantity = 25
Version = 8
Now imagine two application instances reading and updating this record concurrently.
Both instances start from the same version:
Application A
Version 8
-------------------
Application B
Version 8
Application A performs an update first, and the database successfully applies the change:
Version = 9
When Application B attempts to apply its update, it is still working with the outdated version 8. The database detects this mismatch and rejects the update.
At this point, the application can retry the operation using the latest state from the database.
No distributed lock was required at any point in this process.
No application had to wait for exclusive ownership of the record.
Instead of preventing concurrent work, optimistic concurrency allows concurrent operations to proceed and resolves conflicts only when they actually occur. This approach is particularly effective when conflicts are relatively rare.
It provides high throughput because operations are not blocked unnecessarily. It minimizes coordination overhead between services. It also offers a simple recovery model, where failed updates are retried with fresh state.
This combination of properties is why optimistic concurrency is widely used in modern backend systems.
25. Idempotency Eliminates Many Locking Problems
Consider a scenario where a payment request is received more than once.
This can happen for many reasons in distributed systems.
A client might retry after a timeout, even though the first request already succeeded.
A message broker might deliver the same event more than once.
A network failure might occur after the operation completes but before the response is acknowledged.
One possible solution is to use a distributed lock to ensure the payment is only processed once:
Acquire Lock
|
Charge Customer
|
Release Lock
However, there is another approach that avoids coordination entirely by changing the nature of the operation itself.
In this design, every request includes an idempotency key that uniquely identifies the operation:
Payment Request
Idempotency Key
PAY-48291
Before processing the payment, the system checks whether this key has already been used.
The flow becomes:
Receive Request
|
Already Processed?
|
+---+---+
| |
Yes No
| |
Return Process
Result Payment
If the request has already been processed, the system simply returns the previous result. If not, it proceeds with the payment and records the key. This makes duplicate requests safe by design.
Instead of coordinating ownership between multiple application instances, the system ensures that repeating the same operation does not change the outcome.
This approach is often more robust than distributed locking because retries are not edge cases in distributed systems—they are expected behavior.
Reliable systems are designed with retries in mind. They do not merely try to prevent them; they make them safe.
26. Partition Ownership Instead of Global Coordination
Consider a payment platform that processes millions of transactions every day.
If every payment required a distributed lock, the system would quickly become a bottleneck due to the overhead of coordination.
A more scalable approach is to partition the work.
Payments A-H
|
v
Consumer A
-------------------
Payments I-P
|
v
Consumer B
-------------------
Payments Q-Z
|
v
Consumer C
In this model, each consumer is responsible for a specific subset of the workload. Each partition has a clear owner, and that ownership is determined by the partitioning strategy rather than dynamic locking.
As a result, applications no longer compete for individual operations. Instead, they operate independently within their assigned partitions. This pattern is commonly used in message brokers and stream processing systems.
Rather than coordinating every single operation globally, the system only coordinates partition ownership. This significantly reduces synchronization overhead and improves scalability.
Partitioning is one of the most effective techniques for reducing the need for distributed coordination in large systems.
27. Sharding Reduces Shared Ownership
A similar idea applies to data storage systems.
Instead of storing all data in a single shared database, the data can be divided into shards.
For example, customer data might be partitioned alphabetically:
Customers A-M
|
v
Database A
-------------------
Customers N-Z
|
v
Database B
Each shard is responsible for a subset of the data and operates independently.
Most requests only interact with a single shard, which means there is no need for global coordination across the entire dataset. Applications only need to coordinate within the boundaries of a single shard, not across all shards.
By reducing the amount of shared ownership, sharding naturally reduces the need for distributed locking and other global coordination mechanisms.
28. Sometimes Coordination Isn't Necessary
Not all distributed problems require strict exclusivity or ownership. In some cases, concurrent operations can safely proceed without coordination at all.
For example, consider a distributed counter that is updated by multiple application instances.
Instead of preventing concurrent updates, the system can use a data structure designed to merge updates safely. In this model, multiple instances can update the counter at the same time without conflict.
The applications are not competing for control; they are contributing to a shared result. Some distributed data structures are specifically designed to support this kind of behavior.
Their goal is not to prevent concurrent updates, but to ensure that all updates eventually converge to a consistent final state. While the details of these structures are beyond the scope of this discussion, they highlight an important principle in distributed system design.
Coordination should only be introduced when it is truly required for correctness.
29. Choosing the Simplest Correct Solution
As distributed systems grow in complexity, a clear pattern emerges.
Different problems require different coordination strategies, and not all of them require distributed locking.
Using a distributed lock simply because multiple machines are involved often leads to unnecessary complexity and reduced scalability.
A better approach is to choose the simplest mechanism that still guarantees correctness.
| Engineering Problem | Recommended Approach | Why It's a Better Choice |
|---|---|---|
| Prevent duplicate customer registration | Database Unique Constraint | The database is the source of truth and can enforce uniqueness atomically without external coordination. |
| Concurrent updates to the same record | Optimistic Locking | Detects write conflicts efficiently while allowing high concurrency and minimal coordination. |
| Duplicate payment or API requests | Idempotency | Makes retries safe by ensuring repeated requests produce the same outcome. |
| One instance should execute scheduled jobs | Leader Election | Elects a single coordinator while allowing automatic failover if the leader becomes unavailable. |
| Exclusive access to a shared resource | Lease + Fencing Tokens | Provides temporary ownership and prevents stale owners from modifying shared state after lease expiration. |
| Maintain consistent cluster configuration or metadata | Consensus | Ensures all healthy nodes agree on a single, consistent view of shared state despite failures. |
| Process high-volume event streams or queues | Partition Ownership | Assigns exclusive ownership of partitions to consumers, eliminating the need for global locking. |
| Distribute ownership of large datasets | Sharding | Reduces shared ownership by partitioning data, minimizing the need for cross-node coordination. |
| Merge concurrent updates safely without coordination | Conflict-Free Data Structures (CRDTs) | Allows concurrent updates to converge automatically, avoiding locks for specific classes of problems. |
One important observation is that distributed locking appears only once in this table.
This is intentional.
Most real-world problems are better solved using mechanisms that are specifically designed for those problems, rather than relying on a general-purpose locking abstraction.
30. Common Misconceptions
There are several misconceptions that frequently appear when discussing distributed locking in system design.
One common belief is that distributed locks guarantee correctness. In reality, this is not always true. Correctness depends not only on acquiring the lock, but also on how the protected resource validates ownership. Without mechanisms like fencing tokens, stale lock holders may still be able to modify shared state incorrectly.
Another misconception is that lease expiration implies the original owner has stopped working. In practice, this is not guaranteed. The application may still be running but temporarily unable to renew the lease due to garbage collection pauses, CPU starvation, or network delays. Distributed systems must reason in terms of timeouts and uncertainty, not absolute state.
It is also often assumed that leader election replaces consensus. This is not the case. Leader election determines which node is responsible for coordination, while consensus determines what the system agrees upon. These are fundamentally different problems that operate at different layers of the system.
Finally, there is a belief that every distributed system requires distributed locking. In reality, many systems do not. Techniques such as unique constraints, optimistic concurrency, idempotent operations, and partition ownership often provide simpler and more scalable solutions.
Final Thoughts
We began this discussion with a simple question: how can multiple application instances ensure that only one of them performs a particular operation?
At first, distributed locking seems like a complete solution. It enforces exclusivity across machines, giving the impression that once a lock is acquired, all other contenders are safely excluded.
However, each approach introduces trade-offs that matter in real systems.
Permanent locks can become stale if the holder crashes, leaving the system blocked indefinitely.
Leases reduce this risk by introducing time-bounded ownership, but they also allow ownership to expire mid-operation, even when work is still valid. Even with leases, stale ownership can still cause incorrect writes unless the system can distinguish between current and outdated owners.
Fencing tokens address this by enforcing monotonic ordering, ensuring only the latest owner can modify shared state and preventing delayed operations from corrupting data.
At a higher level, leader election coordinates long-running responsibilities by ensuring a single active coordinator, while consensus protocols allow multiple nodes to agree on a single consistent state despite failures.
Together, these mechanisms highlight a core truth about distributed systems:
Distributed locking is not really about locking—it is about coordination under partial and unreliable information, where no node has a complete or up-to-date view of the system.
In practice, machines pause, networks delay or drop messages, and failures often look identical to transient slowness. This makes correctness fundamentally harder to reason about.
As a result, correctness is not achieved by removing uncertainty, but by designing systems that remain correct despite it.
The key lesson is simple:
Before introducing distributed locking, first ask whether the problem can be solved without adding coordination at all, since coordination is often the main source of complexity and fragility.
The most resilient distributed systems are not those with the most advanced locking mechanisms, but those that minimize coordination while still preserving correctness under all expected failure modes.
Assisted AI to paraphrase.
Top comments (0)