15. Distributed Locking Is Only One Coordination Pattern
By this point, we've established an important progression in how distributed systems handle coordination.
A distributed lock is designed to solve a very specific problem: coordinating ownership of a resource across multiple machines. It ensures that only one node can act on a shared resource at a time.
As systems evolved, we introduced leases to handle a different failure mode—abandoned locks caused by crashed or unreachable nodes. Leases ensure that ownership is temporary and automatically expires if not renewed. We then added fencing tokens to address a more subtle issue: stale owners. Even if a node believes it still holds a lock, a higher-numbered token can prevent it from making unsafe updates.
This naturally leads to a broader question.
If distributed locking has become this sophisticated, why do modern distributed systems still rely on concepts such as leader election and consensus?
The answer lies in a key realization: exclusive ownership is not always the problem we're trying to solve.
In many systems, the goal is not to have multiple machines competing for the same resource. Instead, the goal is much simpler: ensure that one machine coordinates the rest.
That difference shifts the entire coordination model.
16. From Ownership to Leadership
Consider a cluster of application instances responsible for scheduling background jobs.
+-------------+
| Instance A |
+-------------+
+-------------+
| Instance B |
+-------------+
+-------------+
| Instance C |
+-------------+
Each instance runs the same code. Every minute, all of them check whether invoices should be generated.
If all instances execute the scheduler independently, duplicate invoices become inevitable.
A straightforward solution is to use a distributed lock. Before running the scheduler, each instance attempts to acquire the lock. The winner proceeds, and the others wait.
This works correctly, but it introduces a repeating pattern of contention.
Minute 1
Acquire Lock
↓
Generate Jobs
↓
Release Lock
-------------------
Minute 2
Acquire Lock
↓
Generate Jobs
↓
Release Lock
-------------------
Minute 3
Acquire Lock
↓
Generate Jobs
↓
Release Lock
Every execution cycle requires coordination. Every minute, the system competes again for the same ownership.
But the real requirement is simpler: only one instance should be responsible for scheduling. That responsibility does not change frequently.
Continuously competing for it introduces unnecessary overhead.
This is where leader election becomes a better fit.
- Leadership Is Long-Lived Ownership
Instead of competing for every operation, the cluster elects a single instance to act as the coordinator.
+-------------+
| Instance A |
+-------------+
│
│
Leader Elected
│
▼
+-------------+
| Leader |
| Instance A |
+-------------+
+-------------+
| Instance B |
+-------------+
+-------------+
| Instance C |
+-------------+
Once elected, the leader takes responsibility for shared tasks such as:
- scheduling background jobs
- coordinating cluster state
- assigning work to nodes
- monitoring cluster health
- managing shared configuration
The other instances continue serving requests, but they no longer compete for leadership unless a failure occurs.
This changes the coordination model fundamentally.
Instead of repeatedly asking:
"Who owns this operation right now?"
the system asks:
"Who is currently the leader of the cluster?"
Leadership becomes a long-lived role rather than a short-lived lock acquisition.
17. Leaders Can Fail Too
Leader election does not remove failure scenarios. It simply changes how the system responds to them.
Suppose Instance A is currently the leader.
Leader
Instance A
│
│
Coordinates Cluster
While performing its responsibilities, Instance A crashes unexpectedly.
At that moment, the cluster loses its coordinator. No other node is actively managing scheduling or coordination.
To recover, the system must elect a new leader.
This process is called leader election.
Before Failure
Leader
Instance A
---------------------
After Failure
Leader
Instance B
Unlike distributed locking, leader election is not triggered for every operation. It only occurs when leadership is lost.
Most of the time, the leader continues operating without interruption, making it a relatively stable coordination role.
- Leadership Is Also Temporary
A natural question arises at this point: how does the system know the leader has actually failed?
The honest answer is that it does not know with certainty.
The same uncertainty we saw earlier still applies.
The leader may have crashed.
The network may be partitioned.
The process may be paused.
The machine may be overloaded.
From the perspective of other nodes, all they can observe is that communication has stopped.
Because of this uncertainty, leader election relies on the same foundational mechanisms we have already discussed:
- temporary ownership
- heartbeats
- leases
- timeouts
The problem has not disappeared. It has simply shifted from protecting a resource to protecting the leadership role itself.
18. Why Leader Election Is Not Consensus
Engineers often encounter leader election and consensus together, but they solve different problems.
Leader election answers a narrow question:
Which node should coordinate the cluster?
Consensus answers a broader one:
How do multiple nodes agree on shared state despite failures?
Consider a cluster of three nodes maintaining configuration data.
Node A
Node B
Node C
Now imagine a configuration change is introduced. Every node must eventually agree on the same final state.
If one node applies the change while another rejects it, the cluster becomes inconsistent.
Leader election alone cannot solve this. It only determines who proposes or coordinates the change. It does not guarantee agreement.
Consensus is what ensures that all nodes converge on the same decision.
This distinction is subtle but critical.
Leadership assigns responsibility.
Consensus ensures agreement.
19. Coordination Requires Agreement
To make this more concrete, consider a distributed storage system with three nodes maintaining metadata.
The leader decides that a new storage node should join the cluster.
If the update is not applied consistently across all nodes, the system becomes fragmented.
Node A
Storage 1
Storage 2
Storage 3
------------------
Node B
Storage 1
Storage 2
Storage 3
------------------
Node C
Storage 1
Storage 2
Even though a leader exists, the cluster is now inconsistent. One node has a different view of membership than the others.
This demonstrates an important principle: leadership alone is not enough.
Consensus mechanisms exist to ensure that all healthy nodes eventually agree on the same state.
The exact algorithms behind this—such as Paxos or Raft—are complex and deserve their own discussion.
For now, the key architectural insight is:
Leader election determines who coordinates.
Consensus determines what everyone agrees on.
20. Where Modern Coordination Systems Fit
Modern distributed coordination systems often combine all of these concepts into a single platform.
They typically provide:
- temporary ownership through leases
- fencing via monotonic tokens
- leader election
- consensus mechanisms
- failure detection
- cluster membership management
Instead of requiring every application to implement these primitives independently, the coordination system provides them as shared infrastructure.
From an application perspective, the architecture becomes much simpler:
Application
│
▼
Coordination Platform
│
├── Leases
├── Leader Election
├── Cluster Membership
├── Consensus
└── Coordination Metadata
The application focuses on business logic, while the platform handles correctness under failure conditions.
An important observation here is that the specific technology—whether ZooKeeper, etcd, Consul, or another system—is often less important than understanding the underlying coordination problem.
Choosing a tool before understanding the problem often leads to unnecessary complexity.
21. Choosing the Right Coordination Primitive
Throughout this article, we have explored several coordination mechanisms. While they are often discussed together, each one solves a distinct problem.
Understanding these differences makes system design significantly clearer.
| Engineering Problem | Coordination Primitive | Primary Guarantee | Typical Use Cases |
|---|---|---|---|
| Ensure only one node performs a critical operation | Distributed Lock / Lease | Temporary exclusive ownership | Scheduled jobs, file processing, cache refresh, resource ownership |
| Prevent stale owners from modifying shared state | Fencing Tokens | Rejects operations from older owners | Inventory updates, payment processing, distributed storage, metadata updates |
| Ensure one active coordinator exists | Leader Election | Single active leader for cluster-wide coordination | Job schedulers, controllers, coordinators, cluster management |
| Keep multiple nodes in agreement | Consensus | Consistent cluster state despite failures | Cluster membership, configuration management, metadata, distributed coordination |
| Recover ownership automatically after failures | Leases | Ownership expires unless renewed | Long-running tasks, distributed locks, leadership management |
| Detect failed coordinators and trigger recovery | Heartbeats + Timeouts | Failure detection based on liveness | Leader monitoring, cluster health, node membership |
This table reflects the progression we have followed throughout the article.
We began with distributed locks for shared resource coordination.
We then addressed their limitations with leases.
We fixed stale ownership with fencing tokens.
We improved coordination efficiency with leader election.
Finally, we ensured correctness across nodes with consensus.
Each mechanism builds on the limitations of the previous one. None replaces the others. Instead, they form a layered toolkit for building reliable distributed systems.
The final question in this series naturally follows from here:
Do we actually need distributed locking at all?
In many real-world systems, correctness is achieved through alternative approaches such as optimistic concurrency, idempotent operations, database constraints, partition ownership, or queue-based processing. Understanding when to use these alternatives is often more valuable than implementing a distributed lock itself.
Top comments (0)