In this article, we'll explore the mechanisms to solve the coordination problem.
8. Introducing Leases
To address the problem of permanent ownership, distributed systems typically replace it with temporary ownership.
This concept is known as a lease.
Instead of granting indefinite control over a resource, the coordination service assigns ownership for a limited period of time.
Rather than stating, “You own this resource until you explicitly release it,” the system instead says, “You own this resource for the next 30 seconds.”
This changes the interaction model significantly.
Acquire Lease
|
v
Execute Work
|
v
Renew Lease
|
v
Continue Processing
As long as the application remains healthy, it periodically renews the lease to maintain ownership. If the application crashes or becomes unresponsive, it can no longer renew the lease. Once the lease duration expires, ownership is automatically revoked.
At that point, another application becomes eligible to acquire the lease and continue the work.
Leases solve a critical problem in distributed systems: they prevent abandoned locks from blocking progress indefinitely. The system can recover automatically without manual intervention.
However, while leases improve availability, they also introduce a new class of subtle and more complex problems.
- Leases Depend on Time
To understand the next challenge, assume the lease duration is thirty seconds.
Application A successfully acquires the lease.
Lease Granted
Duration = 30 seconds
After twenty seconds, the JVM begins a long Full Garbage Collection cycle. This pause lasts forty seconds, significantly longer than the lease duration.
The timeline now becomes problematic.
Lease Granted
|
|
Processing
|
|
GC Pause (40 sec)
|
|
Lease Expires
While Application A is paused, the lease expires. During this time, another application requests access to the same resource.
The coordination service observes that the previous lease has expired and therefore grants ownership to Application B.
Application A Application B
Lease Holder
|
GC Pause
Acquire Lease
|
Lease Granted
After the garbage collection completes, Application A resumes execution. From its perspective, it simply continues where it left off, still believing it owns the resource.
At the same time, Application B also believes it legitimately owns the resource because it was granted a valid lease by the coordination service.
Neither application is behaving incorrectly. Both are operating based on information that was valid at different points in time.
However, the system now has two active owners for the same resource.
This situation is significantly more dangerous than an abandoned lock because both participants are actively performing work under the assumption of exclusive ownership.
9. Split Brain Without a Network Partition
Many engineers associate split-brain scenarios exclusively with network partitions, where parts of the system become isolated from each other.
However, as the previous example demonstrates, split-brain conditions can occur even without a network failure.
Application A still believes it holds a valid lease, while Application B has legitimately acquired a newer lease from the coordination service.
Coordination Service
Lease Expired
|
+---------+---------+
| |
v v
Application A Application B
Believes Believes
It Owns It Owns
Both applications continue processing independently.
Importantly, the coordination service is behaving correctly. It expired the old lease and issued a new one based on its rules. Application A is also behaving correctly, because it has not yet observed that its lease has expired. Application B is also correct, because it received a valid lease.
The problem is not incorrect behavior by any single component. The problem is that ownership information has become stale and inconsistent across the system.
This leads to a crucial realization in distributed systems design: acquiring a lease does not guarantee continuous or permanent ownership. Ownership is not a static property; it can change while an application is temporarily unable to observe that change.
Leases solve one important problem by eliminating abandoned locks, but they introduce another, more subtle challenge.
How do we ensure that an application does not continue performing work after it has lost ownership?
Answering this question leads directly to one of the most important concepts in distributed coordination: fencing tokens.
10. Why Leases Alone Cannot Protect Your Data
The previous section ended with an uncomfortable situation.
Application A acquired a lease, but the JVM paused long enough for that lease to expire. During that time, Application B legitimately acquired a new lease. When Application A eventually resumed, both applications believed they owned the same resource.
This leads to a natural question: why doesn't the coordination service simply reject any further requests from Application A once its lease has expired?
The answer is simple, but important. The coordination service has no control over what Application A does after it acquires the lease. Once ownership is granted, the application runs independently and continues performing work on its own.
It cannot intercept every database update, file write, API call, or business operation that the application performs.
To make this concrete, consider a payment processing service:
Coordination Service
|
Lease Granted
|
v
Payment Service
|
v
Banking System
After the lease is granted, the payment service communicates directly with the banking system. At this point, the coordination service is no longer in the execution path.
If the lease expires while the payment service is paused, nothing prevents it from continuing to submit payment requests after it resumes. The coordination service may know the lease has expired, but the banking system does not.
This reveals a key insight: a distributed lock controls ownership, but it does not automatically control every operation performed by the owner.
- Ownership and Authority Are Different
To understand the problem more clearly, consider a warehouse system that reserves inventory.
Application A acquires a lease and begins processing. While it is still working, the lease expires. At that point, Application B becomes the new owner.
Lease Owner
Application A
↓
Lease Expires
↓
Application B
Now imagine Application A resumes execution. Even though it is no longer the owner, it still holds references to in-memory objects, still has an open database connection, and still has network access.
Nothing physically prevents it from continuing to update inventory.
The coordination service cannot reach back in time and undo or block operations that are already in progress.
So while ownership has changed, the authority to modify the shared resource has not been automatically revoked in the running application.
This is why leases alone cannot guarantee correctness in distributed systems.
11. The Missing Piece
Let’s revisit the inventory example in more detail.
The inventory contains a single remaining item. Application A acquires a lease and begins processing. A few seconds later, it pauses unexpectedly. During this pause, the lease expires.
Application B then acquires a new lease and successfully reserves the inventory item.
Several seconds later, Application A resumes and also attempts to reserve the same item.
At this point, the inventory becomes inconsistent.
What is important here is that neither application violated the lease protocol. The coordination service behaved correctly. Both applications followed the rules as designed.
The problem is that the shared resource had no way to distinguish between the current owner and a previous owner that resumed late.
Both requests looked valid.
The missing capability is not another lock or another lease mechanism. The missing capability is a way to determine which owner is newer.
This is exactly the problem that fencing tokens solve.
12. Introducing Fencing Tokens
Instead of only granting a lease, the coordination service also issues a monotonically increasing number along with it.
This number is called a fencing token.
Each time a new lease is granted, the token increases. Every new owner receives a strictly larger value than the previous one.
The sequence might look like this:
Application A
Lease Granted
Token = 101
--------------------
Application B
Lease Granted
Token = 102
--------------------
Application C
Lease Granted
Token = 103
The fencing token represents the freshness of ownership. A larger token always means a more recent owner.
Unlike timestamps, fencing tokens do not depend on clock synchronization. They are simply increasing numbers generated by the coordination system.
This small addition fundamentally changes how safety is enforced in distributed systems.
- Why Increasing Numbers Matter
Now consider what happens when Application A pauses after receiving token 101. While it is paused, Application B acquires a lease and receives token 102.
Time
Application A
Acquire Lease
Token 101
|
|
GC Pause
|
|
Resume
-------------------------
Application B
Acquire Lease
Token 102
Both applications continue running independently.
Without fencing tokens, both would appear equally valid when they attempt to modify the shared resource.
With fencing tokens, every operation now carries proof of ownership.
Application A sends a request like this:
Reserve Inventory
Token = 101
Application B sends:
Reserve Inventory
Token = 102
The resource can now immediately determine which request belongs to the most recent owner.
13. The Resource Must Enforce the Token
This is the point where many explanations stop, and also where many real-world systems fail.
The coordination service generates fencing tokens, but it does not enforce them. The responsibility of validation belongs entirely to the protected resource.
For example, imagine the inventory database keeps track of the highest token it has seen so far.
Initially, the state might be:
Highest Token = 101
When Application B submits its request with token 102, the database compares it with the stored value. Since 102 is greater, the request is accepted, and the state is updated:
Highest Token = 102
Later, Application A resumes and submits a request with token 101. The database performs the same comparison:
101 < 102
Since the incoming token is older than the current highest token, the request is rejected.
This prevents the stale owner from modifying the shared resource.
The key insight is that correctness is enforced by the resource itself, not by the coordination service and not by the lease mechanism.
- Why the Coordination Service Cannot Do This
At first glance, it might seem simpler if the coordination service itself rejected stale operations.
However, this is not possible in practice.
The coordination service only participates during lease acquisition. After that point, all business operations happen directly between the application and the resource.
Consider this flow:
Application
|
Acquire Lease
|
Coordination Service
Application
|
|
v
Database
Once the lease is granted, every subsequent operation bypasses the coordination service entirely. Because of this, only the database (or the protected resource) can determine whether a request is stale.
This is why fencing tokens must accompany every operation that modifies shared state.
14. Where Fencing Tokens Are Useful
Fencing tokens are useful anywhere multiple distributed processes can modify a shared resource.
Common examples include:
- updating inventory systems
- processing payment batches
- writing to shared storage systems
- updating distributed metadata stores
- controlling scheduled or background jobs
- modifying cluster configuration or leadership state
The underlying technology does not matter. What matters is that multiple actors can attempt to modify the same resource, and correctness depends on ensuring only the newest owner succeeds.
Whenever a system must distinguish between stale and current ownership, fencing tokens provide a reliable solution.
- Fencing Tokens Are Not a Replacement for Leases
It is important to understand that leases and fencing tokens solve different problems.
Leases answer the question:
"Who currently owns the resource?"
Fencing tokens answer a different question:
"Is this request coming from the most recent owner?"
A system that uses leases without fencing tokens can still suffer from stale writes. A system that uses fencing tokens without leases has no mechanism for transferring ownership in the first place.
In practice, robust distributed systems use both together.
The lease establishes temporary ownership.
The fencing token allows the resource to verify that ownership before accepting any operation.
This separation of concerns is a key idea behind production systems like ZooKeeper and etcd, which must remain correct even in the presence of failures, pauses, and network partitions.
Top comments (0)