Distributed systems solve many problems by dividing work across multiple machines. The same characteristic also introduces an entirely new class of problems.
Machines must coordinate.
Unlike threads running inside a single JVM, independent services have no shared memory, no common execution context, and no built-in synchronization mechanism. Every instance makes decisions based on the information available to it at that moment. When several instances attempt to perform the same business operation simultaneously, coordination becomes significantly more difficult than it first appears.
Distributed locking is often introduced as the solution to this problem.
It certainly plays an important role.
It is also one of the most misunderstood concepts in distributed systems.
Many discussions focus on implementing distributed locks using a particular technology. Much less attention is given to understanding what distributed locking actually guarantees, what it cannot guarantee, and why many production systems solve coordination problems without using distributed locks at all.
This article explores distributed locking from an engineering perspective.
Rather than starting with technologies, we'll begin with the coordination problem itself.
As the discussion progresses, each solution will naturally expose its own limitations. Those limitations lead us toward leases, split-brain scenarios, fencing tokens, leader election, consensus, and finally the situations where a distributed lock is not the right solution.
Because distributed locking is ultimately not about acquiring a lock.
It is about keeping distributed systems correct while machines fail, networks become unreliable, and time itself becomes uncertain.
1. The Coordination Problem
Imagine an e-commerce platform during a flash sale. Only one unit of a limited-edition product remains in inventory. Two customers place an order at nearly the same time.
The application is deployed across multiple instances.
Inventory = 1
+----------------------+
| |
v v
Application A Application B
| |
| |
Reserve Item Reserve Item
Both application instances receive the request almost simultaneously.
Both read the inventory.
Both conclude that one unit is available.
Both attempt to reserve it.
The inventory now becomes negative.
No service crashed.
No exception occurred.
The database remained healthy.
Every component behaved exactly as designed.
The failure was not caused by incorrect code. It was caused by the absence of coordination.
Both applications made independent decisions about the same business resource.
This problem appears in many business systems.
Examples include processing the same payment twice, generating duplicate invoices, executing the same scheduled job on multiple servers, refreshing the same cache simultaneously, and assigning the same delivery request to multiple drivers.
Although these scenarios appear unrelated, they all share the same underlying problem.
Multiple independent machines need to agree that only one of them should perform a particular operation. That is coordination problem.
- Coordination Is Different From Concurrency
Concurrency is a familiar concept for most Java developers.
Two threads execute simultaneously inside the same JVM. They share memory, they share execution context, and they rely on built-in synchronization primitives provided by the language and runtime. Because of this shared environment, coordination is relatively straightforward: threads can directly block, signal, and enforce mutual exclusion using well-defined mechanisms.
Distributed systems operate under entirely different conditions.
Application instances do not share memory. They cannot directly observe each other's state. Communication happens over a network, which introduces latency, partial failures, and uncertainty. Messages may be delayed, reordered, or lost. Machines may fail independently without affecting others. As a result, every instance has only a partial and potentially outdated view of the overall system.
Imagine three application instances running in different environments.
+-------------+ +-------------+ +-------------+
| Application | | Application | | Application |
| A | | B | | C |
+-------------+ +-------------+ +-------------+
Each instance executes independently, making decisions based solely on local information. None of them can infer the state of the others unless explicit communication occurs. This lack of shared state fundamentally changes how synchronization must be designed in distributed systems.
2. Why Traditional Locks Stop Working
Most Java developers are comfortable solving concurrency problems using synchronization primitives such as synchronized blocks or explicit locks.
Consider a simple example of protecting a critical section in a single JVM.
public class InventoryService {
private final Object lock = new Object();
public void reserve(String productId) {
synchronized (lock) {
// Reserve inventory
}
}
}
Inside a single JVM, this approach works exactly as expected. Only one thread can enter the critical section at a time, and the JVM enforces mutual exclusion reliably. The lock effectively serializes access to shared in-memory state.
The problem becomes apparent when the same application is deployed across multiple instances.
Suppose the service runs on two different servers.
JVM A JVM B
synchronized(lock) synchronized(lock)
Each JVM maintains its own memory space. Each instance creates its own independent lock object. There is no shared state between them, and neither JVM is aware of the other’s existence. As a result, both threads can enter their respective synchronized blocks simultaneously without any coordination.
The synchronization mechanism has not failed. It has simply been applied outside its intended scope. It only guarantees mutual exclusion within a single process, not across multiple processes.
The same limitation applies to other in-memory primitives such as ReentrantLock, Semaphore, and ReadWriteLock. All of them assume a shared memory model, which distributed systems fundamentally do not provide.
- What About Database Locks?
A natural question arises at this point: if multiple application instances share the same database, can the database itself act as a coordination mechanism?
The answer is nuanced.
In some cases, it works well. In others, it is appropriate but limited. In many cases, it is not suitable at all.
Consider a scenario where two applications attempt to update the same customer record. A relational database can enforce row-level locking, ensuring that only one transaction modifies the row at a time.
Application A
|
Row Locked
|
Application B Waits
This works effectively because the resource being protected—the database row—already exists within the database’s control domain. The database is naturally responsible for managing consistency of its own data, so using its locking mechanism aligns with its design.
Now consider a different scenario: a scheduled reconciliation job that must run only once per night. The job performs multiple external operations such as downloading files, calling third-party services, generating reports, and uploading results.
None of these operations map cleanly to a single database row. There is no single database entity that represents the job’s execution state in a way that the database can meaningfully lock. The resource requiring coordination exists outside the database’s transactional boundary.
This highlights an important distinction. Database locks are highly effective for protecting database state, but they are not a general-purpose coordination mechanism for distributed workflows that span multiple systems and external dependencies.
Recognizing this boundary prevents overusing database locking as a distributed coordination strategy.
3. What Problem Are We Actually Solving?
At this point, it is easy to assume that the solution is simply a lock that works across machines. However, this intuition only partially captures the real problem.
The actual goal is not the lock itself. The goal is ensuring that only one application instance performs a specific business responsibility at a given time.
This distinction is subtle but critical.
Consider an application responsible for generating monthly customer statements. If three instances of the application start simultaneously, each instance may attempt to perform the same job independently.
Instance A
|
|
Generate Statements
Instance B
|
|
Generate Statements
Instance C
|
|
Generate Statements
If all three proceed, the system produces duplicate work and potentially inconsistent results. However, the business requirement is not about locking—it is about ensuring exclusivity of execution.
Only one instance should perform the task.
This requirement is fundamentally about ownership rather than synchronization. The system needs a way to assign temporary responsibility for a task to a single participant.
This pattern appears repeatedly in distributed systems. One instance may need to refresh a shared cache, process a payment batch, act as a leader, execute scheduled tasks, or migrate shared data. In all cases, the requirement is consistent: a single owner must perform a specific responsibility for a defined period of time.
Distributed locking is one possible mechanism to achieve this outcome, but it is not the outcome itself.
4. The First Distributed Lock
To address coordination across multiple machines, a shared coordination service is often introduced. Before performing a critical operation, each application instance requests permission from this service.
The interaction typically follows a simple pattern.
Application
|
Acquire Lock
|
v
Coordination Service
|
Lock Granted
|
v
Execute Work
|
Release Lock
From a conceptual standpoint, this resembles traditional synchronization. One participant acquires exclusive access, others wait or retry, and only a single instance proceeds with execution.
At first glance, this appears to solve the problem effectively. The coordination service replaces shared memory, and mutual exclusion is enforced across distributed instances. The business requirement of single execution is satisfied.
If distributed systems were perfectly reliable, this model would be sufficient and straightforward.
However, real-world systems introduce complexity that does not exist in single-process environments. Time becomes unreliable. Machines can fail unexpectedly. Network communication can be delayed or interrupted. Processes may pause due to garbage collection or crash entirely.
These factors introduce uncertainty into the system. A lock that was granted moments ago may no longer represent a valid or safe assumption about system state.
As a result, the problem evolves.
It is no longer sufficient to ask, “Who holds the lock?”
The more important question becomes, “Is the current lock holder still alive and capable of safely completing the work?”
This shift in perspective fundamentally changes how distributed locking must be designed and evaluated. It also explains why acquiring a distributed lock is only the beginning of the problem, not its solution.
5. Failure Changes Everything
At the end of the previous section, we arrived at a seemingly simple solution.
Every application instance requests a distributed lock before performing a critical operation.
Only one instance receives the lock, while all other instances are forced to wait until it becomes available again.
From a business perspective, this appears to satisfy the requirement because only one application performs the work at any given time.
If distributed systems behaved like a single JVM, this would be sufficient. In a single process, synchronization primitives provide strong guarantees about mutual exclusion and ownership.
However, distributed systems do not behave like a single JVM.
The difficulty begins the moment we acknowledge a simple reality: machines fail.
More importantly, machines sometimes appear to fail even when they are still running correctly. This distinction fundamentally changes how distributed coordination must be designed.
- When a Machine Stops Responding
Let’s assume an application successfully acquires a distributed lock.
Application A
|
Acquire Lock
|
Lock Granted
|
Processing...
At this point, everything appears normal. The system has successfully established exclusive access, and the application begins performing its critical work.
A few seconds later, the application stops responding.
This immediately raises an important question: should another application be allowed to acquire the lock?
At first glance, the answer seems obvious. If the application is not responding, it feels reasonable that another instance should take over and continue the work.
However, distributed systems rarely provide clarity about why a machine stopped responding. There are many possible explanations, and from the outside, they often look identical.
The application may have crashed unexpectedly due to a runtime error or fatal exception. The operating system itself may have restarted the machine as part of maintenance or recovery. The network connection may have been interrupted, isolating the machine even though it is still running. The machine may simply be overloaded, causing extreme delays in processing requests. In some cases, the process may still be alive and functioning correctly, but temporarily unable to communicate with the rest of the system.
From the perspective of other machines, all of these scenarios produce the same observable behavior: the application becomes silent.
This silence is deceptively simple, yet extremely difficult to interpret correctly in a distributed environment.
- Failure Is Often Uncertainty
One of the most important differences between concurrent programming and distributed systems is that failures are rarely definitive in distributed environments.
Inside a single JVM, the behavior is clear and deterministic. A thread either holds a lock or it does not. A method either completes successfully or it throws an exception. The system has complete visibility into execution state.
Distributed systems, however, operate with incomplete and delayed information.
Imagine three application instances interacting in a network.
+-------------+
| Instance A |
+-------------+
|
Communication Lost
X
+-------------+-------------+
| |
v v
+-------------+ +-------------+
| Instance B | | Instance C |
+-------------+ +-------------+
In this scenario, Instances B and C stop receiving messages from Instance A. From their perspective, communication has simply ceased.
There are several possible explanations for this condition. Instance A may have crashed entirely. The network between the instances may have failed or become partitioned. Instance A may still be running but temporarily paused or overloaded. Alternatively, messages may simply be delayed in transit due to congestion or resource constraints.
The key challenge is that the remaining instances cannot immediately distinguish between these possibilities. They only observe the absence of communication, not its cause.
This uncertainty is not an edge case in distributed systems; it is the default operating condition. As a result, engineering decisions must often be made without complete or reliable information.
6. Time Introduces New Problems
Now consider a different scenario. Application A successfully acquires a distributed lock and begins processing a large batch of financial transactions.
Initially, everything proceeds as expected. The system is stable, and the application is actively performing work.
However, after a short period, the JVM triggers a Full Garbage Collection cycle. During this process, the application is paused while memory is being reclaimed and reorganized.
Acquire Lock
|
v
Begin Processing
|
v
Full GC Pause
|
|
|
v
Resume Processing
From the perspective of the application itself, nothing unusual has occurred. Execution has simply been paused temporarily and will resume once garbage collection completes.
From the perspective of the rest of the system, however, the situation looks very different. During the pause, the application is unable to respond to requests, send heartbeats, or communicate its state. For several seconds, it appears completely unresponsive.
Importantly, the application is not dead. It is still running. It is simply not making progress.
Distributed systems struggle with this distinction because external observers cannot easily determine whether a system is paused or permanently failed.
- A Pause Can Look Like a Failure
This ambiguity often surprises engineers who are new to distributed systems.
A useful analogy is waiting for a colleague to respond to an important message. If several minutes pass without a reply, multiple interpretations become possible. The colleague may be busy with other work. Their phone may be turned off or disconnected. They may have lost internet connectivity. In the worst case, they may no longer be available at all.
Without additional context, all of these explanations remain equally plausible.
Distributed systems face the same problem at scale. A temporary pause caused by garbage collection, heavy CPU contention, thread starvation, or resource exhaustion can appear identical to a complete system failure.
Because of this ambiguity, distributed systems cannot rely on perfect failure detection. There is no reliable mechanism that can always distinguish between a slow system and a dead one.
Instead, systems must make decisions based on incomplete and sometimes misleading information.
7. Locks Cannot Last Forever
To understand why distributed locking is fundamentally different from local locking, consider what would happen if a coordination service granted locks indefinitely.
In that model, the interaction would look familiar:
Acquire Lock
|
v
Work
|
v
Release Lock
This mirrors traditional synchronization inside a single JVM, where a thread holds a lock until it explicitly releases it.
Now revisit a failure scenario. Application A acquires the lock and begins processing. Shortly afterward, the machine hosting the application crashes unexpectedly.
Application A
|
Acquire Lock
|
Machine Crash
Because the application crashes, it never reaches the step where it releases the lock.
Release Lock
As a result, the lock remains permanently held. No other application can acquire it, and every other instance is forced to wait indefinitely.
The system effectively stops making progress.
This illustrates a critical problem: in distributed systems, permanent ownership is dangerous because failures are not rare exceptions—they are expected events.
To maintain availability, the system must be able to recover from abandoned locks automatically.
Assisted AI to paraphrase.
Top comments (0)