Distributed Locking & The 'Old Overwrites New' Problem (Fencing Tokens)

#chores

When designing Distributed Systems, we often use a Distributed Lock to ensure that at any given time, only one process is allowed to perform a specific task. For example: writing data to a shared file, or deducting money in a database.

However, there is a deadly trap. Even if you use the most sophisticated locking algorithms, your system can still break due to unintended data overwriting. Today, we will explore this problem and its solution: Fencing Tokens.

The Problem: The "Awakened" Node and Data Overwrites

Typically, when using a Distributed Lock, our code looks something like this:

// THIS CODE IS NOT NECESSARILY SAFE (UNSAFE)
function writeData(filename, data) {
    var lock = lockService.acquireLock(filename);
    if (!lock) {
        throw 'Failed to acquire lock';
    }
    try {
        var file = storage.readFile(filename);
        var updated = updateContents(file, data);
        storage.writeFile(filename, updated);
    } finally {
        lock.release();
    }
}

Although the logic seems airtight, in a distributed system environment, the code above has a critical flaw. Let's look at 2 common cases that cause errors:

Case 1: Node A pauses for too long

Node A acquires the lock but is paused (e.g., due to Garbage Collection, network issues...) until the lock's Time To Live (TTL) expires. At this point, Node B sees the lock is available, jumps in to acquire it, and successfully writes its data. Later, Node A suddenly "wakes up", continues its remaining execution, and overwrites the result on top of the data Node B just wrote. Result: Node B's data is unintentionally overwritten by Node A.

Illustration: Case 1 (Data Overwrite due to Pause)

Case 2: Intersection Time Conflict (Race Condition)

Even if Node A finishes processing before Node B (meaning Node B overwriting Node A would be logically correct sequentially), there is a problem. If Node A's TTL expires right as it is preparing to write, an "intersection" of time occurs between when Node A is finishing its operation (without the lock) and when Node B has just acquired the lock to start reading/processing. This resource overlap creates potential Race Condition bugs that are extremely hard to debug.

Illustration: Case 2 (Intersection Conflict)

As you can see, the root cause is that we are relying solely on the client to determine if it has permission to write. But in a distributed system, the time and state of the client are unreliable.

The Solution: Fencing Tokens

To completely solve this problem, Martin Kleppmann (author of Designing Data-Intensive Applications) emphasizes the importance of Fencing Tokens.

The idea here is: Every time a Node acquires a lock, the Lock Service doesn't just return "OK", it must also grant the Node a Token (a monotonically increasing number).

The process changes as follows:

Node A acquires the lock, receives Token = 33.
Node A is "frozen", lock expires.
Node B acquires the lock, receives Token = 34 (since the token number always increases).
Node B sends a write request along with Token: 34. The Storage Service records the highest current token as 34 and saves the data.
Node A"wakes up", sends a write request along with Token: 33.
The Storage Service checks: 33 is less than 34 (Token is old). Node A's request is REJECTED. Data is safe!

Solution Illustration (Fencing Token)

The Big Takeaway

Never completely trust the Client: Time (Clocks), Network Delays, or process interruptions (GC Pauses) can trick the client into thinking it's still holding the lock safely.
Protect from the Resource side (Storage): The destination of the data (Storage, Database, External API...) must have a "Gating Mechanism" to verify the validity of the request. Fencing Tokens are a perfect mechanism for this.
If you only need a Lock for Efficiency optimization (avoiding duplicate work), the lock occasionally slipping might be okay. But if you need a Lock to ensure data Correctness , lacking Fencing Tokens (like some simple locking algorithms) is a fatal flaw!
Core principle: When a task takes longer to process than the lock's TTL, that execution flow is inherently Unsafe. Always be careful with the actual execution time of a task!

Reference: The article How to do distributed locking by Martin Kleppmann.

DEV Community