DEV Community

Cover image for Read-Modify-Write isolation in NoSQL: the distributed-lock hell.
Hugo Vantighem
Hugo Vantighem

Posted on

Read-Modify-Write isolation in NoSQL: the distributed-lock hell.

In part 1, the single-document case was easy. In part 2, two documents brought Write Skew, and we saw that even a native ACID transaction — snapshot isolation — lets it through.

So teams reach for the reflex fix: a distributed lock — Redis-based, often a Redlock-style implementation. Acquire a lock on a key, do your Read → Modify → Write, release. On paper, you've finally serialized the critical section — operationally, at least. In practice, you've stepped on three mines.

1. Network latency

Every guarded transaction now makes extra round-trips to Redis — before and after hitting your NoSQL store.

You've doubled your coordination surface and taken a hard dependency on a second system being up, reachable, and fast on the hot path of every write.

The "fast" database is now gated by the lock service. And the coupling bites harder than the average latency suggests: every Redis tail-latency spike becomes your write-latency spike — your p99 inherits Redis's p99 — and if Redis fails over mid-transaction, the lock you think you're holding can effectively vanish on the new primary, dropping you straight into the corruption case below.

2. Deadlock

You can dodge deadlock entirely with a single coarse lock — but then every writer serializes on it, and you've thrown away the very concurrency you reached for NoSQL to get.

So to keep throughput you go fine-grained, one lock per resource — and the moment an invariant touches more than one key (across this series, it always does), deadlock is back on the table:

  • Transaction A locks key X, then needs Y.
  • Transaction B locks Y, then needs X.
  • Both block until timeout or intervention.

The textbook cure — real deadlock detection, maintaining a wait-for graph across every lock holder and breaking cycles as they form — is a distributed-systems project in its own right: not something you bolt onto a cache you reached for precisely to save engineering time.

So nobody builds it.

Instead teams impose a standing discipline: always acquire locks in the same canonical order — sort the keys by document ID, lock them in that order, in every code path, every time, forever.

One writer that grabs them out of order — a refactor, a new feature, a teammate who never got the memo — and the deadlock is back. That convention, plus the hopeful timeouts bolted on for when it slips, is the exact accidental complexity you adopted NoSQL to escape.

The cognitive load you tried to delete just came back wearing a Redis sticker.

3. The TTL dilemma

A distributed lock needs a TTL, or a crashed holder blocks everyone forever.

But the moment you set one, you've broken the one guarantee that mattered: you lose any coordination between the lease and the real execution time.

The lock expires on a wall clock; your transaction finishes whenever it finishes — GC pause, slow disk, a network hiccup — and those two clocks have no idea the other exists.

So there is no safe value:

  • TTL too short → the lock expires mid-transaction. A second writer walks in, and you get exactly the corruption the lock was supposed to prevent — now with a false sense of safety.
  • TTL too long → a crashed pod blocks the resource until expiry, and your throughput tanks under the very contention you were trying to handle.

No single value is correct across all loads — because correctness constraints and throughput constraints are in direct tension, and the TTL is the single knob you're forced to trade them off against.

This is the heart of the well-known Kleppmann-vs-Redlock debate: a lock that can expire on its own is not a sound mutual-exclusion primitive when you need it for correctness, only for efficiency.

Using it to protect an invariant means betting your data integrity on a timeout.

So are we doomed?

Are we really condemned to choose between the latency and deadlocks of locks, and the silent corruption of going without?

No. There is a fourth path — still within NoSQL, with no distributed lock and no external coordination service — that delivers strict isolation against inter-document Write Skew.

An approach where either everything commits, or everything rolls back, with the invariant intact. It doesn't make the contention disappear — nothing does — it just stops bolting a Redis layer on top to manage it, and collapses the coordination back into the database itself.

That trade is the whole point, and I'll lay it out honestly.

I laid out the full mechanics — the pure-NoSQL pattern almost nobody reaches for — in the next article.

Top comments (0)