DEV Community: Muhammad Ahsan Farooq

LSM-Trees Explained: The Storage Engine Behind Millions of Writes per Second

Muhammad Ahsan Farooq — Sun, 11 Oct 2026 07:58:41 +0000

MemTables, write-ahead logs, SSTables, Bloom filters, and compaction, from the first write to the last merge

Most databases are designed around reading. Then you build a chat app, an IoT platform or an event log, and the job flips. The database spends its life absorbing writes, millions per second.

Classic relational engines, such as InnoDB in MySQL, are built around B+ trees that update fixed-size disk pages in place. Scattered updates mean random disk I/O, and under heavy write load that becomes the bottleneck.

Log-structured merge-trees (LSM-trees) take the opposite approach: never modify anything on disk. Buffer writes in memory, record them in a log, flush them as sorted immutable files, and tidy up later in the background. This design powers Apache Cassandra, ScyllaDB, HBase, Google Bigtable, LevelDB and RocksDB.

This post explains how LSM-trees work from the inside, with a worked example and diagrams for each step.

In this post:

Why B+ trees struggle with writes, and the idea behind LSM-trees
The parts: MemTable, write-ahead log, SSTables, sparse indexes and Bloom filters
A worked example, following real writes from memory to disk
Reads, deletes and tombstones
Compaction, and the read, write and space trade-offs it creates
Use cases, case studies and tuning advice

Why B+ trees struggle with heavy writes

A B+ tree keeps data sorted inside fixed-size pages, commonly 4 to 16 KB. To change a record, the database finds the right page, reads it into memory, modifies it, and writes the page back to the same place. If a page is full, it splits in two and the tree rebalances.

That works well when updates are local. It works badly when keys arrive in random order, because every write lands on a different page. The disk jumps around, and each small change costs a full page read and write.

in-place page updates vs sequential flush · 3 writes

Sequential I/O is much cheaper than random I/O. The gap is largest on spinning disks, but SSDs also handle large sequential writes more efficiently than many small scattered ones.

The LSM-tree idea

An LSM-tree turns every write into an append. It does four things:

Accept the write in memory, in a sorted buffer.
Record it in a log on disk, so a crash cannot lose it.
Flush the buffer to disk as one sorted, immutable file when it fills up.
Merge files in the background so reads stay fast.

An analogy: the library cart

Imagine a library that receives thousands of new books a day. A strict librarian would walk each book to its exact shelf spot, shifting other books to make room. That is a B+ tree.

An LSM librarian drops new books on a cart. When the cart is full, they sort it once and seal it into a labeled box. Boxes are never reopened. Every so often, a few boxes are merged into one bigger sorted box. Adding a book is almost free. Finding one means checking the cart and then the boxes, newest first. That is the trade you accept with an LSM-tree: cheap writes, more work on reads and in the background.

Anatomy of an LSM-tree

An LSM-tree has a few cooperating parts spread across memory and disk. Follow one write through them, from left to right:

LSM-tree write path · 5 stages across memory and disk

1. The write-ahead log (WAL)

RAM is volatile, so a MemTable alone would lose data in a crash. Before a write is acknowledged, it is appended to an on-disk write-ahead log. The log is strictly append-only, so it uses fast sequential I/O. After a crash, the engine replays the log to rebuild the MemTable.

How strong the guarantee is depends on the sync setting. If the engine waits for the log to reach the disk, the write is durable. Some systems default to syncing on a timer instead. Cassandra's commit log, for example, defaults to periodic sync, and relies on replication to cover a single node's crash.

2. The MemTable

The MemTable is an in-memory buffer that keeps keys in sorted order. Engines typically build it from a skip list or a balanced tree. Writes touch only RAM, so they finish almost instantly. RocksDB, for instance, defaults to a 64 MB write buffer.

3. SSTables

When the MemTable reaches its size limit, the engine freezes it into a read-only MemTable and starts a fresh active one for incoming writes. A background thread then writes the frozen table to disk as a Sorted String Table (SSTable).

An SSTable has two defining properties:

Sorted: keys are stored in order, which makes lookups and merges efficient.
Immutable: once written, the file is never modified. An update is a newer record in a newer SSTable. A delete is a special marker called a tombstone.

SSTable anatomy · 3 data blocks, 2 helpers

4. The sparse index and Bloom filters

Over time, many SSTables pile up. A lookup could in theory open every one of them. Two small in-memory structures prevent that:

Sparse index: because keys inside a file are sorted, the engine only stores the first key of each block, for example every 4 KB. To find a key, it binary-searches this index and reads just one block.
Bloom filter: a compact probabilistic structure that answers "is this key in the file?" with either definitely not or maybe. It never gives a false negative, so a definitely not lets the engine skip the file without touching the disk. It can give false positives, but at about 10 bits per key the rate is around 1%.

A worked example: eight writes

Let's trace a tiny key-value store that maps a user name to a city. To keep the example small, assume the MemTable flushes after every 4 writes. A real engine flushes after tens of megabytes.

Step	Operation	MemTable afterwards	On disk afterwards
1	`put(alice, Lahore)`	alice=Lahore	nothing yet
2	`put(bob, Karachi)`	alice=Lahore, bob=Karachi	nothing yet
3	`put(alice, Dubai)`	alice=Dubai, bob=Karachi	nothing yet
4	`put(carol, Berlin)`, then the MemTable is full and flushes	empty	SSTable 1: alice=Dubai, bob=Karachi, carol=Berlin
5	`put(bob, Oslo)`	bob=Oslo	SSTable 1
6	`delete(carol)`	bob=Oslo, carol=TOMBSTONE	SSTable 1
7	`put(dave, Paris)`	bob=Oslo, carol=TOMBSTONE, dave=Paris	SSTable 1
8	`put(erin, Rome)`, then the MemTable is full and flushes	empty	SSTable 1, plus SSTable 2: bob=Oslo, carol=TOMBSTONE, dave=Paris, erin=Rome

Four things are worth noticing:

Overwrites in memory are free. In step 3, Lahore was replaced inside the MemTable. It never reached the disk.
A flush is one sequential write. Each SSTable was produced in a single pass, already sorted.
The same key can live in several files. bob is in both SSTables, with different values. The newest one wins.
A delete is a write. In step 6, nothing was erased. carol=Berlin still sits in SSTable 1, and a tombstone in a newer place now hides it.

All eight writes were also appended to the write-ahead log. Once a MemTable is safely flushed, the log segment that covered it can be discarded.

This is the state the next section starts from.

The read path and tombstones

Reading from an LSM-tree means searching from the newest data to the oldest and stopping at the first answer:

Check the active MemTable, then any frozen MemTables.
For each SSTable, newest first: ask the Bloom filter, then the sparse index, then read one data block.
Return the newest version found. If the newest record is a tombstone, the key does not exist.

Here are three lookups against the store from the worked example:

read path · 3 lookups across 3 layers

Notice that get(alice) needed one disk read, even though two files exist. The Bloom filter ruled out SSTable 2 for free.

Why reads cost more

A key might need to be checked in several files. This extra work is called read amplification. Bloom filters, block caches and compaction all exist to keep it small. Compaction matters most, because it reduces the number of files a read can touch.

Deletes and tombstones

SSTables are immutable, so a delete cannot erase anything. The engine writes a tombstone, a marker saying "this key is deleted as of now". Reads treat a newest-record tombstone as a missing key. The tombstone and the data it hides are physically removed later, during compaction.

Tombstones cannot always be dropped right away. Cassandra, for example, keeps them for gc_grace_seconds, which defaults to 10 days, so that a replica that was offline during the delete still learns about it. Drop the tombstone too early and the deleted data can come back.

This makes delete-heavy workloads risky. Using a table as a queue, where rows are inserted and then deleted, leaves piles of tombstones that every read has to scan past. Discord ran into a version of this: when it migrated its message store, the migration stalled at 99.9999% because the last token ranges held huge runs of tombstones that had never been compacted away.

Compaction: keeping the files in check

Without cleanup, SSTables accumulate forever. Reads must check more files, and old versions of keys and tombstones waste disk space. Compaction is the background process that fixes this. It reads several SSTables, merge-sorts them, discards overwritten values and deleted data, and writes new consolidated files.

Here is compaction applied to the two files from our worked example:

compaction merge · 2 files into 1

Because both inputs are already sorted, the merge is a single streaming pass over the disk. It is sequential I/O again, which is exactly what the design wants.

A tombstone can be dropped only when it is safe, meaning the compaction includes every older file that could still hold the deleted key. Otherwise the old value would reappear.

Two main strategies

Engines differ in which files they merge and when. The two classic strategies make opposite trade-offs:

size-tiered vs leveled compaction

Strategy	How it works	Strengths	Costs	Fits best
Size-tiered (STCS)	Groups files of similar size. When enough exist, merges them into one larger file	Low write amplification and fast ingestion	High space amplification. Large merges can need up to 50% free disk	Write-heavy workloads such as event logs and metrics
Leveled (LCS)	Stores data in levels. Each level is about 10x larger than the last, and files do not overlap above L0	Low read amplification and good space efficiency, with roughly 10% overhead	High write amplification and steady background I/O	Read-heavy or space-constrained workloads

In the leveled diagram, a read checks at most one file in each of L1 and below, because those files do not overlap. It must still check every L0 file, since fresh files can overlap.

Time-series data has its own option. Cassandra's time-window compaction groups files by time window, so when data carries a time-to-live, whole windows can be dropped cheaply once they expire.

The three amplifications

Compaction is a balancing act between three costs. Every tuning decision moves weight from one to another:

Amplification	What it measures	What drives it up
Read	How many files or blocks a read must check	Many overlapping files
Write	Bytes written to disk for each byte the application wrote	Compacting often, since every merge rewrites data
Space	Disk used compared with the size of the live data	Old versions and tombstones waiting to be merged away

Size-tiered compaction favors low write amplification. Leveled compaction favors low read and space amplification. No strategy wins on all three.

Compaction is not free

Compaction competes with your queries for disk and CPU. If it falls behind, reads touch more files and latency climbs. Discord described this problem in its Cassandra clusters: nodes fell behind on compactions, which made reads more expensive and caused cascading latency. Watching the compaction backlog is as important as watching query latency.

Where LSM-trees shine, and where they do not

LSM-trees fit workloads that write constantly, read mostly recent data, and rarely update or delete in place.

Use cases

Chat and message history: messages are appended per channel and read mostly newest first. Discord stores its messages this way.
IoT telemetry and metrics: millions of sensor readings per second, each keyed by device and time.
Event logs, audit trails and activity feeds: append-mostly data that is queried by key and time range.
User activity history: Netflix described its viewing-history store as write-heavy, with a write-to-read ratio of about 9:1.
Embedded state stores: RocksDB sits inside stream processors such as Kafka Streams and Apache Flink to hold large local state.
Foundations of other databases: CockroachDB, TiKV and YugabyteDB build on LSM-tree engines, and MyRocks brings RocksDB to MySQL.

Example: sensor readings with automatic expiry

A time-series table in Cassandra pairs the write path with a compaction strategy built for time data:

CREATE TABLE sensor_readings (
  sensor_id text,
  day       date,
  ts        timestamp,
  value     double,
  PRIMARY KEY ((sensor_id, day), ts)
) WITH compaction = {
    'class': 'TimeWindowCompactionStrategy',
    'compaction_window_unit': 'DAYS',
    'compaction_window_size': 1
  }
  AND default_time_to_live = 2592000;  -- 30 days

Here is why this fits:

Appends only. Every reading is a new row, so there are no in-place updates.
Bounded partitions. The (sensor_id, day) partition key caps how large any one partition grows.
Cheap expiry. Files are grouped by day, so data older than 30 days disappears when a whole file expires, instead of through millions of individual deletes.

When an LSM-tree is the wrong tool

The costs we covered earlier define the limits:

Read-heavy random lookups that a cache cannot absorb. A B+ tree reads one predictable path.
Strict, flat tail latency. Compaction competes with queries and can cause spikes.
Delete-heavy or queue-like tables. Tombstones pile up, and reads must scan past them.
Tight disk budgets with size-tiered compaction, which can need a lot of free space during merges.

Workload	Better fit	Why
Millions of appends per second, mostly recent reads	LSM-tree	Writes stay sequential
Read-heavy transactions with random lookups	B+ tree	One predictable lookup, no compaction
Time-series data with a retention window	LSM-tree with time-window compaction	Whole files expire at once
A table used as a work queue	A real message queue	Insert-then-delete churn creates tombstones
Strict, flat read latency	B+ tree, or a carefully tuned LSM-tree	Compaction can cause latency spikes

Case studies

Two engineering teams have written openly about running LSM-tree databases at scale. Their stories show both the strengths and the sharp edges.

Discord: from Cassandra to ScyllaDB

Discord stores its chat messages in a database partitioned by channel and a time bucket. In 2017 the cluster ran 12 Cassandra nodes holding billions of messages. By early 2022 it had grown to 177 nodes holding trillions, and it had become what Discord called a high-toil system. Three problems traced back to the LSM-tree design and its runtime:

Reads cost more than writes. A write goes to the commit log and the MemTable. A read must check the MemTable and possibly several SSTables. Heavy concurrent reads on one channel created hot partitions, which slowed the nodes serving them.
Compaction fell behind. When compaction lags, reads touch more files. Discord's operators resorted to taking nodes out of rotation so they could catch up on compactions without serving traffic.
JVM garbage collection. Cassandra runs on the JVM, and garbage collection pauses caused latency spikes that took a lot of tuning to contain.

Discord moved the messages cluster to ScyllaDB, a Cassandra-compatible database written in C++ with no garbage collector. It also put Rust data services in front of the database. These coalesce identical concurrent requests into a single query, and route requests by channel ID so that coalescing works well. The cut-over happened in May 2022. Here is how the two clusters compared afterwards, according to Discord:

	Cassandra	ScyllaDB
Nodes	177	72
Disk per node	about 4 TB on average	9 TB
p99 latency, fetching historical messages	40 to 125 ms	15 ms
p99 latency, inserting a message	5 to 70 ms	about 5 ms, steady

The lesson is that both databases use the same LSM-tree design. The gains came from removing garbage collection pauses, giving each CPU core its own shard of work, and shielding the database from hot partitions. An LSM-tree is only as predictable as its compaction, its runtime and its access pattern.

Netflix: Cassandra for write-heavy data

Netflix has used Cassandra's LSM-tree engine in two well-documented places.

Viewing history. Netflix's first cloud-native viewing-history store used Cassandra because the data is time-series shaped and write-heavy, at a ratio of about 9 writes to 1 read. Netflix later redesigned it for a smaller storage footprint and consistent read and write performance as each member's history grows, splitting each member's data into a small set of recent records that change often and a larger set of older records that are stored compressed.

Live streaming origin. For live events, Netflix needed storage with very high write availability. It treats any write that fails within 500 ms as a bug. Its requirements and how they were met:

Requirement	How it was met
Writes almost never fail	A key-value layer on Cassandra with local-quorum writes, which keep working even if an entire availability zone is down
Large payloads of a few megabytes	Values are split into chunks, and each chunk can be retried and hedged safely
High write throughput	Cassandra's write-optimized LSM-tree engine
Read storms without hurting writes	A write-through cache in front of Cassandra, so almost all reads are served from memory

Netflix reports that Cassandra could serve reads at network line rate, but that doing so degraded concurrent writes. Adding a cache separated the read path from the write path. The write latency at the 99th percentile ended up close to the old S3-based design's median. The new setup was significantly more expensive, but low latency and high availability mattered more than cost.

The pattern to take away is common to both companies. An LSM-tree gives you cheap writes, and you pair it with a cache or a coalescing layer to protect the read path.

A practical tuning checklist

Exact settings differ between Cassandra, ScyllaDB and RocksDB, but the same levers exist everywhere:

Lever	What it changes	Guidance
Compaction strategy	The balance of read, write and space amplification	Size-tiered for write-heavy ingest with disk headroom. Leveled for read-heavy data. Time-window for time series with a TTL
MemTable size	How often data flushes, and how many files appear	Larger buffers mean fewer, bigger flushes and less compaction, but use more RAM and mean a longer log replay after a crash
Bloom filter size	False-positive rate against memory	About 10 bits per key gives roughly a 1% false-positive rate. Spend more memory on read-heavy tables
Block and key caches	Repeated reads of hot data	Size them to hold your hot set
Free disk space	Whether big merges can finish	Leave generous headroom, especially with size-tiered compaction
Compaction throughput limit	Background I/O against query latency	Cap it so queries are not starved, but not so low that compaction falls behind
Partition and key design	Hot spots and partition size	Bound partition size, for example with a time bucket, so no single partition grows without limit
TTLs and deletes	Tombstone volume	Prefer TTLs with time-window compaction over mass deletes

What to monitor

Pending compactions. A growing backlog is an early warning that reads are about to slow down.
Files touched per read. This is your read amplification, measured directly.
Tombstones scanned per read. High numbers point to a delete-heavy access pattern.
Free disk space. Running low during a large merge can stop compaction entirely.
p99 read latency, not just the average. Compaction trouble shows up in the tail first.

Common mistakes

Benchmarking a fresh database. An LSM-tree behaves differently once data volume is large and compaction is running. Test at realistic size and let compaction reach a steady state.
Treating the log as a guarantee. Check whether the write waits for the log to reach disk, or only to reach the operating system.
Unbounded partitions. One very large or very hot key undermines everything else you tuned.
Using deletes as a core operation. If your design needs constant deletes, reconsider the design.

Summary: B+ tree vs LSM-tree

	B+ tree	LSM-tree
Write pattern	In-place page updates	Sequential appends to a log and to immutable files
Write throughput	Limited by random I/O and page splits	Very high
Read cost	One tree lookup	Checks the MemTable and possibly several files. Bloom filters and caches soften this
Deletes	Removed in place	Written as tombstones, removed later by compaction
Background work	Page splits and rebalancing	Compaction
Space overhead	Fragmentation inside pages	Old versions and tombstones until they are compacted
Typical systems	MySQL (InnoDB), SQLite, and the indexes in PostgreSQL	Cassandra, ScyllaDB, HBase, Bigtable, LevelDB, RocksDB

Conclusion

An LSM-tree makes one big bet: writes should be cheap and sequential, and the cost of keeping data organized can be paid later, in the background. That bet pays off for chat history, telemetry, event logs and any workload that writes far more often than it reads in place.

Keep three ideas in mind:

Every write is an append. The write-ahead log protects it, the MemTable sorts it, and a flush turns it into an immutable sorted file.
Reads pay for it. Bloom filters, sparse indexes and caches keep that cost down, and compaction keeps the file count under control.
Compaction is the engine's real workload. Choose its strategy deliberately, watch its backlog, and design keys and deletes so it can keep up.

Sources

Discord: How Discord Stores Trillions of Messages
Netflix TechBlog: Netflix Live Origin
Netflix TechBlog: Scaling Time Series Data Storage, Part I (republished on Cassandra.link)

PostgreSQL Locks and Deadlocks: How They Work and How to Fix Them

Muhammad Ahsan Farooq — Sat, 10 Oct 2026 12:06:03 +0000

Table, row, and advisory locks, safe migrations, deadlock detection, and the queries to run when production is stuck

PostgreSQL lets thousands of queries run at once without trampling each other's data. Locks are what make that safe. Deadlocks are what you get when the waiting goes in a circle.

Thanks to multi-version concurrency control (MVCC), a reader never blocks a writer, and a writer never blocks a reader. Every query works from a consistent snapshot. But two writers changing the same row, or a schema change on a busy table, still have to take turns. That waiting is where slow queries, stuck migrations and deadlock detected errors come from.

This post explains how Postgres locks work, how they go wrong, and how to fix them. Each idea comes with a runnable example and a real-world use case.

In this post:

The three lock levels: table, row and advisory
Why one ALTER TABLE can freeze an app, and how to migrate safely
How deadlocks form, and how Postgres detects and breaks them
Prevention patterns: consistent ordering, short transactions, retries and SKIP LOCKED
Diagnosis: the queries to run when production is stuck

The big picture: three lock levels

Postgres locks at three levels. Most locks are taken for you. You choose explicitly only at the row and advisory levels.

three lock levels · table, row, advisory

Each level matters in a different situation:

Table-level: schema changes and bulk operations such as ALTER TABLE and CREATE INDEX.
Row-level: many transactions updating the same rows, such as balances, counters and inventory.
Advisory: your application needs a mutex across servers, such as one nightly job at a time.

Locks live until the transaction ends. Postgres releases table and row locks at COMMIT or ROLLBACK, not when the statement finishes. This one rule explains most lock trouble, because a long transaction holds its locks for its whole lifetime.

Table-level locks

Every statement takes a table-level lock automatically. There are eight modes. Four of them cover most of what you will meet:

Mode	Taken by	In plain English
`ACCESS SHARE`	`SELECT`	The weakest lock. Only `ACCESS EXCLUSIVE` conflicts with it
`ROW EXCLUSIVE`	`INSERT`, `UPDATE`, `DELETE`	Many writers can hold it on one table at once
`SHARE`	`CREATE INDEX` (without `CONCURRENTLY`)	Others can read, but nobody can write
`ACCESS EXCLUSIVE`	`ALTER TABLE` (most forms), `DROP TABLE`, `TRUNCATE`, `VACUUM FULL`	Blocks everything, including plain `SELECT`

The other four modes sit in between. For example, CREATE INDEX CONCURRENTLY and regular VACUUM take SHARE UPDATE EXCLUSIVE, which lets reads and writes continue.

Can two sessions hold these at once?

This matrix shows which of the four modes can coexist on the same table.

	`ACCESS SHARE`	`ROW EXCLUSIVE`	`SHARE`	`ACCESS EXCLUSIVE`
`ACCESS SHARE`	Yes	Yes	Yes	No
`ROW EXCLUSIVE`	Yes	Yes	No	No
`SHARE`	Yes	No	Yes	No
`ACCESS EXCLUSIVE`	No	No	No	No

Two writers never block each other at the table level, because ROW EXCLUSIVE does not conflict with itself. If they collide, it happens at the row level (next section).

Example: a schema change that blocks reads

-- Session A
BEGIN;
ALTER TABLE orders ADD COLUMN note text;  -- takes ACCESS EXCLUSIVE
-- transaction stays open...

-- Session B
SELECT count(*) FROM orders;              -- needs ACCESS SHARE, so it waits

Session B hangs until Session A commits or rolls back. Remember the rule from earlier: locks live until the transaction ends.

Use case: running a migration without an outage

The dangerous case is not the ALTER TABLE itself, which is often instant. It is the waiting. Postgres queues lock requests in order, so a pending ACCESS EXCLUSIVE request makes every later query line up behind it.

lock queue · how one migration stalls an app

A safe migration practice has four parts:

Set a lock_timeout so the migration gives up instead of queuing the whole app behind it.
Build indexes with CREATE INDEX CONCURRENTLY so writes continue. It cannot run inside a transaction block, and a failed run can leave an invalid index to drop.
Check for long-running transactions in pg_stat_activity before you start.
Keep the migration transaction short and retry later if it times out.

SET lock_timeout = '3s';
ALTER TABLE orders ADD COLUMN note text;  -- fails fast if it cannot get the lock

CREATE INDEX CONCURRENTLY idx_orders_customer ON orders (customer_id);

Row-level locks

Row locks are what writers actually fight over. Postgres takes them automatically on UPDATE and DELETE, and you can request them yourself with SELECT ... FOR UPDATE.

Postgres records a row lock inside the row itself, in the xmax field of the tuple header. That has two consequences:

No lock escalation. Some databases convert many row locks into one table lock when they run low on memory. Postgres does not, so it can lock millions of rows individually.
Locking has a write cost. Marking a row changes its page on disk, so SELECT ... FOR UPDATE over many rows creates write I/O.

Example: two writers, one row

-- Session A
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;  -- locks row 1

-- Session B
UPDATE accounts SET balance = balance + 50 WHERE id = 1;   -- waits for A

-- Session C
SELECT balance FROM accounts WHERE id = 1;                 -- returns at once, old value

MVCC row locking · 3 sessions on one row

Session C is not blocked because MVCC gives it the last committed version of the row. Session B has to wait because only one transaction can modify a row at a time.

The four row-lock modes

Mode	Taken by	Conflicts with
`FOR KEY SHARE`	Foreign key checks, such as inserting a child row	`FOR UPDATE`
`FOR SHARE`	`SELECT ... FOR SHARE`	`FOR NO KEY UPDATE`, `FOR UPDATE`
`FOR NO KEY UPDATE`	`UPDATE` that leaves key columns alone, or `SELECT ... FOR NO KEY UPDATE`	`FOR SHARE`, `FOR NO KEY UPDATE`, `FOR UPDATE`
`FOR UPDATE`	`DELETE`, `UPDATE` of key columns, or `SELECT ... FOR UPDATE`	All four modes

The weaker modes exist so that foreign keys do not block unrelated work. A normal UPDATE takes FOR NO KEY UPDATE, which does not conflict with the FOR KEY SHARE that a foreign key check takes. You can update a customer row while new orders for that customer are being inserted.

Use case: selling the last item in stock

Two buyers check out at once and only one item is left. If both read stock = 1 and then both decrement it, you oversell.

BEGIN;
SELECT stock FROM products WHERE id = 42 FOR UPDATE;  -- the second buyer waits here
-- the application checks that stock > 0
UPDATE products SET stock = stock - 1 WHERE id = 42;
COMMIT;

The second buyer waits at the SELECT, then sees stock = 0 once the first buyer commits. When the check is simple, one atomic statement is lighter and needs no explicit lock:

UPDATE products SET stock = stock - 1 WHERE id = 42 AND stock > 0;
-- 1 row updated: success. 0 rows updated: sold out.

Advisory locks

Advisory locks are locks whose meaning only your application knows. You pass Postgres a number, and it guarantees that only one session holds that number at a time. It never ties the lock to any table or row. That makes it a handy mutex for coordinating work across servers.

Example: run a nightly job on one server

Every server schedules the same job. Only the one that wins the lock runs it.

SELECT pg_try_advisory_lock(42);  -- true: you hold it. false: someone else does.

-- ... run the job if you got true ...

SELECT pg_advisory_unlock(42);

advisory lock · 3 servers, 1 winner

Four functions to know

Function	If the lock is taken	Released
`pg_advisory_lock(key)`	Waits	When you call `pg_advisory_unlock`, or the session ends
`pg_try_advisory_lock(key)`	Returns `false` at once	Same
`pg_advisory_xact_lock(key)`	Waits	Automatically at the end of the transaction
`pg_try_advisory_xact_lock(key)`	Returns `false` at once	Automatically at the end of the transaction

Use cases

Single-runner jobs: cron-style tasks that must not overlap across instances.
Per-customer serialization: call pg_advisory_xact_lock(customer_id) at the start of a transaction so two workers never process the same customer at once.
Migration guards: stop two deploys from running migrations at the same time. Several migration tools do this.

Two pitfalls

Prefer the transaction-level versions. Session-level locks outlive commits and rollbacks, so a forgotten unlock holds the lock until the connection closes. They are also unsafe behind a pooler in transaction mode, such as PgBouncer, because your next statement may run on a different connection.
Keys are shared by the whole database. Two unrelated features that both pick 42 will block each other. Use the two-integer form, pg_advisory_lock(namespace, id), or a documented key scheme.

How deadlocks form

A deadlock is a standstill. Two or more transactions each hold a lock the other needs, so each waits for the other forever. Waiting alone is normal in a database. A deadlock is waiting in a circle.

Deadlocks depend entirely on the order of operations. If every transaction touches resources in the same order, a cycle cannot form.

Example: two bank transfers in opposite directions

-- Transaction A: move 100 from account 1 to account 2
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;  -- waits for B

-- Transaction B: move 50 from account 2 to account 1, at the same moment
BEGIN;
UPDATE accounts SET balance = balance - 50 WHERE id = 2;
UPDATE accounts SET balance = balance + 50 WHERE id = 1;   -- waits for A

deadlock · 2 transactions, 2 rows, 1 cycle

Step by step:

A locks account 1.
B locks account 2.
A tries to lock account 2 and waits for B.
B tries to lock account 1 and waits for A.

Neither can finish, so neither will ever release its lock. Run each transfer alone and both succeed. The bug only appears when they overlap in time, which is why deadlocks often show up under load and not in testing.

Where deadlocks hide in real applications

Ledgers and transfers: money moves between two accounts in either direction, as above.
Overlapping batch jobs: two jobs update the same set of rows but walk through them in different orders.
A single multi-row statement: UPDATE ... WHERE id IN (...) does not promise any lock order. Two concurrent statements over overlapping rows can lock them in opposite orders.
Foreign keys: inserting child rows takes FOR KEY SHARE on the parent row, which can collide with another transaction that updates or deletes that parent.

How Postgres detects and breaks a deadlock

Postgres does not check for deadlocks on every lock wait. The check is expensive, and most waits end on their own within milliseconds. So Postgres is lazy: it waits first, and checks only if a transaction is still blocked after deadlock_timeout, which defaults to 1 second.

deadlock detection · 2 checks, 3 outcomes

The process has three steps:

Wait. A blocked transaction sleeps for deadlock_timeout.
Build the wait-for graph. If it is still blocked, Postgres traces who is waiting on whom.
Break the cycle. If the graph contains a loop, Postgres aborts one transaction. The others are unblocked and complete.

The transaction that runs the check is normally the one that gets aborted, which is usually the one whose wait timer fired first. Do not write code that depends on which side loses.

What your application sees

The aborted transaction fails with SQLSTATE 40P01 and its whole transaction is rolled back:

ERROR:  deadlock detected
DETAIL:  Process 6021 waits for ShareLock on transaction 8113; blocked by process 6044.
         Process 6044 waits for ShareLock on transaction 8112; blocked by process 6021.
HINT:  See server log for query details.

The process IDs and transaction IDs differ every time. Postgres also writes the full details, including the queries involved, to the server log.

Tuning `deadlock_timeout`

Lower values find real deadlocks sooner, but add checking work to every lock wait that lasts longer than the timeout.
Higher values reduce that work, but a real deadlock then stalls the stuck transactions for longer.

The 1 second default suits most systems. It is also the threshold for log_lock_waits, which logs any lock wait longer than deadlock_timeout and is worth enabling.

Preventing locks and deadlocks

You cannot remove locking, and you should not try. You can make waits short and cycles impossible. Five habits do most of the work.

1. Lock in a consistent order

This is the most effective fix. If every transaction locks rows in the same order, a cycle cannot form. The second transaction simply waits for the first.

lock ordering · deadlock vs short wait

In the transfer example, always lock the lower account ID first:

BEGIN;
SELECT id FROM accounts WHERE id IN (1, 2) ORDER BY id FOR UPDATE;  -- lowest ID first
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;

For batch jobs, sort the IDs in your application before you send a bulk UPDATE, so every job walks the rows in the same order.

2. Keep transactions short

A transaction holds its locks until it ends, so every millisecond inside it widens the window for a collision. Do slow work before BEGIN or after COMMIT, never between.

// Bad: the row stays locked during a slow network call
await client.query('BEGIN');
await client.query('UPDATE orders SET status = $1 WHERE id = $2', ['paid', id]);
await paymentApi.capture(id);
await client.query('COMMIT');

// Better: slow work first, then one quick statement
const receipt = await paymentApi.capture(id);
await client.query('UPDATE orders SET status = $1 WHERE id = $2', ['paid', id]);

Also set idle_in_transaction_session_timeout. It ends sessions that open a transaction and then sit idle, which would otherwise hold locks indefinitely.

3. Retry when a deadlock happens

In a busy system, an occasional deadlock is normal. Catch SQLSTATE 40P01 (deadlock) and 40001 (serialization failure), roll back, and run the whole transaction again after a short, randomized delay.

import { Pool, PoolClient } from 'pg';

const pool = new Pool();
const RETRYABLE = new Set(['40P01', '40001']);

async function withRetry<T>(
  work: (client: PoolClient) => Promise<T>,
  attempts = 5
): Promise<T> {
  for (let attempt = 1; ; attempt++) {
    const client = await pool.connect();
    try {
      await client.query('BEGIN');
      const result = await work(client);
      await client.query('COMMIT');
      return result;
    } catch (err: any) {
      await client.query('ROLLBACK').catch(() => {});
      if (!RETRYABLE.has(err.code) || attempt >= attempts) throw err;
      // exponential backoff with jitter
      const delay = Math.min(1000, 50 * 2 ** attempt) * (0.5 + Math.random() / 2);
      await new Promise((resolve) => setTimeout(resolve, delay));
    } finally {
      client.release();
    }
  }
}

The retried work must be safe to run twice. Keep external side effects, such as emails and payments, outside the transaction.

4. Do not wait when you do not have to

Sometimes the best way to avoid lock trouble is to refuse to queue. Postgres gives you three tools:

SKIP LOCKED: ignore rows another transaction has locked and take the next ones.
NOWAIT: fail immediately with error 55P03 if a row is locked.
lock_timeout: fail with the same error after a set time.

The classic use case is a job queue. With a plain FOR UPDATE, every worker reaches for the same first row and the rest line up behind it.

job queue · plain FOR UPDATE vs SKIP LOCKED

BEGIN;
SELECT id, payload FROM jobs
WHERE status = 'pending'
ORDER BY id
LIMIT 1
FOR UPDATE SKIP LOCKED;
-- process the job, then:
UPDATE jobs SET status = 'done' WHERE id = $1;
COMMIT;

Workers no longer block each other on the same job. SKIP LOCKED gives each caller a deliberately incomplete view of the table, so use it for queues and work distribution, not for general reads. Strict first-in-first-out ordering is also not guaranteed.

5. Lock only what you need

Pick the weakest row lock that works. If you only need to stop concurrent updates and are not changing key columns, FOR NO KEY UPDATE blocks less than FOR UPDATE.
Touch fewer rows. Narrow the WHERE clause and do not update rows whose values will not change.
Index foreign key columns. Without an index on the referencing column, checks and cascading deletes scan more of the table and hold locks for longer.

Diagnosing lock problems in production

There are two kinds of symptoms. Queries that hang point to a lock wait, and the fix is to find the blocker. deadlock detected errors point to a lock-ordering bug, and the fix is in your code. Start by finding out who is blocking whom.

Find the blocked queries

SELECT
  pid,
  pg_blocking_pids(pid)  AS blocked_by,
  wait_event_type,
  wait_event,
  now() - query_start    AS waiting_for,
  left(query, 80)        AS query
FROM pg_stat_activity
WHERE cardinality(pg_blocking_pids(pid)) > 0;

Each row is a session that is waiting. blocked_by lists the process IDs holding the lock it needs.

Find the blockers

SELECT pid, state, now() - xact_start AS transaction_age, left(query, 80) AS last_query
FROM pg_stat_activity
WHERE pid IN (SELECT unnest(pg_blocking_pids(pid)) FROM pg_stat_activity);

The usual culprit is a session in state idle in transaction. An application opened a transaction, did some work, and then stopped talking to the database without committing. Its locks stay held. The last_query column shows the last statement it ran, which usually points to the code path at fault.

If you must unblock production, pg_cancel_backend(pid) cancels the running query. pg_terminate_backend(pid) closes the connection and rolls back its transaction. Fix the cause afterwards, because the same blocker will come back.

For the full picture, the pg_locks view lists every lock that is held or awaited. A row-lock wait shows up there as a wait on the holder's transactionid.

Settings worth reviewing

Setting	What it does	Advice
`log_lock_waits`	Logs any lock wait longer than `deadlock_timeout`	Turn it on
`deadlock_timeout`	How long to wait before checking for a deadlock	Keep the 1 second default
`lock_timeout`	Makes a statement give up if it waits too long for a lock	Set it per session for migrations
`idle_in_transaction_session_timeout`	Ends sessions that sit idle inside a transaction	Set a value that fits your app

Reading a deadlock in the server log

When a deadlock happens, the log shows each process, the lock it waited for, and the query it was running. Find the two statements and write down the order in which each one locks rows. They will be opposite. Make both code paths lock in the same order and the deadlock goes away.

Cheat sheet: symptom, cause, fix

Symptom	Likely cause	Fix
Everything hangs right after a deploy	A migration is waiting for `ACCESS EXCLUSIVE`, and other queries have queued behind it	Cancel it. Rerun with `lock_timeout`, and use `CREATE INDEX CONCURRENTLY`
Many queries stuck on one row	A hot row updated inside long transactions	Shorten transactions, or use one atomic `UPDATE`
`deadlock detected` (`40P01`)	Transactions lock rows in different orders	Lock in a consistent order, and retry with backoff
Sessions `idle in transaction` holding locks	The app opened a transaction and then stalled, or crashed	Set `idle_in_transaction_session_timeout`, and keep transactions short
Workers queue up on a job table	Plain `SELECT ... FOR UPDATE`	Use `FOR UPDATE SKIP LOCKED`
Two servers run the same cron job	No coordination between instances	Use `pg_try_advisory_lock`

Conclusion

Locks are not the enemy. They are how Postgres keeps concurrent writers from corrupting each other's data. Trouble starts when locks are held too long or requested in conflicting orders. Keep three ideas in mind:

Locks last until the transaction ends. Short transactions mean short waits.
Readers never block writers, but writers block writers, and schema changes block everyone. Treat ALTER TABLE on a busy table with respect.
Deadlocks come from inconsistent ordering. Lock in the same order everywhere, and retry when one slips through.

When something does go wrong, pg_stat_activity and pg_blocking_pids() will usually show you the blocker within a minute.

How Databases Scale to Millions of Reads and Writes per Second

Muhammad Ahsan Farooq — Fri, 09 Oct 2026 20:31:02 +0000

Replicas, caches, sharding, LSM-trees, and the consistency trade-offs that come with each

Every database hits a ceiling. One server can take only so many CPU cores and so much RAM before the next upgrade costs far more than it delivers. To serve millions of operations per second, you stop making one machine faster and start spreading the work across many.

Two terms get mixed up here. Performance is how fast the system answers a single request. Scalability is whether it keeps answering that fast as load grows. A database can be quick at 100 requests per second and fall over at 100,000. This post is about the second problem.

The core idea is to split the read path from the write path, because they fail in different ways and need different tools. Then choose storage engines, caches and consistency models that fit each path, and accept the trade-offs between consistency, availability and latency.

read path and write path · 6 components, 2 lanes

Reads are served from copies: an edge cache, a distributed cache and replicas. Writes are buffered by a queue and split across shards.

In this post:

Part 1: scaling reads with replicas, caches, indexes and denormalization
Part 2: scaling writes with sharding, LSM-trees, queues and connection pools
Part 3: consistency models, CAP, PACELC and distributed SQL
A one-page cheat sheet you can keep

Part 1: Scaling reads

In most web applications, reads outnumber writes by 100:1 or even 1,000:1. That imbalance is good news: reads are easier to scale, because the same data can be copied and served from many places.

1. Primary-replica replication

Send every write to one primary node. The primary streams those changes to one or more read-only replicas, and a load balancer spreads read traffic across the replica pool. Add replicas and read capacity grows almost linearly.

primary-replica replication · 1 primary, 3 replicas

Replicas can lag the primary slightly. The dashed path sends a user who just wrote back to the primary, so they always see their own change.

The catch is replication lag. Replication is usually asynchronous, so the primary does not wait for replicas before acknowledging a write. That keeps write latency low, but a replica can briefly serve stale data.

For cases where staleness is visible to users, such as seeing your own post right after publishing it, use read-your-own-writes. Route that user's reads to the primary (or a replica known to be caught up) for a short window, while everyone else reads from the asynchronous replicas.

2. Multi-layer distributed caching

Sending millions of concurrent reads to a disk-backed database is unsustainable. Put a distributed in-memory cache such as Redis or Memcached in front of it. Large platforms report hit rates above 95-99%. Netflix's EVCache is a well-known example. At those rates the database sees only a small fraction of the real traffic.

To see why this matters, take 1,000,000 reads per second. At a 90% hit rate, 100,000 reads reach the database. At 95%, 50,000 do. At 99%, only 10,000 do.

Two patterns cover most cases:

Cache-aside (lazy loading): the application checks the cache first. On a miss, it reads from the database, fills the cache and returns the result.
Write-through: every update is written synchronously to both the cache and the database, so later reads get fresh data. The price is slightly higher write latency.

cache-aside read · 1 decision, 2 paths

The three common strategies behave differently on writes. Write-behind is covered in Part 2.

Strategy	How it works	Write latency	Main risk	Best for
Cache-aside	The app reads the cache and loads from the database on a miss	Unchanged	Stale entries until the TTL expires or you invalidate them	Read-heavy data that tolerates brief staleness
Write-through	Every write goes to the cache and the database together	Slightly higher	Extra write cost for data that is rarely read	Data that must be fresh on the next read
Write-behind	Writes hit the cache, which flushes to the database later	Lowest	Unflushed writes are lost if the cache node fails	Bursty writes where small loss risk is acceptable

3. Preventing cache stampedes

When a popular key expires, thousands of requests can miss the cache in the same millisecond. If nothing stops them, they all hit the database with the same query. This is a cache stampede, also called the thundering herd, and it can turn one expired key into an outage.

Two defenses work well together:

Request coalescing: the first request for a missing key goes to the database. Duplicate requests wait for that single result and are served from the new cache entry.
Probabilistic early expiration: each request has a small, growing chance of refreshing a key in the background as it nears its TTL, so the key is usually refreshed before it ever expires.

request coalescing · before and after

The database sees one query instead of thousands, and the waiting requests are served from the fresh entry.

Here is a minimal cache-aside read with in-process coalescing in TypeScript:

const inflight = new Map<string, Promise<User>>();

async function getUser(id: string): Promise<User> {
  const cached = await redis.get(`user:${id}`);
  if (cached) return JSON.parse(cached);

  // Coalesce: one database query per key, everyone else awaits it
  if (!inflight.has(id)) {
    const load = (async () => {
      const user = await db.users.findById(id);
      await redis.set(`user:${id}`, JSON.stringify(user), { EX: 300 });
      return user;
    })().finally(() => inflight.delete(id));
    inflight.set(id, load);
  }
  return inflight.get(id)!;
}

This coalesces per process only. Across a fleet of servers, you still get one query per server, so use a distributed lock or a caching proxy for global coalescing. Adding random jitter to TTLs also stops many keys from expiring together.

4. Indexing and denormalization

Indexes: a secondary index (usually a B-tree or B+ tree) turns a full-table scan, O(N), into a logarithmic lookup, O(log N). It is the cheapest read optimization you have.
Denormalization: multi-table JOINs get expensive at scale in both CPU and disk I/O. Duplicating the fields you need into the target record, or into a materialized view, replaces the join with a fast, flat lookup. You pay for it with extra storage and more work to keep copies in sync.

In short: copy the data (replicas), keep hot data in memory (caches), and make each lookup cheaper (indexes and denormalization).

Part 2: Scaling writes

Write-heavy systems such as real-time messaging, IoT telemetry and clickstream logging are harder. Read replicas do not help, because every write still has to land on a primary storage engine. You need to spread the writes themselves.

1. Sharding (horizontal partitioning)

Sharding splits one large dataset across several independent database nodes. Each node, or shard, holds a distinct slice of the data. A shard key, such as user_id or a hash of a composite key, decides which shard stores a given row.

shard key choice · balanced vs hot shard

With range partitioning, the newest IDs always land on the last shard. A hashed key spreads the same writes across every shard.

The key you pick matters more than almost anything else:

Avoid hot keys. With range-based partitioning, sequentially increasing IDs send all new writes to the last shard. The rest of the cluster sits idle.
Prefer high-cardinality, evenly distributed keys. UUIDs or hash-based partitioning spread writes across every node.
Know the trade-off. Hash sharding balances load but makes range scans and cross-shard queries more expensive.

2. Write-optimized storage engines: LSM-trees vs B-trees

Traditional relational databases use B-trees, which update pages in place. Under heavy write volume, those random disk writes become the bottleneck.

Log-structured merge-trees (LSM-trees) avoid this. Wide-column and key-value stores such as Apache Cassandra, ScyllaDB and RocksDB use them. A write goes to an in-memory buffer (the MemTable) and is flushed to disk as an immutable, sorted file (an SSTable). Because the disk work is sequential, these engines sustain millions of writes per second at very low write latency.

LSM-tree write path · memory to disk to compaction

The trade-off shows up on reads. A lookup may check the MemTable and several SSTables, and background compaction has to merge files over time. LSM-trees buy write throughput with read and compaction cost.

	B-tree	LSM-tree
Write pattern	Random, in-place page updates	Sequential appends
Write throughput	Limited by random disk I/O	Very high
Read cost	One tree lookup	May check the MemTable and several SSTables
Background work	Page splits and rebalancing	Compaction
Typical engines	Traditional relational databases	Cassandra, ScyllaDB, RocksDB

3. Asynchronous buffering and event streaming

To absorb bursts, put a durable queue such as Apache Kafka or RabbitMQ between your application and the database. The API accepts the event and returns quickly. Ingestion workers then consume from the queue, group individual writes into batches and bulk-insert them.

asynchronous write buffering · 4 stages

This load leveling protects the database from sudden spikes and prevents thread starvation in the application tier. The cost is that writes become eventually visible, which your product has to be able to tolerate.

A related pattern is write-behind (write-back) caching. The application writes to the cache, and the cache flushes to the database asynchronously. It is very fast, but data not yet flushed can be lost if the cache node fails, so use it only where that risk is acceptable.

4. Connection pooling

Opening a new database connection for every API request costs real CPU and memory. Unclosed connections are worse. They leak, exhaust the pool and eventually starve threads.

Keep a pool of persistent, reusable connections, or put a database proxy in front of the database. This removes most handshake overhead and keeps throughput stable under high concurrency.

In short: split the writes across nodes, use an engine built for appends, and buffer bursts behind a queue.

Part 3: Consistency, CAP and distributed SQL

Once data spreads across nodes and regions, network partitions stop being hypothetical. They happen. The CAP theorem says that during a partition you must choose between consistency and availability. PACELC extends it: even with no partition, you still trade latency against consistency, because coordinating replicas takes time.

Read the decision tree below from the top. First ask whether a partition is happening, then follow the trade-off that applies.

CAP and PACELC decision tree · 1 question, 2 branches, 4 outcomes

Cassandra and DynamoDB sit on the availability and low-latency side. Spanner and Aurora DSQL pay in latency to keep consistency.

AP vs CP systems

AP (availability + partition tolerance): Apache Cassandra and Amazon DynamoDB, in their default setups, keep accepting writes during a partition and synchronize in the background. The result is eventual consistency. Replicas converge, but readers may briefly see different values.
CP (consistency + partition tolerance): MongoDB and HBase favor consistency. If a node cannot reach a quorum, it refuses reads or writes instead of risking stale or conflicting data. You give up availability for the duration of the partition.

Many of these systems are tunable, so treat the labels as defaults, not absolutes. A practical rule: choose AP when a stale read is tolerable (feeds, carts, telemetry), and CP when a wrong answer costs money (payments, inventory, ledgers).

Distributed SQL

Distributed SQL databases such as Google Spanner and Amazon Aurora DSQL aim for both worlds: SQL, ACID transactions and strong guarantees, with multi-region, active-active availability and no hand-written sharding in your application.

Optimistic concurrency control (OCC): Aurora DSQL runs a read-write transaction locally without taking locks up front. At commit time, an adjudicator service checks whether a conflicting transaction wrote to the same keys during that window. If one did, a transaction is rejected and must be retried. OCC works best when conflicts are rare, and your application code must be ready to retry.
Synchronized clocks: Spanner (TrueTime) and Aurora DSQL (Amazon Time Sync) rely on precisely synchronized clocks to give transactions a consistent global order. That lets reads be served from the local region with strong snapshot semantics, without a cross-region round trip for every read.

This is not free. Commits that span regions still pay a latency cost for coordination, which is exactly the latency-versus-consistency trade-off PACELC describes.

In short: pick consistency or availability per feature, and know what each database does when the network breaks.

Cheat sheet

Reads scale by copying data to more places. Writes scale by splitting data across more nodes.

Layer	Read-heavy workloads	Write-heavy workloads
Data partitioning	Read replicas (primary-replica)	Horizontal sharding with evenly distributed hash keys
Caching	Cache-aside or write-through (Redis, Memcached)	Write-behind (asynchronous cache-to-database flush)
Storage engine	B+ tree indexes and denormalized views	LSM-trees
Ingestion	CDN or edge caching, request coalescing	Message queues (Kafka) and batching
Consistency	Eventual consistency, read-your-own-writes	Distributed SQL with OCC, or CRDTs for conflict-free multi-region writes

Conclusion

Scaling to millions of operations per second is rarely one big trick. It is a series of small decisions, made in a sensible order:

Index and denormalize before you buy more hardware.
Add replicas and a cache to absorb reads, and guard against stampedes.
When writes outgrow one node, shard on an evenly distributed key and consider a write-optimized engine.
Buffer bursts with a queue and pool your connections.
Decide, feature by feature, how much staleness you can accept, and choose AP or CP behavior to match.

Every step buys capacity and charges you in consistency, complexity or operational cost. Measure before you add the next layer.

Two topics worth exploring next: write-behind caching with Redis Streams, and conflict-free replicated data types (CRDTs) for resolving conflicts in multi-region AP databases.

Beyond Microservices: Surviving the Storm with Cell-Based Architecture

Muhammad Ahsan Farooq — Tue, 03 Mar 2026 16:39:30 +0000

Containing failures before they become outages, with lessons from Slack, DoorDash, and AWS

For a decade, the answer to scaling was to break the monolith. Slice a giant codebase along domain lines, and let independent services scale, deploy and fail on their own. Microservices delivered on that promise, up to a point.

Then systems grew. Dozens of services became hundreds. They shared databases, caches and queues, and each of those became load-bearing. Teams added circuit breakers, retries and timeouts. Those tools help, but they treat symptoms. Slowly, teams had rebuilt the monolith with network hops in between: a networked monolith, where one sick component can slow down everyone.

Cell-based architecture goes after the root cause. Instead of one big shared pool that serves all customers, you run many small, identical, self-contained copies of your whole stack, called cells. Each cell serves a slice of your customers. When something breaks, only that slice is affected.

This post explains how cells work, why they contain failures, and how to adopt them, with examples from Slack, DoorDash and AWS.

In this post:

The problem: gray failures, and why shared infrastructure turns small faults into global outages
What a cell is, and the one rule that makes the pattern work
The router: partition keys, cell mapping and request flow
Failover and draining, and the arithmetic of blast radius
Case studies: Slack, DoorDash and AWS shuffle sharding
An adoption playbook, the costs, and a readiness checklist

The problem: from monolith to networked monolith

A classic microservice stack looks clean on a whiteboard. A pool of web servers feeds a pool of application servers, which share one database cluster and one global Redis cache. Traffic round-robins across everything, and scaling out is easy: add more pods.

The weakness is not scale. It is interconnection. Every request path depends on the same shared infrastructure, so a small fault in one corner can reach everyone.

Binary failures vs gray failures

Binary failures are the easy kind. A server crashes or a database goes offline. Health checks notice, the load balancer stops sending traffic, and automation takes over.

Gray failures are the dangerous kind. A component is degraded but not dead, so nothing triggers automated remediation. Slack describes them as failures where different components have different views of whether the system is available. A highway with one lane crawling at 10 mph does not stop traffic. It backs it up until everything gridlocks.

Common gray failures include:

Packet loss within a single Availability Zone
A Redis shard slowing down under memory pressure
A retry storm caused by a bad deployment config
Connection pool exhaustion behind a slow dependency
A noisy neighbor throttling CPU on a shared host
DNS resolution latency spikes

How one slow node takes everything down

gray failure cascade · illustrative, 4 stages

The numbers in the picture are illustrative, but the arithmetic behind the mechanism is not. Little's law says the number of requests in flight equals the arrival rate times the average time each one takes. Suppose an app tier has 100 worker threads, receives 1,000 requests per second, and normally answers in 50 ms. Now 10% of requests touch the slow node and take 2 seconds:

Scenario	Average latency	Threads needed	Threads available
Healthy	50 ms	50	100
10% of requests hit the slow node	0.9 × 50 ms + 0.1 × 2,000 ms = 245 ms	245	100

The pool is exhausted, even though 90% of requests are perfectly healthy. Once threads run out, the healthy requests queue up behind the slow ones, and everyone sees the slowdown. Retries make it worse by adding load to a tier that is already struggling.

The standard defenses are circuit breakers, bulkheaded thread pools, timeouts and retries with jitter. They help at the margins, but they all treat symptoms. The root cause is that all requests share all infrastructure, so the blast radius of any failure is effectively the whole platform.

Slack's own incident review asked the question this way after a 2021 network fault in a single Availability Zone: why was this outage visible to users at all? The answer led to the design in this post.

What a cell is

The name comes from marine engineering. Ships are built with watertight bulkheads, walls that divide the hull into compartments. If the hull is breached, only the damaged compartment floods. A common retelling of the Titanic's sinking is that its bulkheads did not reach high enough, so water spilled from one compartment into the next. The design assumption was violated, and the failure cascaded the way it does in a poorly isolated distributed system.

AWS's guidance defines it precisely: a cell-based architecture uses multiple isolated instances of a workload. Each instance, a cell, is independent, shares no state with other cells, and handles a subset of the overall requests. If a workload uses 10 cells to serve 100 requests, a failure in one cell leaves 90% of the requests unaffected.

cell-based architecture · router, 4 cells, control plane

The pattern has three parts:

Component	Its job	Design rule
Cell router	Maps each request to exactly one cell	The thinnest possible layer. It routes and does nothing else
Cell	A complete workload that can run on its own	No shared state, and no calls to other cells
Control plane	Provisions and removes cells, assigns and migrates tenants	Keep it thin, read-mostly, and cacheable at the cell edge

Building cells does not necessarily mean multiplying your infrastructure. AWS notes that an application with 30 hosts can keep the same 30 hosts, grouped into cells behind a router.

Anatomy of a cell

A cell is not a single service or a single database. It is a complete copy of your application stack, scoped to a bounded set of tenants:

inside one cell · compute, data, observability

Every cell owns its own:

Compute: API and worker containers
Storage: database shards, not a connection to a shared cluster
Cache: its own Redis or Memcached cluster
Queues: its own Kafka topics or SQS queues
Observability: its own metrics namespace, so a sick cell stands out on a dashboard instead of being hidden inside a global average

The golden rule: cells never talk to each other

A request that enters Cell 4 lives and dies in Cell 4. There are no cross-cell service calls, no shared databases and no shared caches.

This rule is not negotiable. The moment one cell depends on another, a slow cell can drag its neighbor down, and you have rebuilt the networked monolith with extra steps. If a request really needs data owned by another tenant, solve it at the routing layer by sending the request to the right cell, or by replicating data asynchronously. Never let cells call each other synchronously.

The control plane

Some concerns sit outside any one cell: assigning tenants to cells, provisioning new cells, migrating tenants, global feature flags and billing state. These belong to the control plane.

The discipline is to keep it read-mostly and to cache what cells need at the cell edge. If the control plane has a brief outage, each cell should keep serving traffic from its cached state. A control plane outage must never become a cell outage.

The routing layer

If cells are the body of this architecture, the router is the brain. AWS calls it the thinnest possible layer: it routes requests to the right cell, and only that. Every request passes through it, so it must be deterministic and extremely reliable. This is one of the few components worth over-engineering from the start.

Step 1: choose the partition key

The partition key decides how customers are sliced across cells. It is the most consequential design decision in the whole system. AWS says the key should align with the grain of the service, meaning the natural way its workload can be divided with minimal cross-cell interaction.

A good partition key has three properties:

It co-locates related data. Nearly everything a request needs belongs to the same key value.
It distributes reasonably evenly. No single value carries a disproportionate share of traffic.
It is present on every request. The router can read it without extra lookups.

Partition key	Typical fit	Example
Organization or tenant ID	B2B SaaS and multi-tenant platforms	A workspace, company or account
User ID	Consumer apps with isolated user data	Social, gaming and messaging apps
Region	Compliance and data residency needs	EU-only customer data
Order or delivery ID	Workflows scoped to one transaction	Food delivery, ride-hailing
Merchant or store ID	Marketplaces	One seller's catalog and orders

Avoid keys that are too coarse, such as country, where a single value could be larger than a cell can handle. Avoid keys that are too fine, such as session_id, which would need an unmanageable number of cells. The sweet spot is an entity that owns a meaningful, bounded set of data.

Step 2: map keys to cells

The router needs to know which cell owns which key. There are three common approaches:

Approach	How it works	Strength	Weakness
Lookup table	A small, fast store maps `org_id` to `cell_id`	Flexible. Moving a tenant is one row update	An extra lookup per request, so it must be cached aggressively
Deterministic hash	`cell = hash(org_id) % number_of_cells`	No lookups, stateless and fast	Adding cells reshuffles most tenants, and you cannot move one tenant alone
Hybrid	Hash by default, with a lookup table for exceptions	Speed on the common path and flexibility where it matters	Two mechanisms to maintain

Most production systems end up with the hybrid. Large enterprise tenants get pinned to specific cells for compliance, capacity or contractual reasons, and everyone else is hashed. A useful refinement is to hash into a fixed number of buckets, such as 1,024, and map buckets to cells. Adding a cell then means reassigning a few buckets, not rehashing everyone.

const BUCKETS = 1024;

function cellFor(orgId: string): string {
  // 1. Pinned tenants (compliance, very large customers) come from the lookup table
  const pinned = pinnedTenants.get(orgId);
  if (pinned) return pinned;

  // 2. Everyone else: a stable hash picks a bucket, and the bucket maps to a cell
  const bucket = stableHash(orgId) % BUCKETS;
  return bucketToCell[bucket];
}

Step 3: follow one request

Here is the full lifecycle of a request for the tenant acme-corp-7291:

request lifecycle through the cell router · 7 steps

Step 4 matters. The router tracks cell health, so it can steer traffic away from a sick cell. That capability is what makes failover possible, which is the next topic.

Failover and draining

Because the router tracks cell health, it can act when a cell goes bad. What it should do depends on what kind of cell you built.

Two kinds of cells, two kinds of failover

Tenant-partitioned cells own the data for specific tenants. If one fails, other cells cannot simply take over, because the tenant's data lives in the failed cell. The value here is containment: only that cell's tenants are affected, and the fix is scoped to one small environment. Recovery means repairing the cell, restoring it from backups, or failing over to a standby that holds replicated data.
Replica cells are interchangeable copies that can serve any customer, for example one per Availability Zone. Here the router can shift traffic from a sick cell to healthy ones. This is the model Slack uses.

Many systems combine both: cells aligned to Availability Zones, with tenants partitioned inside them.

Draining, as Slack does it

Slack's cells are Availability Zones. After a 2021 incident, its engineers set four goals for an AZ drain:

Move as much traffic as possible out of an AZ within 5 minutes.
Cause no user-visible errors. In-flight requests finish normally.
Make drains and undrains incremental, down to 1% of traffic, so a recovered AZ can be tested gently.
Never depend on resources inside the AZ being drained.

To meet them, Slack made each service talk only to services in its own AZ, a design it calls siloing. Draining an AZ then needs no change inside the services. It only requires steering new user requests away at the edge. Edge Envoy load balancers receive per-AZ weights from a control plane, and an AZ whose weight is set to zero stops getting new requests while existing ones complete. The change propagates in seconds. Slack also chose to keep a human in the loop: an operator decides when to drain.

The same idea works for any cell router. This picture shows the effect on three cells as Cell A is drained:

illustrative router weights · 3 cells, 3 states

Here is an illustrative control-plane routine. It lowers the weight in steps and stops if the receiving cells start to struggle:

async function drainCell(cellId: string, steps = [50, 20, 0]) {
  for (const weight of steps) {
    await controlPlane.setCellWeight(cellId, weight);   // the router applies it within seconds
    await sleep(60_000);                                // watch error rates before going further
    if (await errorRatesRising(otherCells(cellId))) {
      throw new Error('Drain paused: receiving cells are struggling');
    }
  }
}

Hard failover vs draining

	Hard failover	Draining
How it works	Remap all of a cell's traffic at once	Shift weight gradually while in-flight requests finish
Main risk	A sudden surge overwhelms the receiving cells	Slower to complete
Capacity needed	Spare capacity everywhere, immediately	Spare capacity, claimed in steps
Best for	A cell that is completely down	Gray failures, where you are still deciding

Either way, the receiving cells need spare capacity. The next section works out how much.

Blast radius: the math that sells it

With N equal cells, a failure in one cell affects about 1/N of your customers. AWS's example: with 10 cells serving 100 requests, a failure in one cell leaves 90% of the requests untouched.

computed: 1/N users affected, 1/(N-1) spare capacity, for N equal cells

The chart also shows the price of recovery. If a failed cell's traffic is spread evenly over the survivors, each one needs 1/(N-1) of spare capacity. With 10 cells that is about 11%. With 100 cells it is about 1%. If you instead pair cells so each fails over to a single neighbor, every cell must keep 100% spare capacity, which is why spreading the load matters. More cells mean a smaller blast radius and a smaller proportional safety margin, but also more environments to run.

Before and after

Failure	Without cells	With 100 cells
Bad deployment	Reaches everyone as the rollout proceeds	Rolled out cell by cell. The first cell exposes about 1% of customers
Corrupt data or a bad migration	Every tenant is at risk	Only the tenants in that cell
Infrastructure fault	Everyone sharing the infrastructure	Contained to the affected cell, and traffic can be drained from it
Noisy tenant	Degrades all other tenants	Degrades only its own cell
Poison pill request	Can take down the shared pool	Crashes one cell's workers, and the rest keep serving

Before cells, one engineer pushing a bad config at 2 a.m. could take down the whole platform. After cells, the same mistake reaches 1% of customers. It is still an incident, but it is a small one that is recoverable and debuggable by a single on-call engineer.

Deploy one cell at a time

Safe deployments get dramatically easier. In a pooled system, a canary sends 5 to 10% of global traffic to the new version, so a bug hits a random slice of all customers. With cells, you deploy to one cell. The affected customers are known, bounded and can even be notified. AWS recommends applying changes gradually, compartment by compartment.

A rollout plan for 100 cells might look like this:

Wave	Cells	Cumulative customers exposed
1	One canary cell, ideally with internal or low-risk tenants	1%
2	The next 4 cells	5%
3	The next 20 cells	25%
4	Everything else	100%

Pause between waves, watch per-cell error rates, and halt automatically if a wave looks worse than the baseline.

Real-world case studies

Three companies show three different ways to apply the idea. I checked each against the team's own write-up.

Slack: Availability Zones as cells

On June 30, 2021, a network link between Availability Zones in Slack's main region began to fail intermittently. The cloud provider removed it, put it back, and removed it permanently later that day. On paper it was a routine hardware failure, so Slack's engineers asked why users saw it at all.

The answer was a gray failure. One user request can fan out into hundreds of calls to backends, and every call must succeed. Failure detection needs some failures before it can exclude a bad server. Slack's main datastore also needs a single available primary for each write, so a shard whose primary is unreachable cannot accept writes. Different components held different views of what was healthy, and the whole system suffered.

Slack's response was to treat each Availability Zone as a cell and to silo services so they only talk within their own zone. That turns an AZ drain into an edge routing change, as described in the previous section. The design target was to drain most of a zone's traffic within 5 minutes, which matters because Slack's 99.99% availability SLA allows less than an hour of total unavailability per year.

Note what Slack did not do. Its cells are replicas aligned to Availability Zones, not partitions of customers by tenant. The pattern is flexible: what matters is that the isolation boundary is real, and that you can steer traffic around it.

DoorDash: cells for isolation, plus cheaper traffic

DoorDash's microservices run in multiple isolated cells. Every service has one Kubernetes deployment per cell, and traffic between cells is not permitted, which limits the blast radius of a single cell failure. Services not yet migrated live in a global cell, reached through internal Envoy routers. Each cell is made of several Kubernetes clusters, and each cluster spans multiple Availability Zones.

The cost problem came from success. As the number of microservices grew, so did cross-AZ data transfer charges, which are billed on both sending and receiving. DoorDash's answer was to enable Envoy's zone-aware routing in its service mesh, so callers prefer callees in their own zone, while still balancing load so no pod is overloaded. Even with pods spread evenly across zones, about 10% of traffic still crosses zones.

The investigation is instructive. Flow logs and metrics showed that the largest source of cross-AZ cost was non-storage traffic within each cell, and about 20% of that cost was traffic to a single load balancer in front of an event ingestion pipeline. That path had several extra hops. DoorDash moved the service into the same cell as its clients and let their sidecars route directly to its pods. The savings were large enough that the cloud provider reached out to ask whether DoorDash was having a production incident.

The lesson is that cells give you a bounded place to optimize. Because traffic stays inside a cell, you can count the hops and measure the cost. Note that the savings came from zone-aware routing, which the cell boundary made tractable. DoorDash did not publish a single overall percentage.

AWS: shuffle sharding, cells without hard partitions

Amazon uses a variant called shuffle sharding for workloads where pinning each customer to one cell is impractical, such as DNS. Instead of one cell per customer, each customer gets a random subset of workers, and different customers get different subsets.

shuffle sharding · 8 workers, 3 customers

The arithmetic is simple. With 8 workers and 2 per customer, there are 28 possible pairs. Only a customer assigned the very same pair as the troublemaker loses both workers, which is 1 in 28, or about 3.6%. Ordinary sharding into 4 shards would put 25% of customers on the failed shard. Customers who share a single worker still have the other one, as long as their clients retry.

Amazon Route 53 applies this at scale. It arranges capacity into 2,048 virtual name servers and gives every domain a shuffle shard of four, which allows about 730 billion distinct shards. No domain shares more than two virtual name servers with any other. If one domain is hit by a DDoS attack, its four name servers see a spike, and no other customer notices.

Shuffle sharding needs no partition key and works well for stateless workloads. It is a useful tool when you cannot commit to hard cells.

Implementation playbook

You do not need to boil the ocean. Most teams start with two or three cells for one critical service and expand from there. Five phases cover the work.

Phase 1: find your partition key

Look at your data model for the entity that naturally owns everything else. In B2B SaaS that is usually org_id or tenant_id. Draw the dependency graph and check that the vast majority of queries can be answered inside one tenant's scope.

If you cannot find a clean partition key, stop here. Cells without one produce a system that is complex without being resilient.

Phase 2: build the router

Stand up a gateway that does four things:

Extracts the partition key from the request, such as a JWT claim, a path parameter or a header
Looks up the cell assignment from a fast, cached store
Proxies the request to that cell's ingress
Applies a configurable fallback policy when a cell is unavailable

Keep it as thin as AWS recommends. Build on a battle-tested proxy such as Envoy or NGINX, and add cell-aware routing on top rather than writing a proxy from scratch.

Phase 3: provision cells as code

Every cell must be identical by definition. Use Terraform or Pulumi modules that take a cell_id and stamp out the whole stack. Manual drift is a ticking time bomb: if Cell 7's Redis configuration differs subtly from Cell 12's, you will find out at the worst possible moment.

module "cell_12" {
  source  = "./modules/cell"
  cell_id = "cell-12"
  region  = "us-east-1"
  az      = "us-east-1b"

  # Capacity is identical for every cell
  api_instance_count    = 3
  worker_instance_count = 2
  db_instance_class     = "db.r6g.large"
  redis_node_count      = 3
}

Tenant assignments belong in the control plane's data, not in the infrastructure code. Adding a cell should be a one-line change that triggers an automated pipeline. If it takes days of manual work, you will not spin up cells when you need them.

Phase 4: per-cell observability

This is not optional, because it is how the whole design pays off. Tag every metric and log line with cell_id. Build dashboards that show all cells side by side, so a degraded cell stands out like a red square in a grid of green. Alert on per-cell error rates, not just global averages, because global averages are exactly the metric that hides localized failures.

A useful test: if your on-call engineer has to query logs to find out which cell is sick, your observability is not good enough.

Phase 5: build tenant migration tooling

Build the tool that moves a tenant between cells before you need it in an emergency. You will need it when a cell becomes a hot spot, when an enterprise customer asks for data residency, and when you drain a degraded cell. One safe pattern is a dual-write with a delayed read flip:

tenant migration · 6 stages, where reads and writes go

Start dual-writing. All new writes go to both cells. Reads still come from the old cell, the source of truth.
Backfill. Copy the historical data to the new cell, then compare checksums until they match.
Flip reads to the new cell and watch error rates for a few minutes.
Cut over. Stop writing to the old cell and update the cell map so the tenant routes to the new cell.
Clean up the old cell's copy of the data.

The writes in step 1 must be idempotent and ordered, or the backfill can overwrite newer data with older data. Build that guarantee in from the start.

The cost: trade-offs you must accept

Cell-based architecture is not free. Anyone selling it without discussing the costs is doing you a disservice.

You now run N production environments. With 50 cells you have 50 databases to back up, 50 Redis clusters to monitor and 50 queue setups to watch. A schema migration becomes 50 coordinated operations. Infrastructure as code stops being a best practice and becomes a requirement for survival.
Not everything fits in a cell. Authentication, global feature flags, payment methods and anything shared between tenants must live in the control plane or be replicated to each cell. Keep the control plane as thin as possible, because it is a shared dependency and the more it does, the more its failures matter.
Cross-cell queries are gone. A query that used to join across all users now has to fan out to every cell and merge the results, which is slow and complex. The standard answer is a separate analytics pipeline: every cell streams its events to a warehouse, and analytics queries run there instead of against operational databases. That is a good pattern anyway, but cells make it mandatory.
You need spare capacity. As the previous chart showed, each surviving cell needs about 1/(N-1) of spare capacity to absorb a failed neighbor's traffic when load is spread evenly. That is a modest overhead for large N, but real.
Big tenants do not fit neatly. A tenant larger than a typical cell needs a cell of its own, or a larger one.

Challenge	Mitigation	Effort
N environments to manage	Strict infrastructure as code and automated cell provisioning	High
Global data needs	A thin control plane with caching at the cell edge	Medium
Cross-cell analytics	A separate asynchronous analytics pipeline	Medium
Moving tenants between cells	Dual-write migration tooling	High
Very large tenants	Dedicated or larger cells	Low
Spare capacity for failover	Spread load evenly, and use more, smaller cells	Low

Is cell-based architecture right for you?

The pattern is powerful, but it is not for everyone. Adopting it too early is premature optimization. Adopting it too late means living through incidents that cells would have contained.

It fits well when:

You run B2B SaaS with clear tenant boundaries
A single Availability Zone outage is unacceptable
Your volume is high enough that blast radius is a real concern
You have data residency or compliance requirements, such as EU-only data
Your team has strong infrastructure-as-code and platform engineering maturity

It fits poorly when:

You are an early-stage startup still finding product-market fit
Cross-user queries sit at the core of your product
You have a small team with no platform engineering capacity
Your workload is inherently global, such as a CDN or a search index
No natural partition key exists in your data model

Readiness checklist

Be honest with yourself before committing:

I can name a partition key that keeps more than 90% of related data together
All of my production infrastructure is covered by infrastructure as code
I can see per-service and per-dependency health, not only global averages
My organization will fund a platform investment of several months
I know which data is global and cannot live in a cell
I have sized the spare capacity needed to absorb a failed cell
I have, or can build, tooling to move a tenant between cells without downtime

As a rule of thumb, if you cannot tick more than a few of these, the architecture will fight you instead of helping you. Start with the gaps.

Conclusion

Microservices gave us deployment independence and team autonomy. Cells add something more fundamental: failure containment.

Traditional reliability engineering asks how to prevent failures. Cell-based architecture asks a different question: given that failures are inevitable, how far can we stop them from spreading? Cells do not reduce the probability of failure. They reduce its consequence, and at scale, reducing consequences is far more tractable than eliminating causes.

For a smaller company the economics differ, but the pattern scales down gracefully. Even three to five cells for your most critical service can cut your blast radius without a platform-wide overhaul.

The place to start is always the same: find your partition key. Identify the entity around which everything else pivots, and ask whether slicing along that line could save you from your next all-hands incident at 2 a.m.

Sources

Caching, Consistency, and Trade-offs: Designing Scalable Distributed Systems

Muhammad Ahsan Farooq — Sat, 24 Jan 2026 17:00:10 +0000

When building distributed systems, performance rarely comes for free. Every optimisation introduces a trade-off, and nowhere is this more visible than in caching and consistency.

Modern applications — social networks, e-commerce platforms, streaming services — rely heavily on caching to achieve low latency and massive scale. But once data is cached and replicated across systems, a fundamental question arises:

How consistent does the data need to be?

The uncomfortable truth is that there is no universal answer. The right consistency model depends entirely on what your system is doing and what the cost of being wrong actually is. A stale like count on a social post is annoying. A stale account balance during a fund transfer is catastrophic.

This post walks through caching strategies, consistency models, and how production systems at scale actually navigate these trade-offs — not in theory, but in practice.

Why Caching Exists in the First Place

At scale, databases are expensive — both in latency and throughput. A single database can handle tens of thousands of queries per second under ideal conditions. A cached response from Redis or Memcached can handle millions.

Caching exists to:

Reduce database load
Improve response times
Absorb traffic spikes
Enable global scalability

A cache stores frequently accessed data closer to the application, often in-memory, making reads orders of magnitude faster than hitting a database.

But caching also introduces a fundamental challenge: the cache and the database can temporarily disagree. The moment you put data in two places, you have to decide which one is right when they differ — and that question has no clean answer.

That disagreement is where consistency models come into play.

Common Caching Strategies

Caching is not a single technique — it's a family of patterns, each with different trade-offs around freshness, write latency, and failure behavior. Choosing the wrong one for a given workload is one of the most common sources of subtle bugs in distributed systems.

1. Cache-Aside (Lazy Loading)

This is the most widely used caching pattern, and for good reason — it's simple, resilient, and maps naturally to how most applications think about data access.

How It Works

Application checks the cache
If data exists (cache hit) → return it immediately
If not (cache miss) → fetch from database, store in cache, return to caller

Read Flow:

Client → Cache → Hit? → Return data
                ↓ Miss
              Database → Populate cache → Return data

Pros

Simple to implement and reason about
Cache failures don't break the system (you fall back to the database)
Database remains the source of truth at all times
Only data that is actually requested gets cached — no wasted memory on cold data

Cons

Cache misses cause latency spikes, especially under a cold start or after a flush
Cached data can serve stale values between the write and the next TTL expiry
Writes require careful invalidation logic — miss one code path and stale data lingers indefinitely

The cold start problem is real. When a cache-aside system restarts or is flushed, the first wave of requests all miss and hammer the database simultaneously. This is called a thundering herd and can bring a database to its knees if not handled with request coalescing or staggered cache warming.

This pattern naturally produces eventual consistency — cached data lags behind the database by up to one TTL window.

2. Write-Through Cache

Instead of lazily populating the cache on reads, write-through caches keep the cache proactively up-to-date on every write.

How It Works

Every write goes to the cache first
The cache synchronously writes to the database before acknowledging success
Reads always find fresh data in the cache

Write Flow:

Client → Cache → Database (synchronous)
                ↓
          Acknowledge write only after both succeed

Pros

Cache always contains fresh data — reads are both fast and consistent
No stale reads under normal operation
Simpler read logic since cache misses are rare

Cons

Higher write latency — every write pays for two round trips (cache + database)
Cache becomes a critical dependency — if the cache is unavailable, writes fail
You pay the cost of caching even for data that is rarely read

Write-through is particularly well-suited for read-heavy workloads where write latency is acceptable — user profile data, configuration, reference tables. It leans naturally toward strong consistency for reads since the cache and database are kept in sync on every write.

3. Write-Behind (Write-Back) Cache

Write-behind inverts the durability guarantee. Writes are acknowledged immediately after hitting the cache, and the database is updated asynchronously in the background.

How It Works

Client writes to the cache; acknowledgement is immediate
The cache (or a background process) flushes writes to the database asynchronously
Reads are served from the cache while the flush is pending

Write Flow:

Client → Cache → Acknowledge immediately
                ↓ (async, background)
              Database

Pros

Extremely fast writes — no waiting for database round trips
High write throughput — ideal for bursty workloads
Can batch multiple writes to the database for efficiency

Cons

Data loss risk — if the cache fails before flushing, writes are gone
Complex recovery logic — what do you do with unflushed writes after a crash?
The database is temporarily inconsistent with the cache by design

Write-behind is appropriate when throughput matters more than durability — metrics aggregation, clickstream logging, analytics counters. It is rarely appropriate for anything where losing a write would be unacceptable to the user or the business.

The Consistency Question

Once data is cached and replicated across multiple nodes, systems must make an explicit decision:

Should all users see the same data at the same time — or is "eventually correct" good enough?

This is not a philosophical question. It's an engineering constraint with direct consequences for latency, availability, and correctness. The answer differs not just between systems, but often between different features within the same system.

Strong Consistency

What It Means

Strong consistency guarantees that:

Every read returns the most recent write — no exceptions
All users see the same data at the same time, regardless of which replica serves them

From the user's perspective, the system behaves like a single, perfectly synchronized machine. There is no observable "lag" between writing data and reading it back, anywhere in the system.

How Strong Consistency Is Achieved

Guaranteeing this across distributed nodes is non-trivial. Common mechanisms include:

Synchronous replication — every replica must acknowledge a write before the write is considered successful
Distributed consensus protocols — algorithms like Raft or Paxos ensure all nodes agree on the order of operations
Quorum-based reads and writes — in a cluster of N replicas, reads and writes both contact a majority (quorum), ensuring at least one node in the read set always has the most recent write

The cost is visible: writes block until all required replicas confirm. On a healthy network this adds tens of milliseconds. During a network partition or a slow replica, it can cause writes to fail or stall indefinitely.

Pros

Predictable, easy-to-reason-about behavior
No stale reads under any circumstances
Correct by construction — no conflict resolution needed

Cons

Higher write and read latency
Reduced availability during network issues — some operations fail rather than proceeding with stale data
Poor global scalability — coordinating across geographically distributed replicas is slow

When Strong Consistency Is the Right Choice

Strong consistency is essential when the cost of returning incorrect data is unacceptable:

Banking and financial transactions
Payment processing and fund transfers
Inventory reservation (overselling is a real, expensive problem)
Distributed locks and leader election
Medical records and regulated data

In these domains, being slow is always better than being wrong. A user waiting 500ms for a fund transfer to confirm is fine. A user being debited twice because two replicas disagreed is not.

Eventual Consistency

What It Means

Eventual consistency guarantees that:

If no new writes occur, all replicas will eventually converge to the same value
Temporary inconsistencies between replicas are explicitly permitted

The word "eventually" is doing a lot of work here. In practice, well-designed eventually consistent systems converge in milliseconds to seconds — not minutes or hours. But the window exists, and software must be designed to tolerate it.

How Eventual Consistency Works

Eventual consistency is typically achieved through:

Asynchronous replication — writes are acknowledged immediately, replicas catch up in the background
Time-based cache expiration (TTL) — cached values expire and are refreshed from the source on demand
Background synchronization — periodic reconciliation jobs that detect and resolve divergence

Writes return fast. The system reconciles differences later.

A Concrete Example

Picture this: you like a photo on Instagram.

Your like is written to the nearest data center — acknowledged in ~50ms
You immediately see the updated count (your local replica is current)
A user in Tokyo sees the old count for another 200ms while replication catches up
The system converges — both users see the same count

Nothing broke. No data was lost. The system optimized for speed and availability, accepting a brief window of inconsistency as a known and acceptable cost.

Pros

Very low write latency — no waiting for remote replicas
High availability — writes succeed even during partial failures
Excellent global scalability — no cross-region coordination required

Cons

Stale reads are possible, and software must be written to handle this gracefully
More complex conflict resolution when two replicas receive concurrent writes
Harder to reason about system state at any given moment

When Eventual Consistency Is the Right Choice

Eventual consistency is ideal when availability and latency matter more than momentary precision:

Social media feeds and timelines
Like counts, view counts, follower counts
Product recommendations
Search indexes
Analytics dashboards
Content delivery via CDN

For these use cases, serving a value that is 200ms stale has zero impact on the user experience. Serving a 500ms delayed response does.

CAP Theorem: Why This Trade-off Is Inevitable

The CAP Theorem, formulated by computer scientist Eric Brewer, explains why no distributed system can fully escape these trade-offs.

A distributed system can guarantee only two of three properties simultaneously:

Property	What it means
Consistency	Every read returns the most recent write
Availability	Every request receives a response (not an error)
Partition Tolerance	The system continues operating despite network partitions

The critical insight is that network partitions are not optional. Networks fail. Packets drop. Data centers lose connectivity. A distributed system that can't tolerate partitions isn't really distributed — it's just a single node with extra steps.

Since partition tolerance is non-negotiable, the real trade-off is:

CP systems sacrifice availability during a partition — some requests fail rather than return stale data (e.g., ZooKeeper, HBase)
AP systems sacrifice consistency during a partition — requests succeed but may return stale data (e.g., Cassandra, DynamoDB in eventual mode, CouchDB)

⚠️ Important nuance: CAP is often misread as a binary choice. In reality, it describes behavior during a network partition. Most of the time, partitions aren't happening — and during normal operation, well-designed systems can offer both strong consistency and high availability. The trade-off only becomes forced under failure conditions.

Caching-heavy systems almost always choose AP — they prioritize availability. The cost of failing a request is higher than the cost of temporarily serving stale data.

Conflict Resolution: The Hidden Cost of Eventual Consistency

When writes happen concurrently on different replicas — before replication catches up — you get conflicting values. The system has to decide which one wins. This is not a hypothetical edge case; at scale, it happens constantly.

Last-Write-Wins (LWW)

The simplest strategy: the write with the most recent timestamp wins. All other concurrent writes are discarded.

Simple and widely used (Cassandra, Redis)
Dangerous if clocks are not synchronized — clock skew can cause a newer write to be overwritten by an older one
Data loss is possible — a legitimate write can be silently discarded

LWW is appropriate when losing occasional writes is acceptable (counters, metrics) and unacceptable when every write must be preserved (financial records, user content).

Vector Clocks

Instead of trusting wall-clock time, vector clocks track the causal history of each write — which writes each node was aware of when it made its update.

Accurate conflict detection — can identify when two writes are genuinely concurrent vs. when one clearly happened after another
Enables intelligent merging — rather than discarding one write, the system can attempt to merge both
Complex to implement and reason about; adds overhead to every read and write

DynamoDB uses a variant of this approach. It's the right choice when data loss is unacceptable and writes must be reconciled rather than discarded.

Application-Level Merging

For some data structures, domain logic produces a better result than any generic strategy.

Consider a shopping cart: if two sessions concurrently add different items, the correct merge is to include both items — not to pick one and discard the other. This requires the application to understand the semantics of its data, not just timestamps.

Amazon's Dynamo paper describes exactly this approach for shopping carts and it remains a canonical example of why application-level merging is sometimes the only right answer.

Choosing the Right Approach

There is no universally "best" consistency model — only contextual decisions driven by what the cost of being wrong actually is.

Requirement	Best Fit
Financial accuracy and transactions	Strong Consistency
Global low-latency reads	Eventual Consistency
High write throughput	Eventual Consistency
Regulatory compliance	Strong Consistency
User-facing metrics and counters	Eventual Consistency
Inventory management	Strong Consistency
Social feeds and recommendations	Eventual Consistency

The most important architectural insight is that the right answer is almost always hybrid. Real production systems don't pick one model — they apply strong consistency surgically on critical paths and lean on eventual consistency for everything else.

From Theory to Practice: Caching in Real Production Systems

So far, we've discussed why consistency trade-offs exist and how the two models work conceptually. The real challenge begins when these ideas meet production traffic, global users, and failure scenarios.

In real systems, caching is not an optimization — it is a foundational architectural decision. And with caching comes the unavoidable reality: data will be stale sometimes.

The key question is not how to avoid inconsistency — you can't, not at scale. The question is:

Where can we tolerate it, and how do we control it?

The First Rule of Production Caching

Not all data deserves the same consistency guarantees.

Large-scale systems explicitly separate data into two categories:

Critical paths — money, inventory, correctness-sensitive state. Strong consistency. Minimal or no caching.
Non-critical paths — feeds, counters, recommendations, metadata. Eventual consistency. Aggressive caching.

Failing to make this separation explicit is how systems end up with subtle, hard-to-reproduce bugs where a payment succeeds but the balance doesn't update, or an item shows as in-stock after the last unit was sold.

Social Media Platforms: Optimizing for Speed

Typical stack: SQL or NoSQL primary store, Redis or Memcached distributed cache, CDN for static content, async aggregation pipelines for counters.

How caching is applied:

Like counts, view counts, follower counts → aggressively cached, refreshed via TTL and async pipelines
User feeds → precomputed and cached per user, rebuilt on write events
Writes → fast, non-blocking, acknowledged before replication completes

Consistency model: Eventual consistency everywhere.

If a user briefly sees 1,001 likes instead of 1,002, nothing meaningful has gone wrong. The product experience is unaffected. At millions of reads per second, the alternative — synchronous replication before every read — would require orders of magnitude more infrastructure and deliver noticeably worse latency. The trade-off is obvious and correct.

E-commerce: Deliberately Hybrid

E-commerce systems are the canonical example of why a single consistency model is the wrong approach.

Product Catalog:

Cached aggressively, served via cache + CDN
TTL-based invalidation (minutes to hours)
Consistency model: Eventual
Why: A 30-second delay in reflecting a price change is acceptable. No money is at risk.

Inventory & Checkout:

Minimal caching, strong consistency, database-level transactions and locks
Consistency model: Strong
Why: Overselling a product that doesn't exist is operationally costly and a terrible user experience. The cost of being wrong is concrete and immediate.

The same system, running the same business, using two different consistency models for two different features. This is not a compromise — it's the correct design.

Financial & Payment Systems: Correctness Wins, Always

In financial systems, caching is used around the core, never inside it.

Safe to cache:

Exchange rates and reference data (with short TTLs)
User profile metadata
Authentication tokens and session data
Reporting and analytics dashboards

Never cached:

Account balances
Ledger entries
Transaction state
Authorization results during an active transaction

Consistency model: Strong consistency for all money movement; eventual consistency is acceptable only for analytics, historical reporting, and non-transactional reads.

The rule of thumb in fintech is: if getting this wrong could cause a compliance issue, a customer dispute, or a financial loss — it doesn't go in the cache.

Microservices: Caching at Every Layer

In microservice architectures, caching appears at multiple levels simultaneously:

In-process cache — local memory within a single service instance, fastest possible, no network hop
Regional cache — shared Redis or Memcached cluster within a region, shared across service instances
Global CDN / edge cache — geographically distributed, used for public content and API responses

A common pattern: Service A caches responses from Service B, versioned by a hash of Service B's response payload. When Service B's data changes, it emits an event that Service A's consumers use to invalidate the relevant cache entries. This is fast and keeps caches fresh — but it introduces the hardest problem in distributed systems.

Cache Invalidation: Where Systems Actually Break

"There are only two hard things in Computer Science: cache invalidation and naming things."
— Phil Karlton

Most caching failures in production are not caused by cache misses or cold starts. They're caused by incorrect invalidation logic — a code path that writes to the database but forgets to update the cache, or an invalidation event that gets dropped during a deployment, or a race condition between a write and its corresponding cache delete.

The consequences are insidious because stale cache entries don't produce errors — they produce silently wrong data. Users see outdated information. Bugs are hard to reproduce. By the time the on-call engineer is paged, the bad data has been read by thousands of users and the cache has already self-healed via TTL.

Cache Invalidation Strategies That Actually Work

1. Time-Based Invalidation (TTL)

Every cache entry carries an expiration time. After the TTL expires, the next read fetches fresh data from the source and repopulates the cache.

Pros: Simple, predictable, self-healing. Stale data has a bounded lifetime.

Cons: Data can be stale for up to one full TTL window. TTL values require calibration — too short and you lose the performance benefit; too long and stale data lingers.

In production, TTL is mandatory — even when other invalidation strategies are in place. It is the last line of defense. If explicit invalidation fails for any reason, TTL ensures the cache self-corrects eventually.

2. Explicit Invalidation on Writes

When a write occurs, explicitly delete or update the corresponding cache entry so the next read fetches fresh data.

Pros: Fresher data. Lower probability of stale reads for popular entries.

Cons: Complex. Every write path must know which cache entries to invalidate. Miss one and the stale entry persists until TTL. Partial failures (write succeeds, invalidation fails) produce inconsistency.

Rule: Never rely on explicit invalidation alone. Always combine with TTL. Explicit invalidation is an optimization that improves freshness; TTL is the safety net that handles all the cases explicit invalidation misses.

3. Versioned Caching

Instead of deleting cache entries when data changes, include a version number in the cache key. Bumping the version implicitly invalidates all old entries — they become unreachable, not deleted.

Old key: user_profile:v3:user_9821
New key: user_profile:v4:user_9821  ← bumped after a schema change

Pros: No mass deletion operations. No race conditions between invalidation and concurrent reads. Easy rollback — just decrement the version. Clean handling of schema or format changes.

Cons: Higher memory usage — old versioned entries accumulate until evicted. Requires discipline to bump versions consistently.

Large-scale systems often prefer version bumps over explicit invalidation, particularly during deployments or data migrations. The old entries simply age out naturally.

4. Event-Driven Invalidation

Data changes emit events to a message bus (Kafka, SQS, etc.). Consumers subscribe and invalidate the relevant cache entries in near real-time.

Database write → Event published → Cache consumer → Delete/update cache entry

Pros: Near real-time freshness across services. Scales naturally as you add more consumers. Decouples invalidation logic from write logic.

Cons: Requires reliable event delivery — if an event is dropped, the cache entry stays stale until TTL. Eventual consistency by design. Significantly more complex to debug when things go wrong.

This is the backbone of modern event-driven architectures and is commonly used for cross-service cache invalidation in microservice systems. It's powerful, but it introduces exactly the kind of subtle, hard-to-reproduce failure modes that require strong observability to manage.

The Production Rulebook

Distilling everything above into principles that actually hold under production conditions:

TTL is non-negotiable. It is not optional. It is the safety net under every other strategy.
Never rely on a single invalidation strategy. Combine TTL with explicit invalidation or event-driven invalidation for defense in depth.
Strong consistency is a surgical tool. Reserve it for paths where incorrect data has a real, concrete cost.
Eventual consistency dominates read paths. For most data, most of the time, it is the right default.
Design for failure, not perfection. Your cache will serve stale data sometimes. Design for that reality explicitly — through TTLs, idempotent reads, and graceful degradation — rather than pretending it won't happen.
Separate your critical and non-critical paths explicitly. Don't leave this implicit. Document which data can be stale and by how much, and which data must be fresh.

If your cache occasionally serves stale data but your system stays up and your critical paths remain correct, you're doing it right.

Conclusion: Design for Reality, Not Perfection

Caching and consistency are not opposing forces to be resolved. They are complementary tools that, when applied with precision, allow systems to be both fast and correct — just not always both at once, for every piece of data.

Strong consistency offers correctness. Eventual consistency offers scale. The engineer's job is not to pick one forever, but to draw the line carefully: here we need correctness, here we can afford scale.

The fastest, most reliable systems in production don't eliminate consistency trade-offs — they make them explicit, document them, and design around them deliberately.

That intentionality is what separates systems that are fast-and-correct from systems that are just fast until something breaks.

Idempotency: Making Retries Safe in Distributed Systems

Muhammad Ahsan Farooq — Sun, 04 Jan 2026 13:38:56 +0000

Retries are unavoidable in distributed systems, so any endpoint with side effects must survive being called twice. This post shows how to build that with idempotency keys, Redis and a database constraint.

The double-charge problem

Distributed systems fail. Networks drop packets, services time out, containers restart and clients retry. None of this is exceptional; it is the normal operating environment.

The danger starts when a retry runs a side effect twice. Picture a client sending "Charge the customer $50." The server charges the card, then crashes before it can respond. The client gets no confirmation, assumes failure and retries. Without protection, the customer is charged twice.

Client                    Server                   Payment provider
  |                         |                            |
  |-- POST /charge $50 ---->|                            |
  |                         |-- charge $50 ------------->|
  |                         |<-- success ----------------|
  |                         X  crashes before replying   |
  |   ... timeout ...       |                            |
  |-- POST /charge $50 ---->|   (retry)                  |
  |                         |-- charge $50 ------------->|  <- second charge

This is not a corner case. Two facts every backend engineer should internalize:

A timeout is not a failure. It only means no response arrived, not that the work didn't happen.
A missing response is not an unprocessed request. The server may have finished the job perfectly.

Because of that ambiguity, retries are both necessary and dangerous. Idempotency is what makes them safe.

What idempotency means

An operation is idempotent if performing it many times has the same effect as performing it once.

Safe to retry:

SET balance = 100
DELETE /users/123
PUT /profiles/123

These leave the same final state however often they run. A repeated DELETE may return 404 instead of 204, but the state is unchanged, and the state is what counts.

Dangerous to retry:

balance += 100       // run twice, you get +200
POST /orders         // duplicate orders
sendEmail()          // spams the user

Most real-world POST endpoints are not naturally idempotent, so we have to design them to be. There are two routes:

Make the operation naturally idempotent when you can: set absolute values instead of incrementing, use upserts, or let the client choose the resource ID and use PUT.
Add an idempotency key when you can't, which is the case for charging a card, creating an order or sending a message.

The Idempotency-Key pattern

The standard fix, used by payment APIs such as Stripe and Adyen, is the idempotency key. The client attaches a unique key to each logical operation, and the server uses it to recognise repeats.

The client's job

Generate a unique identifier (a UUID v4 works) once per user intent.
Send it in a header such as Idempotency-Key.
Reuse the same key on every retry of that operation, and a new key for a new operation.

The server's job

Atomically claim the key before doing any work.
If the key was already completed, return the stored response without re-running the side effect.
If it is new, run the operation, store the result under the key, and return it.

Here is the full lifecycle, including the lost response:

Client                      Server                        Redis / DB
  |-- POST + key K -------->|                                |
  |                         |-- claim K (SET NX) ----------->|
  |                         |<-- claimed --------------------|
  |                         |   run side effect              |
  |                         |-- save result under K -------->|
  |<-- 201 + result --------|                                |
  |      (response lost)                                     |
  |-- retry, same key K --->|                                |
  |                         |-- claim K -------------------->|
  |                         |<-- already done + result ------|
  |<-- 201 + saved result --|   side effect NOT run again    |

The server can meet four situations, and each needs a defined answer:

Key is new: claim it, run the operation, store and return the result.
Key is complete, same payload: replay the stored response. Adding an Idempotent-Replayed: true header helps debugging.
Key is still in progress: return 409 Conflict. The client should wait briefly and retry.
Key exists but the payload differs: return 422. The client has a bug, because one key must map to one operation.

With this in place, a retry replays the stored result instead of re-running the side effect, so the outcome is effectively once. The next sections add the layers that keep that true when Redis loses data or a request outlives its lock.

Server implementation (Node.js, Express, Redis)

The middleware below handles all four situations from the lifecycle above. It claims the key atomically with SET NX, so there is no gap between "check" and "set" for a concurrent request to slip through.

import { createHash, randomUUID } from 'node:crypto';
import type { NextFunction, Request, Response } from 'express';
import { createClient } from 'redis';

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

const RESULT_TTL_SECONDS = 60 * 60 * 24; // keep finished results for 24 hours
const LOCK_TTL_SECONDS = 60;             // must exceed your slowest request

type Entry =
  | { state: 'processing'; fingerprint: string; token: string }
  | { state: 'completed'; fingerprint: string; statusCode: number; body: unknown };

// Same key + different request means the client has a bug.
const fingerprintOf = (req: Request) =>
  createHash('sha256')
    .update(`${req.method}:${req.originalUrl}:${JSON.stringify(req.body)}`)
    .digest('hex');

// "Try again" answers are not results. Never store them: release the key instead.
const isFinal = (status: number) => status < 500 && ![202, 409, 429].includes(status);

// Only touch the key if we still own it. If our lock expired and a retry
// claimed the key, its token differs and we leave its claim alone.
const OWNED = `local v = redis.call('GET', KEYS[1])
if not v or cjson.decode(v).token ~= ARGV[1] then return 0 end`;
const RELEASE = `${OWNED}
return redis.call('DEL', KEYS[1])`;
const COMPLETE = `${OWNED}
return redis.call('SET', KEYS[1], ARGV[2], 'EX', ARGV[3])`;

export async function idempotency(req: Request, res: Response, next: NextFunction) {
  const key = req.header('Idempotency-Key');
  if (!key || key.length > 255) {
    return res.status(400).json({ error: 'A valid Idempotency-Key header is required' });
  }

  // Scope keys per account (req.user comes from your auth middleware)
  // so two customers can never collide on the same key.
  const cacheKey = `idem:${req.user.id}:${key}`;
  const fingerprint = fingerprintOf(req);
  const token = randomUUID(); // proves which request owns the claim

  try {
    // 1. Atomically claim the key. Only one concurrent request can win.
    const claimed = await redis.set(
      cacheKey,
      JSON.stringify({ state: 'processing', fingerprint, token } satisfies Entry),
      { NX: true, EX: LOCK_TTL_SECONDS },
    );

    // 2. Someone already claimed it: replay, reject or ask the client to wait.
    if (!claimed) {
      const raw = await redis.get(cacheKey);
      if (!raw) return res.status(409).json({ error: 'Please retry' }); // expired in between
      const entry = JSON.parse(raw) as Entry;

      if (entry.fingerprint !== fingerprint) {
        return res
          .status(422)
          .json({ error: 'Idempotency-Key was reused with a different request' });
      }
      if (entry.state === 'processing') {
        return res.status(409).json({ error: 'Request already in progress' });
      }
      res.setHeader('Idempotent-Replayed', 'true');
      return res.status(entry.statusCode).json(entry.body);
    }

    // 3. We own the key. Capture the response so it can be stored and replayed.
    const sendJson = res.json.bind(res);
    res.json = (body: unknown) => {
      const save = isFinal(res.statusCode)
        ? redis.eval(COMPLETE, {
            keys: [cacheKey],
            arguments: [
              token,
              JSON.stringify({
                state: 'completed',
                fingerprint,
                statusCode: res.statusCode,
                body,
              } satisfies Entry),
              String(RESULT_TTL_SECONDS),
            ],
          })
        : // a transient answer is not final: let a retry run again
          redis.eval(RELEASE, { keys: [cacheKey], arguments: [token] });

      // Persist before replying, so any result the client sees can be replayed.
      save
        .catch((err) => console.error('idempotency store failed', err))
        .finally(() => sendJson(body));
      return res;
    };

    next();
  } catch (err) {
    next(err); // Redis unavailable: fail closed rather than risk a duplicate charge
  }
}

Compared with a simple "check the cache, then set it" middleware, this version fixes five weaknesses:

No race window. SET NX makes claiming the key one atomic step, so two simultaneous requests cannot both run.
Payload fingerprint. Reusing a key for a different request is rejected instead of silently returning the wrong response.
Transient answers release the key. A 5xx, 409, 429 or 202 is not a result, so it is never stored and the client's retry gets a real second attempt.
Only the owner releases or completes the key. Each claim stores a random token, and a small Lua script deletes or overwrites the key only if the token still matches. Without it, a request that outlived LOCK_TTL_SECONDS could wipe out a newer retry's claim.
Store, then reply. The result is saved before the response leaves, which closes the gap where a client sees a result that was never recorded.

Two practical notes. The middleware hooks res.json, so every response from an idempotent route, including your error handler's, must go through res.json. A response sent any other way leaves the key held until the lock expires. And on Express 4, async handlers need a wrapper (Express 5 handles this natively), or a thrown error never reaches your error handler and the request hangs.

The charge endpoint, backed by the database

Redis is a fast first line of defence, but it is a cache. Keys expire, and a Redis restart can lose them. For money, the durable guarantee should live in your primary database, where a unique constraint makes a duplicate record impossible:

CREATE TABLE payments (
  id              uuid        PRIMARY KEY,
  account_id      uuid        NOT NULL,
  idempotency_key text        NOT NULL,
  amount_cents    bigint      NOT NULL CHECK (amount_cents > 0),
  status          text        NOT NULL CHECK (status IN ('charging', 'succeeded', 'failed')),
  created_at      timestamptz NOT NULL DEFAULT now(),
  updated_at      timestamptz NOT NULL DEFAULT now(),
  UNIQUE (account_id, idempotency_key)
);

The endpoint inserts first and charges second. The insert is the durable claim on the key. If it conflicts, an earlier attempt already handled this key, and the endpoint either returns that outcome or, if the attempt died mid-charge, resumes it:

import express from 'express';
import { randomUUID } from 'node:crypto';
import { idempotency } from './idempotency';

const app = express(); // Express 5, or wrap async handlers on Express 4
app.use(express.json());
// Auth middleware runs first so that req.user is set.

const LEASE_SECONDS = 30; // longer than your payment provider's timeout

// Charge the provider and record the outcome. Used by first attempts and resumed ones.
async function chargeAndSettle(paymentId: string, amountCents: number, key: string) {
  try {
    // Forward the same key so the provider deduplicates as well.
    await paymentProvider.charge({ amountCents, idempotencyKey: key });
  } catch (err) {
    if (isDefinitiveDecline(err)) {
      // e.g. card declined: retrying cannot help
      await pool.query(
        `UPDATE payments SET status = 'failed', updated_at = now() WHERE id = $1`,
        [paymentId],
      );
      return 'failed';
    }
    throw err; // outcome unknown: leave the row in 'charging' so a retry can resume it
  }
  await pool.query(
    `UPDATE payments SET status = 'succeeded', updated_at = now() WHERE id = $1`,
    [paymentId],
  );
  return 'succeeded';
}

app.post('/api/charge', idempotency, async (req, res) => {
  const { amountCents } = req.body; // integer cents, never floating-point dollars
  if (!Number.isInteger(amountCents) || amountCents <= 0) {
    return res.status(400).json({ error: 'amountCents must be a positive integer' });
  }
  const key = req.header('Idempotency-Key')!;
  const accountId = req.user.id;

  const send = (id: string, status: string, created: boolean) =>
    status === 'failed'
      ? res.status(402).json({ transactionId: id, status })
      : res.status(created ? 201 : 200).json({ transactionId: id, status });

  // 1. Insert the payment, or discover that this key already created one.
  const inserted = await pool.query(
    `INSERT INTO payments (id, account_id, idempotency_key, amount_cents, status)
     VALUES ($1, $2, $3, $4, 'charging')
     ON CONFLICT (account_id, idempotency_key) DO NOTHING
     RETURNING id`,
    [randomUUID(), accountId, key, amountCents],
  );

  if (inserted.rowCount === 1) {
    const paymentId = inserted.rows[0].id;
    const status = await chargeAndSettle(paymentId, amountCents, key);
    return send(paymentId, status, true);
  }

  // 2. The key was used before. Redis may have forgotten it, but the database remembers.
  const { rows } = await pool.query(
    `SELECT id, status, amount_cents FROM payments
      WHERE account_id = $1 AND idempotency_key = $2`,
    [accountId, key],
  );
  const existing = rows[0];

  // pg returns bigint as a string, so convert before comparing.
  if (Number(existing.amount_cents) !== amountCents) {
    return res.status(422).json({ error: 'Idempotency-Key was reused with a different amount' });
  }
  if (existing.status !== 'charging') {
    return send(existing.id, existing.status, false); // already settled: replay the outcome
  }

  // 3. Still 'charging'. Take over only if the previous attempt's lease has expired.
  //    This single UPDATE is the lock: of several simultaneous retries, exactly one matches.
  const claim = await pool.query(
    `UPDATE payments
        SET updated_at = now()
      WHERE id = $1
        AND status = 'charging'
        AND updated_at < now() - make_interval(secs => $2)
      RETURNING id`,
    [existing.id, LEASE_SECONDS],
  );

  if (claim.rowCount === 0) {
    // Another attempt is working on it right now.
    res.setHeader('Retry-After', '5');
    return res.status(409).json({ error: 'Payment is still being processed, retry shortly' });
  }

  const status = await chargeAndSettle(existing.id, amountCents, key);
  return send(existing.id, status, false);
});

app.listen(3000, () => console.log('Server running on port 3000'));

Here pool is your pg connection pool, paymentProvider is your provider's SDK, and isDefinitiveDecline is a small function that recognises your provider's "this can never succeed" errors.

A few details are worth calling out:

A crash after the insert is recoverable. The row stays in charging. Once its lease expires, the next retry takes it over and resumes the charge with the same provider key. Returning the stale row as if it were finished would report success for a payment that never happened.
The UPDATE is the lock. Two retries arriving together both try to take over the row, but Postgres applies the updates one at a time. The first moves updated_at forward and wins. The second no longer matches the lease condition and gets 409. Nothing holds a transaction open during the slow network call to the provider.
The lease bounds the damage of a second crash. If the winner also dies, the row becomes claimable again after LEASE_SECONDS. Even if two attempts did overlap, the shared provider key means the card is charged once.
The stored amount wins. A reused key with a different amount is rejected with 422 instead of silently charging a different sum.
Unknown outcomes are retried; known ones are final. A timeout leaves the row in charging so a retry can resume it. A definitive decline is recorded as failed and replayed.
Reconcile the leftovers. If a client never retries, a row can sit in charging indefinitely. A small background job that checks the provider and settles old rows covers that.

With the middleware, the unique constraint, the lease and the forwarded provider key working together, this endpoint is safe to retry.

Client implementation (TypeScript, Axios)

The client plays a crucial role. The key must be generated once per user intent, not once per request attempt. If each retry gets a fresh key, the server sees unrelated requests and the protection is gone.

import axios from 'axios';

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

// Retry only when the outcome is unknown or the server asked us to wait.
function isRetryable(err: unknown): boolean {
  if (!axios.isAxiosError(err)) return false;
  if (!err.response) return true; // network failure or timeout
  const { status } = err.response;
  return status === 409 || status === 429 || status >= 500;
}

export async function charge(amountCents: number, idempotencyKey: string, maxAttempts = 4) {
  for (let attempt = 1; ; attempt++) {
    try {
      const { data } = await axios.post(
        'https://api.example.com/api/charge',
        { amountCents },
        { headers: { 'Idempotency-Key': idempotencyKey }, timeout: 10_000 },
      );
      return data;
    } catch (err) {
      if (!isRetryable(err) || attempt >= maxAttempts) throw err;

      // Exponential backoff with jitter, so retries don't arrive in waves.
      const delay = Math.min(1000 * 2 ** attempt, 8000);
      await sleep(delay * (0.5 + Math.random() / 2));
    }
  }
}

// Create the key when the user clicks "Pay", not inside the retry loop.
// crypto.randomUUID() is global in browsers and Node 19+.
const idempotencyKey = crypto.randomUUID();
await charge(5000, idempotencyKey);

A few rules make this robust:

One key per intent. Keep it in component state or sessionStorage, so a page refresh or a double click reuses it instead of starting a second payment. If the user changes the order, that is a new intent, so create a new key.
Retry only what is safe to retry. Network errors, timeouts, 5xx, 409 and 429 qualify. A 400, 402 or 422 will fail the same way again, so surface it.
Back off with jitter. Fixed-interval retries from many clients can hammer a recovering server all at once.
Disable the button while a request is pending, but never treat that as the protection. The key is the protection.

Edge cases that bite in production

The in-flight race. Request A checks Redis and finds nothing, then request B does the same, and both process the payment. This happens with double clicks and with a client that retries while the first attempt is still running. The fix is the atomic SET NX claim shown earlier. The second request loses the claim and gets 409.
Same key, different payload. Fingerprint the method, URL and body when the key is first used, and reject mismatches with 422. Otherwise a bug in the client silently returns the response for a different operation. The database layer repeats the check on the amount.
Crash mid-request. The processing marker expires after the lock TTL, so a dead request never blocks its key forever. A row left in charging is taken over by the next retry once its lease expires, and the provider key protects the side effect itself.
Slow requests. The lock TTL must be longer than your slowest request, or a second request can claim the key while the first is still running. The ownership token stops the late request from overwriting the newer claim, and the database lease and provider key stop a duplicate charge. For genuinely long jobs, return 202 Accepted with a status URL instead.
Redis loses the key. After a restart or failover, a retry arrives with no Redis entry. The unique constraint on (account_id, idempotency_key) catches it, and the endpoint replays or resumes the existing payment.
Failed requests. A 4xx is deterministic, so replaying it is correct. Transient answers (5xx, 409, 429, 202) are never stored; they release the key so the retry can run again.
Key scope. Namespace keys by account or API client. A global namespace lets one customer's key collide with another's.
Retention. Keep results long enough to cover realistic retry windows. 24 hours is a common default. Clients must never reuse a key after that.
Side effects outside your database. Emails, webhooks and queue messages are not covered by a row constraint. Write them to an outbox table in the same transaction, and have consumers deduplicate on an event ID.

When to use it

You don't need this on every endpoint, but if retries are possible and the operation has side effects, idempotency is not optional. It is mandatory for:

Payments: never charge twice.
Order creation: no duplicate orders or shipments.
Webhooks: providers retry aggressively, so receivers should deduplicate on the event ID.
Mobile clients: unstable networks make retries routine.
Notifications: one email or push per event, not three.

A real-world reference: Stripe

Stripe's API is the best-known example. It assumes network failures are normal and that clients will retry. In API v1, once an endpoint starts executing, Stripe saves the status code and body of the first request for a given key and returns them on every retry, including 500 errors. It compares the parameters of a retry with the original and rejects a mismatch, accepts keys up to 255 characters, and may prune keys once they are at least 24 hours old.

Two of Stripe's choices differ from the middleware above, and both are defensible. Stripe replays server errors, while this middleware releases the key on 5xx. Stripe also doesn't save a result when validation fails or a concurrent request conflicts, because no endpoint ran. What matters is that you pick one behaviour and document it for your API's clients.

Production checklist

Before shipping an idempotent endpoint, confirm that:

The client generates one key per intent and reuses it on every retry.
The server claims the key atomically before doing any work.
A random ownership token ensures a request can only release or complete a claim it still holds.
Keys are scoped per account and limited in length.
A different payload under the same key is rejected, in Redis and in the database.
A unique constraint in the primary database backs the guarantee for money-critical writes.
Interrupted work can be resumed safely: a lease on in-progress rows and the same key forwarded to the downstream provider.
Transient responses (5xx, 409, 429, 202) are never stored, and the behaviour is documented for API consumers.
Every response from an idempotent route goes through res.json, including error responses.
Locks and stored results have TTLs, and the lock TTL exceeds your slowest request.

Conclusion

Idempotency is what turns an unreliable network into a dependable system. It moves correctness out of the network layer, which you can't trust, and into durable storage, which you can. Clients can then retry as aggressively as they need to without fear of duplicate side effects.

Next time you design a POST endpoint, ask yourself:

"What happens if this is called twice?"

If the answer is "disaster", it's time to add an idempotency key.

Updated: thanks to Raza Hussain and Saksham Jain for catching two bugs in the first version. A retry now resumes a payment left in progress by a crash instead of returning it as done, and the Redis claim now uses an ownership token with a compare-and-delete script.

Fanout at Scale: Push vs. Pull Strategies in Distributed Systems

Muhammad Ahsan Farooq — Sun, 21 Dec 2025 12:34:01 +0000

Fanout Push vs. Pull: Solving the Feed Distribution Problem at Scale

Modern systems don’t fail because they can’t store data — they fail because they can’t deliver the right data to the right consumers at the right time.

The fanout problem is one of the most fundamental challenges in distributed systems, especially visible in social media feeds, notification systems, messaging platforms, and event-driven architectures.

At its core, fanout answers one question:

When a single event occurs, how do we efficiently deliver it to millions of consumers?

There are two dominant strategies:

Fanout-on-Write (Push)
Fanout-on-Read (Pull)

Both are valid. Both are widely used. And both come with serious trade-offs.

Let’s break it down.

1. The Actual Fanout Problem

Imagine a social media post:

A user posts one update
That user has 10 followers… or 100 million followers
Each follower expects:
- Low latency
- Personalized feeds
- High availability

The naive solution — “just send the post to everyone” — collapses under:

Write amplification
Storage explosion
Hot partitions
Latency spikes

This is fanout:

One write → many reads

The challenge is deciding when and where that fanout happens.

2. Fanout-on-Write (Push Model)

How It Works

When a user creates content:

The system immediately distributes (pushes) the content to followers
Each follower gets a precomputed feed entry
Reading the feed becomes a simple lookup

User posts → System pushes post to N follower feeds

What Problem It Solves

Ultra-fast feed reads
Predictable latency
Minimal computation during reads

Why It’s Powerful

Reads massively outnumber writes in social systems.
Fanout-on-write moves complexity to write time, making reads cheap.

Core Trade-Offs

Advantage	Cost
Extremely fast reads	Massive write amplification
Simple read queries	Heavy storage usage
Predictable latency	Hot users can overload the system

A celebrity with 100M followers can generate:

100M feed writes for one post

This is not theoretical — it’s an operational nightmare.

3. Fanout-on-Read (Pull Model)

How It Works

Content is written once, stored centrally
When a user opens their feed:

The system fetches recent posts from accounts they follow
Merges, ranks, and filters them at read time

User opens feed → System pulls content from many sources

What Problem It Solves

Eliminates write amplification
Handles massive fanout safely
Simplifies writes

Trade-Offs

Advantage	Cost
Minimal write cost	Complex, expensive reads
Scales well for celebrities	Higher latency
Storage-efficient	Harder to cache

This model pushes complexity to query time.

4. Hybrid Fanout (The Real-World Solution)

Almost no large platform uses pure push or pull.

They use hybrids.

Why?

Because:

Not all users are equal
Not all content deserves precomputation
Not all reads need the same latency

5. How Major Social Media Platforms Solve Fanout

Twitter (X)

Hybrid Fanout Strategy

Push for normal users
Pull for celebrities

Why?

The top ~0.01% of users generate extreme fanout
Pushing their tweets would melt storage and queues

So:

Tweets from celebrities are not pushed
They’re fetched dynamically at read time

This is called Selective Fanout

Facebook

Primarily Fanout-on-Write

Posts are pushed into followers’ feeds
Heavy use of:
- Precomputation
- Ranking models
- Feed materialization

Why it works:

Facebook prioritizes low-latency scrolling
Heavy investment in storage and background processing

They mitigate costs with:

Asynchronous workers
Backpressure controls
Partial fanout (not all posts go to all feeds)

Instagram

Push + Pull Hybrid

Push for normal users
Pull for:
- Celebrities
- Suggested content
- Ads
- Reels

Feed is composed from:

Precomputed feed entries
Dynamically fetched content

Mostly Fanout-on-Write

Connections are limited
Professional graph is smaller and denser
Easier to precompute feeds

TikTok / YouTube Shorts

Primarily Fanout-on-Read

Feed is interest-based, not follower-based
Content is pulled from:
- Recommendation pools
- ML-ranked candidate sets

This model would be impossible with push.

6. Fanout Beyond Social Media (Critical Non-Obvious Use Cases)

Fanout strategies solve problems far beyond feeds.

1. Notifications Systems

Push Fanout

Send alerts to millions of devices
Use:
- Message queues
- Topic-based pub/sub
- Rate limiting

Examples:

Mobile push notifications
Emergency alerts
Trading alerts

2. Event-Driven Architectures

Pull Fanout

Consumers pull from Kafka partitions
Enables:
- Backpressure
- Replayability
- Fault isolation

Kafka is fanout-on-read by design.

3. CDN Cache Invalidation

Push

Purge or update cache globally
Used for:
- Security patches
- Breaking content changes

Pull

Lazy fetching of rarely accessed assets

4. Observability & Metrics

Prometheus: Pull fanout (scraping)
Datadog: Push fanout (agents)

Choice depends on:

Control vs latency
System stability during failures

5. Multiplayer Gaming

Push state updates for:
- Nearby players
- Active sessions
Pull world state on reconnect

Hybrid fanout prevents bandwidth explosion.

6. Financial Market Data

Push for:
- Price ticks
- Trade executions
Pull for:
- Historical data
- Analytics queries

Latency requirements dictate fanout model.

7. Configuration & Feature Flags

Push updates to services for:
- Kill switches
- Emergency rollbacks
Pull periodically for consistency

7. How to Choose the Right Fanout Strategy

Use Fanout-on-Write (Push) When:

Read latency must be minimal
Fanout size is bounded
Storage is cheap
Predictable performance is critical

Use Fanout-on-Read (Pull) When:

Fanout size is unbounded
Writes must be cheap
Consumers vary wildly in demand
Backpressure is required

Use Hybrid When:

You want to survive real-world traffic 😄

8. Final Mental Model

Think of fanout like logistics:

Push = Home delivery (fast, expensive, predictable)
Pull = Warehouse pickup (cheap, flexible, slower)
Hybrid = Amazon Prime 😉

Clustering in Node.js: Scaling Your Applications Like Never Before!

Muhammad Ahsan Farooq — Sun, 29 Jun 2025 07:34:14 +0000

The Cluster Module: Architecture and Core Concepts - Comprehensive Guide

The Node.js cluster module implements a sophisticated multi-process architecture that enables applications to scale across multiple CPU cores. This section provides an in-depth exploration of the fundamental concepts and mechanisms that power Node.js clustering.

Master-Worker Architecture

The cluster module operates on a master-worker process model, which forms the foundation of Node.js horizontal scaling. This architecture separates concerns between process management and application logic, creating a robust and scalable system.

Master Process Responsibilities:

Process Lifecycle Management: The master process acts as a supervisor, creating, monitoring, and restarting worker processes as needed
Load Distribution: Manages incoming connections and distributes them among available workers
Resource Coordination: Handles shared resources and coordinates communication between workers
Health Monitoring: Continuously monitors worker health and performance metrics

Worker Process Characteristics:

Independent Execution: Each worker runs as a completely separate process with its own memory space and event loop
Application Logic: Workers handle the actual business logic, HTTP requests, and application-specific operations
Isolated State: Workers cannot directly share memory or variables, ensuring process isolation and stability
Crash Resilience: If one worker crashes, it doesn't affect other workers or the master process

javascriptconst cluster = require('cluster');
const os = require('os');

if (cluster.isPrimary) {
    console.log(`Master ${process.pid} is running`);

    *// Master process responsibilities*
    const numWorkers = os.cpus().length;
    const workers = new Map();

    *// Spawn workers with metadata tracking*
    for (let i = 0; i < numWorkers; i++) {
        const worker = cluster.fork();
        workers.set(worker.process.pid, {
            worker: worker,
            startTime: Date.now(),
            requestCount: 0,
            lastHealthCheck: Date.now()
        });

        console.log(`Worker ${worker.process.pid} spawned`);
    }

    *// Monitor worker health*
    setInterval(() => {
        workers.forEach((workerInfo, pid) => {
            console.log(`Worker ${pid} - Uptime: ${Date.now() - workerInfo.startTime}ms`);
        });
    }, 10000);

} else {
    *// Worker process - handles application logic*
    const express = require('express');
    const app = express();

    app.get('/', (req, res) => {
        res.json({
            message: 'Hello from worker',
            pid: process.pid,
            uptime: process.uptime()
        });
    });

    app.listen(3000, () => {
        console.log(`Worker ${process.pid} listening on port 3000`);
    });
}

Process Lifecycle Management

Process lifecycle management encompasses the creation, monitoring, and termination of worker processes throughout the application's runtime. This system ensures continuous availability and automatic recovery from failures.

Worker Creation Process:

Fork Operation: The master uses cluster.fork() to create new workers, which internally uses the child_process.fork() method
Environment Inheritance: Workers inherit environment variables and configuration from the master process
Port Sharing: All workers automatically share the same server port without conflicts
Initialization Sequence: Each worker goes through startup phases including module loading, event loop initialization, and application bootstrap

Health Monitoring and Recovery:

Death Detection: The master listens for worker exit events and automatically detects crashes or unexpected terminations
Automatic Restart: Failed workers are immediately replaced with new instances to maintain the desired worker count
Graceful Shutdown: Workers can be gracefully terminated during deployments or scaling operations
Resource Cleanup: Proper cleanup of file descriptors, network connections, and memory when workers terminate

javascriptconst cluster = require('cluster');
const os = require('os');

if (cluster.isPrimary) {
    const maxRestarts = 5;
    const restartWindow = 60000; *// 1 minute*
    const workerRestarts = new Map();

    *// Enhanced worker spawning with restart tracking*
    function spawnWorker() {
        const worker = cluster.fork();
        const pid = worker.process.pid;

        *// Track worker lifecycle events*
        worker.on('online', () => {
            console.log(`✅ Worker ${pid} came online`);
        });

        worker.on('listening', (address) => {
            console.log(`🎧 Worker ${pid} listening on ${address.port}`);
        });

        worker.on('disconnect', () => {
            console.log(`🔌 Worker ${pid} disconnected`);
        });

        worker.on('error', (error) => {
            console.error(`❌ Worker ${pid} error:`, error);
        });

        return worker;
    }

    *// Initial worker spawning*
    for (let i = 0; i < os.cpus().length; i++) {
        spawnWorker();
    }

    *// Enhanced exit handling with restart limits*
    cluster.on('exit', (worker, code, signal) => {
        const pid = worker.process.pid;
        const now = Date.now();

        console.log(`💀 Worker ${pid} died (${signal || code})`);

        *// Track restart attempts*
        if (!workerRestarts.has(pid)) {
            workerRestarts.set(pid, []);
        }

        const restarts = workerRestarts.get(pid);
        restarts.push(now);

        *// Remove old restart records outside the window*
        const recentRestarts = restarts.filter(time => now - time < restartWindow);
        workerRestarts.set(pid, recentRestarts);

        *// Restart worker if under limit*
        if (recentRestarts.length < maxRestarts) {
            console.log(`🔄 Restarting worker ${pid}...`);
            spawnWorker();
        } else {
            console.error(`🚨 Worker ${pid} exceeded restart limit, not restarting`);
        }
    });

    *// Graceful shutdown handling*
    process.on('SIGTERM', () => {
        console.log('📴 Master received SIGTERM, shutting down workers...');

        for (const id in cluster.workers) {
            cluster.workers[id].kill('SIGTERM');
        }

        setTimeout(() => {
            process.exit(0);
        }, 10000); *// Force exit after 10 seconds*
    });
}

Inter-Process Communication (IPC)

Inter-Process Communication enables coordination and data exchange between the master and worker processes, as well as between workers through the master as a relay.

IPC Mechanisms:

Message Passing: Processes communicate through JSON message objects sent via the built-in IPC channel
Event-Driven Communication: Both master and workers can emit and listen for custom events
Bidirectional Communication: Messages can flow from master to workers and vice versa
Asynchronous Operations: All IPC operations are non-blocking and asynchronous

Common IPC Use Cases:

Configuration Updates: Dynamically updating worker configuration without restarts
Shared State Synchronization: Coordinating shared data across workers
Performance Metrics: Collecting and aggregating performance data from workers
Graceful Operations: Coordinating graceful shutdowns, deployments, and scaling operations

javascriptconst cluster = require('cluster');

if (cluster.isPrimary) {
    const sharedState = {
        totalRequests: 0,
        activeUsers: new Set(),
        systemStatus: 'healthy'
    };

    *// Spawn workers*
    for (let i = 0; i < 4; i++) {
        const worker = cluster.fork();

        *// Listen for messages from workers*
        worker.on('message', (message) => {
            handleWorkerMessage(worker, message);
        });
    }

    function handleWorkerMessage(worker, message) {
        switch (message.type) {
            case 'request-count':
                sharedState.totalRequests += message.count;
                console.log(`Total requests: ${sharedState.totalRequests}`);
                break;

            case 'user-login':
                sharedState.activeUsers.add(message.userId);
                *// Broadcast to all workers*
                broadcastToWorkers({
                    type: 'user-status-update',
                    activeUserCount: sharedState.activeUsers.size
                });
                break;

            case 'request-shared-state':
                worker.send({
                    type: 'shared-state-response',
                    data: {
                        totalRequests: sharedState.totalRequests,
                        activeUsers: sharedState.activeUsers.size,
                        systemStatus: sharedState.systemStatus
                    }
                });
                break;
        }
    }

    function broadcastToWorkers(message) {
        for (const id in cluster.workers) {
            cluster.workers[id].send(message);
        }
    }

    *// Periodic system updates*
    setInterval(() => {
        broadcastToWorkers({
            type: 'system-update',
            timestamp: Date.now(),
            systemStatus: sharedState.systemStatus
        });
    }, 30000);

} else {
    *// Worker process*
    const express = require('express');
    const app = express();

    let localRequestCount = 0;
    let systemInfo = {};

    *// Handle messages from master*
    process.on('message', (message) => {
        switch (message.type) {
            case 'shared-state-response':
                systemInfo = message.data;
                console.log(`Worker ${process.pid} received shared state:`, systemInfo);
                break;

            case 'user-status-update':
                console.log(`Active users: ${message.activeUserCount}`);
                break;

            case 'system-update':
                console.log(`System status: ${message.systemStatus} at ${new Date(message.timestamp)}`);
                break;
        }
    });

    *// Example routes that use IPC*
    app.get('/', (req, res) => {
        localRequestCount++;

        *// Send request count to master every 10 requests*
        if (localRequestCount % 10 === 0) {
            process.send({
                type: 'request-count',
                count: 10
            });
        }

        res.json({
            message: 'Hello from worker',
            pid: process.pid,
            localRequests: localRequestCount,
            systemInfo: systemInfo
        });
    });

    app.post('/login', (req, res) => {
        const userId = req.body.userId;

        *// Notify master of user login*
        process.send({
            type: 'user-login',
            userId: userId
        });

        res.json({ message: 'Login successful', userId: userId });
    });

    app.get('/system-info', (req, res) => {
        *// Request latest shared state from master*
        process.send({ type: 'request-shared-state' });

        *// Return current known state (in production, you'd want to handle async properly)*
        res.json(systemInfo);
    });

    app.listen(3000);
}

Load Balancing Mechanisms

Node.js clustering incorporates sophisticated load balancing strategies to ensure optimal distribution of incoming connections across worker processes.

Built-in Load Balancing Strategies:

Round-Robin (Default): Connections are distributed sequentially across workers in a circular fashion
Operating System Scheduling: On some platforms, the OS kernel handles connection distribution
Custom Load Balancing: Developers can implement custom strategies based on specific requirements

Connection Distribution Process:

Master Process Listening: The master process binds to the specified port and accepts incoming connections
Worker Selection: Based on the load balancing algorithm, the master selects an appropriate worker
Connection Handoff: The connection is passed to the selected worker for processing
Load Monitoring: The system continuously monitors worker loads to optimize distribution

javascriptconst cluster = require('cluster');
const net = require('net');

if (cluster.isPrimary) {
    const workers = [];
    let currentWorkerIndex = 0;
    const workerStats = new Map();

    *// Create workers with stats tracking*
    for (let i = 0; i < 4; i++) {
        const worker = cluster.fork();
        workers.push(worker);
        workerStats.set(worker.process.pid, {
            connections: 0,
            requests: 0,
            cpuUsage: 0,
            memoryUsage: 0
        });

        *// Track worker performance*
        worker.on('message', (message) => {
            if (message.type === 'stats-update') {
                workerStats.set(worker.process.pid, message.stats);
            }
        });
    }

    *// Custom load balancer implementation*
    function selectWorker(strategy = 'round-robin') {
        switch (strategy) {
            case 'round-robin':
                const worker = workers[currentWorkerIndex];
                currentWorkerIndex = (currentWorkerIndex + 1) % workers.length;
                return worker;

            case 'least-connections':
                return workers.reduce((least, current) => {
                    const leastStats = workerStats.get(least.process.pid);
                    const currentStats = workerStats.get(current.process.pid);
                    return currentStats.connections < leastStats.connections ? current : least;
                });

            case 'least-cpu':
                return workers.reduce((least, current) => {
                    const leastStats = workerStats.get(least.process.pid);
                    const currentStats = workerStats.get(current.process.pid);
                    return currentStats.cpuUsage < leastStats.cpuUsage ? current : least;
                });

            default:
                return workers[0];
        }
    }

    *// Custom server with load balancing*
    const server = net.createServer({ pauseOnConnect: true }, (connection) => {
        const selectedWorker = selectWorker('least-connections');

        *// Pass connection to selected worker*
        selectedWorker.send('sticky-session:connection', connection);

        *// Update connection count*
        const stats = workerStats.get(selectedWorker.process.pid);
        stats.connections++;
    });

    server.listen(3000, () => {
        console.log('Master server listening on port 3000');
    });

    *// Performance monitoring*
    setInterval(() => {
        console.log('\n=== Worker Performance Stats ===');
        workerStats.forEach((stats, pid) => {
            console.log(`Worker ${pid}:`, stats);
        });
    }, 10000);

} else {
    *// Worker process*
    const express = require('express');
    const app = express();

    let connectionCount = 0;
    let requestCount = 0;

    *// Handle connections from master*
    process.on('message', (message, connection) => {
        if (message === 'sticky-session:connection') {
            *// Handle the connection*
            const server = net.createServer((socket) => {
                connectionCount++;

                socket.on('close', () => {
                    connectionCount--;
                });

                *// Simple HTTP response*
                socket.write('HTTP/1.1 200 OK\r\n');
                socket.write('Content-Type: application/json\r\n');
                socket.write('\r\n');
                socket.write(JSON.stringify({
                    message: 'Hello from worker',
                    pid: process.pid,
                    connections: connectionCount
                }));
                socket.end();
            });

            server.emit('connection', connection);
            connection.resume();
        }
    });

    *// Regular stats reporting*
    setInterval(() => {
        const cpuUsage = process.cpuUsage();
        const memoryUsage = process.memoryUsage();

        process.send({
            type: 'stats-update',
            stats: {
                connections: connectionCount,
                requests: requestCount,
                cpuUsage: cpuUsage.user + cpuUsage.system,
                memoryUsage: memoryUsage.heapUsed
            }
        });
    }, 5000);

    console.log(`Worker ${process.pid} started`);
}

javascriptconst cluster = require('cluster');
const net = require('net');

if (cluster.isPrimary) {
    const workers = [];
    let currentWorkerIndex = 0;
    const workerStats = new Map();

    *// Create workers with stats tracking*
    for (let i = 0; i < 4; i++) {
        const worker = cluster.fork();
        workers.push(worker);
        workerStats.set(worker.process.pid, {
            connections: 0,
            requests: 0,
            cpuUsage: 0,
            memoryUsage: 0
        });

        *// Track worker performance*
        worker.on('message', (message) => {
            if (message.type === 'stats-update') {
                workerStats.set(worker.process.pid, message.stats);
            }
        });
    }

    *// Custom load balancer implementation*
    function selectWorker(strategy = 'round-robin') {
        switch (strategy) {
            case 'round-robin':
                const worker = workers[currentWorkerIndex];
                currentWorkerIndex = (currentWorkerIndex + 1) % workers.length;
                return worker;

            case 'least-connections':
                return workers.reduce((least, current) => {
                    const leastStats = workerStats.get(least.process.pid);
                    const currentStats = workerStats.get(current.process.pid);
                    return currentStats.connections < leastStats.connections ? current : least;
                });

            case 'least-cpu':
                return workers.reduce((least, current) => {
                    const leastStats = workerStats.get(least.process.pid);
                    const currentStats = workerStats.get(current.process.pid);
                    return currentStats.cpuUsage < leastStats.cpuUsage ? current : least;
                });

            default:
                return workers[0];
        }
    }

    *// Custom server with load balancing*
    const server = net.createServer({ pauseOnConnect: true }, (connection) => {
        const selectedWorker = selectWorker('least-connections');

        *// Pass connection to selected worker*
        selectedWorker.send('sticky-session:connection', connection);

        *// Update connection count*
        const stats = workerStats.get(selectedWorker.process.pid);
        stats.connections++;
    });

    server.listen(3000, () => {
        console.log('Master server listening on port 3000');
    });

    *// Performance monitoring*
    setInterval(() => {
        console.log('\n=== Worker Performance Stats ===');
        workerStats.forEach((stats, pid) => {
            console.log(`Worker ${pid}:`, stats);
        });
    }, 10000);

} else {
    *// Worker process*
    const express = require('express');
    const app = express();

    let connectionCount = 0;
    let requestCount = 0;

    *// Handle connections from master*
    process.on('message', (message, connection) => {
        if (message === 'sticky-session:connection') {
            *// Handle the connection*
            const server = net.createServer((socket) => {
                connectionCount++;

                socket.on('close', () => {
                    connectionCount--;
                });

                *// Simple HTTP response*
                socket.write('HTTP/1.1 200 OK\r\n');
                socket.write('Content-Type: application/json\r\n');
                socket.write('\r\n');
                socket.write(JSON.stringify({
                    message: 'Hello from worker',
                    pid: process.pid,
                    connections: connectionCount
                }));
                socket.end();
            });

            server.emit('connection', connection);
            connection.resume();
        }
    });

    *// Regular stats reporting*
    setInterval(() => {
        const cpuUsage = process.cpuUsage();
        const memoryUsage = process.memoryUsage();

        process.send({
            type: 'stats-update',
            stats: {
                connections: connectionCount,
                requests: requestCount,
                cpuUsage: cpuUsage.user + cpuUsage.system,
                memoryUsage: memoryUsage.heapUsed
            }
        });
    }, 5000);

    console.log(`Worker ${process.pid} started`);
}

Key Components and APIs of the Cluster Module

The cluster module provides a comprehensive set of APIs and properties that enable fine-grained control over the clustering behavior and worker management.

Core Properties and Methods:

cluster.isPrimary / cluster.isMaster:

Purpose: Boolean property that determines if the current process is the master/primary process
Usage: Used to conditionally execute master-specific or worker-specific code
Note: cluster.isMaster is deprecated in favor of cluster.isPrimary

cluster.isWorker:

Purpose: Boolean property indicating if the current process is a worker
Usage: Alternative to checking !cluster.isPrimary

cluster.fork([env]):

Purpose: Spawns a new worker process
Parameters: Optional environment variables object to pass to the worker
Returns: Worker object representing the spawned process

cluster.workers:

Purpose: Hash table containing all active worker objects, indexed by worker ID
Usage: Iterate over workers for management operations

cluster.settings:

Purpose: Configuration object containing cluster settings
Properties: Includes exec, args, silent, and other cluster configuration options

javascriptconst cluster = require('cluster');
const os = require('os');

*// Comprehensive cluster management example*
class ClusterManager {
    constructor(options = {}) {
        this.maxWorkers = options.maxWorkers || os.cpus().length;
        this.restartDelay = options.restartDelay || 1000;
        this.silent = options.silent || false;
        this.workers = new Map();
        this.isShuttingDown = false;
    }

    start() {
        if (cluster.isPrimary) {
            this.startMaster();
        } else {
            this.startWorker();
        }
    }

    startMaster() {
        console.log(`🚀 Master ${process.pid} starting with configuration:`);
        console.log(`   - Max Workers: ${this.maxWorkers}`);
        console.log(`   - Restart Delay: ${this.restartDelay}ms`);
        console.log(`   - Silent Mode: ${this.silent}`);

        *// Configure cluster settings*
        cluster.setupMaster({
            silent: this.silent,
            stdio: ['inherit', 'inherit', 'inherit', 'ipc']
        });

        *// Spawn initial workers*
        for (let i = 0; i < this.maxWorkers; i++) {
            this.spawnWorker();
        }

        *// Handle worker events*
        this.setupEventHandlers();

        *// Setup graceful shutdown*
        this.setupGracefulShutdown();

        *// Worker health monitoring*
        this.startHealthMonitoring();
    }

    spawnWorker() {
        if (this.isShuttingDown) return;

        const worker = cluster.fork({
            WORKER_ID: Date.now(),
            WORKER_START_TIME: Date.now()
        });

        const workerInfo = {
            worker: worker,
            pid: worker.process.pid,
            startTime: Date.now(),
            restartCount: 0,
            isHealthy: true,
            lastResponse: Date.now()
        };

        this.workers.set(worker.id, workerInfo);

        *// Worker-specific event handlers*
        worker.on('online', () => {
            console.log(`✅ Worker ${worker.process.pid} (ID: ${worker.id}) online`);
        });

        worker.on('listening', (address) => {
            console.log(`🎧 Worker ${worker.process.pid} listening on port ${address.port}`);
        });

        worker.on('message', (message) => {
            this.handleWorkerMessage(worker.id, message);
        });

        worker.on('error', (error) => {
            console.error(`❌ Worker ${worker.process.pid} error:`, error);
            workerInfo.isHealthy = false;
        });

        worker.on('disconnect', () => {
            console.log(`🔌 Worker ${worker.process.pid} disconnected`);
        });

        return worker;
    }

    setupEventHandlers() {
        cluster.on('exit', (worker, code, signal) => {
            const workerInfo = this.workers.get(worker.id);
            if (workerInfo) {
                console.log(`💀 Worker ${worker.process.pid} (ID: ${worker.id}) died: ${signal || code}`);
                this.workers.delete(worker.id);

                *// Restart worker after delay if not shutting down*
                if (!this.isShuttingDown) {
                    setTimeout(() => {
                        this.spawnWorker();
                    }, this.restartDelay);
                }
            }
        });

        cluster.on('fork', (worker) => {
            console.log(`🍴 Worker ${worker.process.pid} (ID: ${worker.id}) forked`);
        });

        cluster.on('listening', (worker, address) => {
            console.log(`📡 Worker ${worker.process.pid} listening on ${address.address}:${address.port}`);
        });
    }

    handleWorkerMessage(workerId, message) {
        const workerInfo = this.workers.get(workerId);
        if (!workerInfo) return;

        switch (message.type) {
            case 'health-check-response':
                workerInfo.lastResponse = Date.now();
                workerInfo.isHealthy = message.status === 'healthy';
                break;

            case 'performance-stats':
                console.log(`📊 Worker ${workerInfo.pid} stats:`, message.stats);
                break;

            case 'error-report':
                console.error(`🚨 Error from worker ${workerInfo.pid}:`, message.error);
                break;
        }
    }

    startHealthMonitoring() {
        setInterval(() => {
            this.workers.forEach((workerInfo, workerId) => {
                *// Send health check*
                workerInfo.worker.send({
                    type: 'health-check',
                    timestamp: Date.now()
                });

                *// Check if worker is responsive*
                const timeSinceLastResponse = Date.now() - workerInfo.lastResponse;
                if (timeSinceLastResponse > 30000) { *// 30 seconds*
                    console.log(`⚠️ Worker ${workerInfo.pid} appears unresponsive`);
                    workerInfo.isHealthy = false;
                }
            });
        }, 10000); *// Every 10 seconds*
    }

    setupGracefulShutdown() {
        const gracefulShutdown = (signal) => {
            console.log(`\n🛑 Received ${signal}, initiating graceful shutdown...`);
            this.isShuttingDown = true;

            *// Disconnect all workers*
            this.workers.forEach((workerInfo) => {
                workerInfo.worker.disconnect();
            });

            *// Force shutdown after timeout*
            setTimeout(() => {
                console.log('⚠️ Force shutdown due to timeout');
                process.exit(1);
            }, 30000);

            *// Wait for all workers to exit*
            const checkWorkersInterval = setInterval(() => {
                if (this.workers.size === 0) {
                    clearInterval(checkWorkersInterval);
                    console.log('✅ All workers stopped, master exiting');
                    process.exit(0);
                }
            }, 1000);
        };

        process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
        process.on('SIGINT', () => gracefulShutdown('SIGINT'));
    }

    startWorker() {
        const express = require('express');
        const app = express();

        *// Worker health check handler*
        process.on('message', (message) => {
            if (message.type === 'health-check') {
                process.send({
                    type: 'health-check-response',
                    status: 'healthy',
                    timestamp: Date.now(),
                    pid: process.pid
                });
            }
        });

        *// Performance monitoring*
        setInterval(() => {
            const memUsage = process.memoryUsage();
            const cpuUsage = process.cpuUsage();

            process.send({
                type: 'performance-stats',
                stats: {
                    memory: memUsage,
                    cpu: cpuUsage,
                    uptime: process.uptime()
                }
            });
        }, 30000);

        *// Sample application routes*
        app.get('/', (req, res) => {
            res.json({
                message: 'Hello from clustered worker',
                pid: process.pid,
                workerId: process.env.WORKER_ID,
                uptime: process.uptime()
            });
        });

        app.listen(3000, () => {
            console.log(`🏃 Worker ${process.pid} listening on port 3000`);
        });
    }
}

*// Usage*
const clusterManager = new ClusterManager({
    maxWorkers: 4,
    restartDelay: 2000,
    silent: false
});

clusterManager.start();

These comprehensive descriptions of the cluster module's architecture and core concepts provide a solid foundation for understanding how Node.js clustering works at a deep level. Each component plays a crucial role in creating scalable, fault-tolerant applications that can effectively utilize multi-core systems.

https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/48469493/0987034c-1b0b-4d03-bfcd-02bf1f646b35/paste.txt

DEV Community: Muhammad Ahsan Farooq

LSM-Trees Explained: The Storage Engine Behind Millions of Writes per Second

Why B+ trees struggle with heavy writes

The LSM-tree idea

An analogy: the library cart

Anatomy of an LSM-tree

1. The write-ahead log (WAL)

2. The MemTable

3. SSTables

4. The sparse index and Bloom filters

A worked example: eight writes

The read path and tombstones

Why reads cost more

Deletes and tombstones

Compaction: keeping the files in check

Two main strategies

The three amplifications

Compaction is not free

Where LSM-trees shine, and where they do not

Use cases

Example: sensor readings with automatic expiry

When an LSM-tree is the wrong tool

Case studies

Discord: from Cassandra to ScyllaDB

Netflix: Cassandra for write-heavy data

A practical tuning checklist

What to monitor

Common mistakes

Summary: B+ tree vs LSM-tree

Conclusion

Sources

PostgreSQL Locks and Deadlocks: How They Work and How to Fix Them

The big picture: three lock levels

Table-level locks

Can two sessions hold these at once?

Example: a schema change that blocks reads

Use case: running a migration without an outage

Row-level locks

Example: two writers, one row

The four row-lock modes

Use case: selling the last item in stock

Advisory locks

Example: run a nightly job on one server

Four functions to know

Use cases

Two pitfalls

How deadlocks form

Example: two bank transfers in opposite directions

Where deadlocks hide in real applications

How Postgres detects and breaks a deadlock

What your application sees

Tuning deadlock_timeout

Preventing locks and deadlocks

1. Lock in a consistent order

2. Keep transactions short

3. Retry when a deadlock happens

4. Do not wait when you do not have to

5. Lock only what you need

Diagnosing lock problems in production

Find the blocked queries

Find the blockers

Settings worth reviewing

Reading a deadlock in the server log

Cheat sheet: symptom, cause, fix

Conclusion

Further reading

How Databases Scale to Millions of Reads and Writes per Second

Part 1: Scaling reads

1. Primary-replica replication

2. Multi-layer distributed caching

3. Preventing cache stampedes

4. Indexing and denormalization

Part 2: Scaling writes

1. Sharding (horizontal partitioning)

2. Write-optimized storage engines: LSM-trees vs B-trees

3. Asynchronous buffering and event streaming

4. Connection pooling

Part 3: Consistency, CAP and distributed SQL

AP vs CP systems

Distributed SQL

Tuning `deadlock_timeout`