DEV Community: Aditya Sharma

Processes vs Threads: What's Actually Different?

Aditya Sharma — Tue, 18 Aug 2026 05:30:14 +0000

Introduction

Type python app.py and press enter.

The Python interpreter starts, your code begins executing, and from your perspective, the program is running. But from the operating system's perspective, something more specific just happened. The OS didn't just start "a program." It created a process: a carefully constructed execution environment with its own identity, its own memory, and its own bookkeeping.

Now suppose that program creates a thread. What changes?

Most explanations reach for a glossary. True, and mostly useless. Let's start from the OS's perspective instead.

1. What a Process Actually Is

When the OS creates a process, it's constructing an isolated execution environment from scratch.

Every process gets its own virtual address space: a private range of memory addresses that the process believes it owns entirely. Physical RAM is shared, but the OS and CPU hardware coordinate to give each process the illusion of dedicated memory. From inside the process, another process's memory is simply not visible.

Within that address space lives everything the process needs: compiled code, a heap for dynamically allocated objects, one or more stacks, and global data. The OS tracks resources on the process's behalf and assigns a unique process ID (PID).

A program is a static artifact on disk. A process is a running instance with its own private resources and OS-managed state.

2. What a Thread Actually Is

A process can contain one or more threads: individual sequences of execution that run within the process's environment.

Each thread has its own execution state: its own program counter tracking which instruction it's executing, its own CPU registers, and its own stack for local variables and call tracking.

What threads don't have is their own address space. Every thread inside a process shares the same virtual memory: the same heap, the same global variables, the same open file descriptors. The process is the container; threads are the execution contexts running inside it.

PROCESS
├── Virtual address space
├── Code, Heap, Global data
├── Open file descriptors and resources
│
├── Thread A  [program counter, registers, stack]
├── Thread B  [program counter, registers, stack]
└── Thread C  [program counter, registers, stack]

Creating a new process means constructing a new address space and all the OS bookkeeping that comes with it. Creating a new thread is typically cheaper because the address-space infrastructure already exists.

3. Why Sharing Memory Changes Everything

Because threads share an address space, they can read and write the same data with no special mechanism. But that simplicity has a catch.

Suppose two threads both increment a shared counter:

counter = 0
# Thread A and Thread B both run:
counter += 1

Counter should end up as 2. But it might not. counter += 1 looks like one operation. It involves at least three steps: read the value, add one, write it back. Execution can be interleaved between these steps.

Thread A: reads counter (gets 0)
Thread B: reads counter (gets 0)   <-- before A writes back
Thread A: adds 1, writes 1
Thread B: adds 1, writes 1         <-- overwrites A's result

Counter ends up as 1. This is a race condition: a bug that depends on the timing of concurrent operations, non-deterministic and hard to reproduce.

The fix is a lock (or mutex):

import threading
lock = threading.Lock()

def increment():
    global counter
    with lock:
        counter += 1

Threads communicate easily because they share memory. That same shared memory is what makes concurrent programming hard.

4. Why Processes Feel Safer

When two processes run side by side, they typically cannot see each other's memory. If one crashes, the others continue. You get independent failure boundaries.

The trade-off is that inter-process communication requires explicit mechanisms: pipes, sockets, or shared memory regions the OS maps into multiple address spaces. A crash in one worker cannot bring down the others, which is why many server architectures prefer separate processes for independent workloads.

5. Context Switching Isn't Free

Your machine might have eight CPU cores and hundreds of runnable threads. The OS scheduler has to take turns. When it pauses Thread A to run Thread B, it performs a context switch: saving A's program counter, registers, and stack pointer, then restoring B's saved state so B continues where it left off.

The cost goes beyond saving registers. Modern CPUs cache recently accessed data. When the scheduler switches threads, the incoming thread's data probably isn't cached yet, so it pays for cache misses until things warm up.

Creating far more runnable threads than CPU cores doesn't create more parallelism. It creates more scheduling overhead.

This is also where concurrency and parallelism diverge. Concurrency is multiple tasks making progress over time, potentially interleaving on a single core. Parallelism is multiple tasks executing simultaneously on different cores. Threads enable both, but only if the hardware and runtime cooperate.

6. Why Python Has a GIL

If threads share memory and can run concurrently, why can't a Python program create threads to fully utilize every CPU core?

In traditional CPython builds, there's a constraint called the Global Interpreter Lock, or GIL: a mutex that ensures only one thread executes Python bytecode at a time, even on a sixteen-core machine.

Why? CPython's interpreter relies on shared internal state and reference counting. Protecting all of that safely with fine-grained locking would be complex and expensive. The GIL is a coarser solution: one global lock that serializes Python bytecode execution within an interpreter.

The practical consequence: CPU-bound Python threads won't run faster on a multi-core machine in a traditional build. For I/O-bound work, it's different: a thread blocking on a network call releases the GIL while it waits, so other threads can run.

The GIL is a property of CPython, not of the Python language itself. CPython now supports a free-threaded build without the GIL, though the GIL-enabled build remains the default and ecosystem compatibility is still maturing.

Conclusion

When you write threading.Thread(target=fn).start(), the OS allocates a new stack, registers a new execution context with its own program counter and registers, and adds it to the scheduler. That thread shares its process's address space and resources.

When you write subprocess.Popen(cmd), the OS constructs a new virtual address space, assigns a new PID, and starts a completely isolated running instance.

One line of code each. Underneath, the OS is creating two very different execution environments. Understanding that doesn't make the API harder to use. It makes the bugs less mysterious and the design choices more intentional.

Your Database Is Making 4 Promises. Here's What ACID Means.

Aditya Sharma — Mon, 17 Aug 2026 18:23:41 +0000

Introduction

Your program keeps opening transactions. A signup writes a new user row. A checkout debits one account and credits another. A form submission updates three related tables at once. You wrap it all in BEGIN and COMMIT and move on, trusting that the database will handle whatever happens in between.

Most of the time it does. But what is it actually promising you when it handles that? And what does it have to do behind the scenes to keep that promise?

Say a user transfers ₹1,000 from Account A to Account B. The application runs two updates: subtract 1,000 from A, add 1,000 to B. Now say the server crashes right after the first update runs but before the second one does.

Account A: -₹1,000
Account B: +₹0

That money didn't move. It vanished. No error message fixes that, and no user accepts "the server restarted" as an explanation for their missing balance.

This is the exact problem a set of guarantees called ACID was built to solve. Most developers can recite the acronym, Atomicity, Consistency, Isolation, Durability, without being able to explain what any of the four words actually promise, or what the database has to do internally to keep those promises. This article tries to fix that.

1. What Is a Transaction?

Before ACID makes sense, you need to understand what a transaction actually is.

A transaction is a group of one or more database operations treated as a single logical unit of work. Either the whole group succeeds, or none of it does. The bank transfer above is a textbook transaction: two updates that only make sense together.

In SQL, a transaction usually looks like this:

BEGIN;

UPDATE accounts SET balance = balance - 1000 WHERE id = 1;
UPDATE accounts SET balance = balance + 1000 WHERE id = 2;

COMMIT;

BEGIN tells the database "everything from here on is one unit." COMMIT tells it "we're done, make it permanent." If something goes wrong in between, a constraint violation, a crash, the application deciding to cancel, the database can issue a ROLLBACK instead, undoing any changes the transaction made so far. From the outside, it's as if the transaction never ran at all.

That single idea, a transaction either fully happens or fully doesn't, is the seed ACID grows out of.

2. Atomicity: All or Nothing

Atomicity is the guarantee that a transaction's operations are treated as one indivisible unit. If any part fails, the transaction's own effects are rolled back, so it does not leave behind a partial result.

This is what prevents the "money disappeared" scenario. If the second UPDATE never runs, the database rolls back the first one too, so Account A never actually loses its money in any state a user or another transaction can observe.

Worth being precise about what atomicity does and doesn't cover. It guarantees the transaction as a whole is all-or-nothing. It doesn't mean every individual statement behaves identically across every database engine, and it says nothing about what other concurrent transactions can see while this one is running.

That's a different guarantee, and it's the interesting one.

3. Consistency: Preserving the Rules of the Database

This is the word people get wrong most often, usually by simplifying it down to "the database has correct data." That's too vague to be useful.

In ACID, consistency means a successful transaction takes the database from one valid state to another valid state, where "valid" is defined by the constraints and rules you've told the database to enforce. Consistency isn't something the database magically guesses. You define it:

A CHECK constraint that says a balance can't go negative

A foreign key that requires an order's customer_id to reference an actual row in the customers table

A UNIQUE constraint that says two users can't share the same email address

If a transaction would violate any of these, the database refuses to commit it and rolls it back instead. Consistency is really the database enforcing your rules, not the database independently deciding what's correct.

It's also worth separating ACID consistency from a completely different idea that shares the same word: consistency in distributed systems, as in "eventual consistency" or "strong consistency" in something like the CAP theorem. That flavor of consistency is about whether different nodes in a distributed system agree on the current value of a piece of data. ACID consistency is about whether a single transaction preserves your integrity rules. Related in spirit, not the same concept. Conflating the two is one of the more common mistakes developers make when they first hear both terms.

4. Isolation: What Happens When Transactions Overlap

Atomicity and consistency mostly deal with a single transaction. Isolation deals with what happens when multiple transactions run at the same time and touch the same data.

Picture two users trying to buy the last seat on a flight at the exact same moment.

-- Transaction A (User 1)
SELECT seats_available FROM flights WHERE id = 42; -- returns 1

-- Transaction B (User 2), running concurrently
SELECT seats_available FROM flights WHERE id = 42; -- also returns 1

Both transactions see one seat available. Both proceed to book it and decrement the count. If isolation is weak and nothing else steps in, both bookings can go through, and the airline has just sold the same seat twice. Isolation is the guarantee that controls how much concurrent transactions can see and interfere with each other's in-progress work, but it isn't the only thing standing between you and this bug. How the transaction is written, whether the update is conditional on the seat still being available, what constraints exist on the table, and which isolation level you've chosen all play a part. Isolation shapes what a transaction can observe. Whether that's enough to prevent a specific race condition still depends on how you've built the transaction around it.

To understand why isolation is hard, it helps to name the specific ways things go wrong when it's weak.

Dirty Read: Transaction A updates a row but hasn't committed yet. Transaction B reads that uncommitted value. Then Transaction A rolls back. Transaction B now has a value in hand that never actually existed as far as the database's committed history is concerned.

Non-Repeatable Read: Transaction A reads a row. While Transaction A is still running, Transaction B updates that same row and commits. Transaction A reads the row again, in the same transaction, and gets a different value than before, even though it never asked to change anything.

Phantom Read: Transaction A runs a query like SELECT * FROM orders WHERE status = 'pending' and gets a set of rows. Transaction B inserts a new row that matches that condition and commits. Transaction A runs the exact same query again and now sees a row that wasn't there a moment ago.

Anomaly

What changes between two reads

Dirty Read

You read a value that was never committed

Non-Repeatable Read

An existing row's value changes underneath you

Phantom Read

The set of rows matching your query changes underneath you

5. Isolation Levels

Databases don't force you into one fixed behavior. The SQL standard defines isolation levels, essentially different trade-offs between correctness and concurrency.

Read Uncommitted, in the standard, lets transactions see uncommitted changes from other transactions, allowing dirty reads. In practice not every database actually implements it that way. PostgreSQL, for instance, accepts Read Uncommitted as a setting but treats it the same as Read Committed internally, so dirty reads never actually happen there.

Read Committed prevents a transaction from reading another transaction's uncommitted changes, but a value can still change between two reads in the same transaction. Prevents dirty reads, not non-repeatable or phantom reads.

Repeatable Read ensures that repeated reads within the same transaction see a consistent version of the data, so the same row won't appear to change value partway through. Some databases achieve this with locks, others with MVCC snapshots, so the exact mechanism differs, but the guarantee the reader experiences is the same. Prevents dirty and non-repeatable reads.

Serializable is the strongest level. Transactions behave as if they ran one after another, even though they may actually execute concurrently under the hood.

A useful mental model: as you move down this list, the database promises a more stable, more predictable view of the world, and pays for that promise with more locking, more version tracking, or more transactions forced to wait or retry.

One thing worth flagging clearly. These levels are not implemented identically everywhere. PostgreSQL's Repeatable Read is stricter about preventing phantoms than the SQL standard technically requires, because of how its underlying mechanism works. MySQL's InnoDB has its own specific behavior at each level too. If you're building something where these guarantees actually matter, financial systems, inventory management, check your specific database's documentation instead of assuming the textbook definition applies exactly as written.

6. How Databases Actually Provide These Guarantees

So far this has all been about what the guarantees are. Now, briefly, how a database actually delivers on them.

Locks. The simplest mechanism. Before a transaction reads or writes a row, it can acquire a lock on that row. Other transactions wanting conflicting access wait until the lock is released. Straightforward, but it can hurt concurrency, since transactions end up queued behind each other.

MVCC (Multi-Version Concurrency Control). A lot of modern databases, including PostgreSQL and MySQL's InnoDB, lean heavily on MVCC instead of pure locking for reads. Rather than making readers wait for writers, the database keeps multiple versions of a row around, and each transaction sees a consistent snapshot of the data as it looked at some point in time. Reads and writes happen concurrently without blocking each other in many cases, which is a big part of why MVCC databases tend to handle read-heavy workloads well.

Write-Ahead Logging (WAL). Before the database modifies its actual data files, it first writes a record of the intended change to a log on disk. If the database crashes right after a commit but before the change is fully reflected in the data files, it can replay the log on restart and recover the committed change. WAL is one important mechanism many databases use to support durable recovery, not a guarantee of durability all by itself.

Commit and rollback. The mechanisms that actually finalize or undo a transaction, working together with the log and whatever locks or version data the transaction was using.

None of these exist in isolation. A modern relational database may use MVCC to provide consistent views of data, locks to handle specific write conflicts, and WAL or a similar recovery mechanism to support durability.

7. Durability: Surviving the Crash

Durability is the promise that once a transaction has committed, its effects are permanent, even if the server crashes a moment later.

The database tells your application "payment successful." One second later, the server loses power. When it comes back online, that payment needs to still be there. If it isn't, durability has failed, and the consequences, a customer charged with no record of the charge, are exactly the kind of thing that makes people distrust software.

This is where WAL earns its keep. With the appropriate durability settings, the database writes the necessary recovery information to durable storage before acknowledging the commit, so it can replay that log during recovery and restore the committed state, even though the crash happened before the main data files were fully updated. The actual guarantee you get depends on how durability is configured, what storage the log itself sits on, and what kind of failures the database is designed to tolerate. WAL is an important mechanism many databases use to support this, not a guarantee that durability holds on its own.

Worth being honest about the limits here too. Durability is provided within the failure assumptions the database system is designed around. If the physical disk itself is destroyed and there's no replica or backup, no amount of WAL saves you. Durability is a strong guarantee against the kinds of failures the system is built to survive, not an absolute guarantee against every conceivable disaster.

8. Why ACID Gets Harder in Distributed Systems

Everything above gets meaningfully harder once your transaction stops living on a single machine.

A single-node database can coordinate a transaction within one database instance, without having to coordinate the transaction protocol across independent machines and networks. Coordinating a transaction within a single database instance is generally simpler than coordinating the same transaction across independent machines, even though the machinery involved, locking, logging, concurrency control, is already fairly sophisticated on its own.

Now imagine the same transaction needs to touch data spread across three separate database nodes: Node A, Node B, Node C. Each node can fail independently. A network partition can cut Node B off from the other two mid-transaction. A message telling Node C to commit can get delayed or lost. A node can crash after agreeing to commit but before actually doing it. Replication between nodes can lag, so a read on one node doesn't reflect a write that already committed on another.

To be clear: this doesn't mean distributed databases abandon ACID. That claim is simply false. Plenty of distributed and distributed-adjacent databases, think systems like Google Spanner or CockroachDB, do provide real transactional guarantees across machines. What changes is the cost of providing them. Coordinating a commit across multiple independent nodes generally requires mechanisms such as two-phase commit, consensus protocols, or other coordination techniques depending on the architecture, along with extra network round trips and careful handling of partial failure. That coordination adds latency and complexity that simply doesn't exist on a single machine.

This is also the point where distributed systems start forcing explicit trade-offs between consistency, availability, and latency. That's a large topic on its own, the kind of thing the CAP theorem tries to formalize, and it deserves a dedicated article rather than a rushed paragraph here. The point to take away is simpler: distributed ACID is achievable, but it is never free.

9. The Real Cost of ACID

None of these four letters are free.

Atomicity requires the database to track everything a transaction touches, so it can undo all of it if needed. Consistency requires constraints to be checked and enforced on every write. Isolation requires locks, versioning, or both, to control what concurrent transactions can see of each other. Durability requires writing to persistent storage in a careful, ordered way before a commit is acknowledged.

These guarantees interact with each other too. Push isolation higher, toward Serializable, and you generally reduce how much concurrency the database can offer, because more transactions end up waiting on each other or getting rolled back and retried. Spread a transaction across multiple nodes, and you add coordination overhead on top of everything else.

ACID isn't four independent boxes to check. It's a set of promises the database has to actively work to keep, and the effort required scales with how strict you ask it to be.

Conclusion

Think back to that ₹1,000 transfer from the start of this article. You should now be able to trace exactly what protects it at every step.

If the transfer fails halfway through, atomicity rolls it back completely. If two transactions try to touch the same account at the same time, isolation controls what each one is allowed to see and change. Once the transfer commits, durability guarantees it survives a crash a second later. Underneath all of it, consistency makes sure the transaction can't leave the accounts table in a state that violates the rules you defined, like a balance going negative.

If the accounts happened to live on different database nodes entirely, you now also know why that transfer would need more coordination, and more care, to get the same guarantees.

Next time you write BEGIN and COMMIT, remember you're not just grouping a couple of statements together. You're asking the database to keep four separate, genuinely difficult promises, all at once, no matter what fails in the middle.

How Garbage Collection Works: Let's Build One From Scratch

Aditya Sharma — Sat, 15 Aug 2026 06:23:09 +0000

Introduction

Your program keeps creating objects. Every function call, every loop iteration, every parsed JSON response produces new ones. You don't manually delete most of them. You've never written a line of code that says "free this memory now." And yet your application doesn't immediately exhaust all available RAM and crash.

So who cleans everything up?

The answer is a garbage collector, a piece of the runtime that runs quietly in the background, deciding what your program no longer needs and reclaiming that memory for future use. Most developers interact with it only when something goes wrong: an unexpected pause, a memory leak, or an out-of-memory error that shouldn't be happening.

Understanding how it actually works turns those confusing moments into solvable problems. And as a bonus, the core algorithm is simple enough to build yourself. We'll do that by the end of this article.

1. The Memory Problem

Every time your program creates an object, the runtime allocates a chunk of memory to hold it. A string, a dictionary, a class instance: they all need memory, and that memory has to come from somewhere.

The somewhere is a region called the heap, a pool of memory that the program draws from as it runs. When you create an object, the runtime finds a suitable slot in the heap and reserves it. When that object is no longer needed, that slot should be freed so it can be used for something else.

In languages like C, you manage this manually. You allocate memory when you need it, and you free it when you're done. This gives you control, but it creates two classic failure modes. Free memory too early and you have a dangling pointer, a reference to memory that's now being used for something else. Forget to free it at all and you have a memory leak: the program slowly consumes more and more memory until it runs out.

Automatic memory management exists to eliminate these failure modes. Instead of relying on the programmer to track every allocation and release, the runtime watches what the program is doing and cleans up on its behalf. The question is: how does it know what's safe to clean up?

2. The Simplest Idea: Reference Counting

The most intuitive approach is to count how many references point to each object. A reference-counting runtime keeps track of how many references currently point to an object. When that count reaches zero, nothing in the program can reach the object anymore, and the runtime can reclaim its memory immediately.

In CPython, the standard Python implementation, this is the primary mechanism. Every Python object carries a reference count. Each time a new reference to it is created, the count goes up. Each time a reference is removed or goes out of scope, the count goes down. When it hits zero, the memory is released on the spot, without waiting for a separate collection phase.

# Python exposes reference counts via sys.getrefcount()
import sys

x = []
print(sys.getrefcount(x))  # Higher than you might expect: getrefcount
                            # itself creates a temporary reference.

y = x
print(sys.getrefcount(x))  # One higher than before: y is a new reference.

del y
print(sys.getrefcount(x))  # Back to what it was: y's reference is gone.

The exact numbers vary depending on context and interpreter internals, so don't rely on specific values. What matters is the pattern: the count goes up when a new reference is created and down when one is removed.

Reference counting is elegant in its simplicity. It distributes the work of garbage collection across the program's normal execution: every assignment and deletion carries a small bookkeeping cost, but there's no separate "stop everything and collect garbage" moment for objects that die by reference count.

There's just one problem.

3. The Problem With Reference Counting

Consider two objects that each hold a reference to the other.

class Node:
    def __init__(self, name):
        self.name = name
        self.other = None

a = Node("A")
b = Node("B")
a.other = b   # A references B
b.other = a   # B references A

del a
del b
# Both reference counts are now 1, not 0.
# Neither object can be freed.

After del a and del b, the variables in your code no longer point to these objects. From your program's perspective, they're gone. But from the runtime's perspective, A still holds a reference to B, and B still holds a reference to A. Each object's reference count is one. Neither will ever reach zero.

This is the circular reference problem. Two objects keeping each other alive, even when the rest of the program has moved on. The memory they occupy is effectively leaked, not because the programmer forgot to delete them, but because pure reference counting has no way to detect or collect cycles.

This is a real limitation in CPython, and it's why CPython includes a second mechanism on top of reference counting: a cyclic garbage collector that periodically searches for isolated reference cycles among container objects and reclaims them. For most programs most of the time, reference counting handles cleanup. The cyclic collector handles the cycles that reference counting cannot. Other Python implementations may handle this differently.

But reference counting is not the only approach to automatic memory management. There's a more general algorithm that handles cycles naturally.

4. Mark and Sweep

Instead of tracking reference counts, tracing collectors ask a different question: starting from what the program can currently access, what objects can be reached?

Mark-and-sweep is one concrete way to implement this idea. The idea begins with a concept called roots: the starting points of the reachability search. Roots are the references the program can directly access at a given moment: local variables in active stack frames, global variables, static fields. Anything the program holds onto directly.

From the roots, the collector follows every reference it can find. Object A points to B, so B is reachable. B points to C, so C is reachable. The collector visits every object it can reach and marks it.

Root --> A --> B --> C     (all marked: reachable)

D --> E                    (unmarked: unreachable, garbage)

Then comes the sweep: the collector walks through all known objects. Any object that wasn't marked is unreachable. The program can never access it again. Its memory can be safely reclaimed.

Mark-and-sweep handles circular references cleanly. If D and E reference each other but nothing reachable points to either of them, neither gets marked. Both get swept. The cycle doesn't protect them.

The trade-off is that mark-and-sweep isn't free. During collection, the collector has to traverse potentially large object graphs. Depending on the implementation, this may require pausing the application while it runs.

5. Why Generational GC Exists

Here's an observation that turns out to be remarkably consistent across real programs: most objects die young.

A temporary variable inside a function lives for a fraction of a second. A string built to format a log message exists for one call and is never seen again. Meanwhile, a database connection pool or a configuration object might live for the entire lifetime of the application.

This pattern, known as the generational hypothesis, is reliable enough that garbage collectors are specifically designed around it. Instead of collecting all objects together, the heap is divided into generations. New objects start in the youngest generation. If an object survives a collection cycle, it gets promoted to an older generation, which is collected less frequently.

The result is that most collection cycles are fast: they sweep through a small, young generation where most objects are already dead, find a lot of garbage quickly, and finish. Long-lived objects in older generations are rarely disturbed.

Modern Java collectors such as G1 and the current generational ZGC use generational techniques, although the exact implementation varies between collectors and JDK versions. Most JavaScript engines use generational collection as well, with young-generation collection ("scavenging") happening frequently and full collections happening rarely.

6. Why GC Can Pause Your Program

A garbage collector needs to understand the state of all objects and all references. If the program is modifying those references while the collector is traversing them, the collector's view of the world becomes inconsistent. An object that was reachable at the start of the traversal might no longer be reachable by the end. A newly allocated object might not get marked at all.

The simplest solution is to pause all application threads while the collector runs: a stop-the-world pause. The application freezes, the collector does its work with a stable view of memory, and then the application resumes. This is reliable but noticeable, especially for latency-sensitive applications.

Modern collectors go to great lengths to reduce or eliminate these pauses. Concurrent collectors do most of their work while the application continues running, using write barriers, small pieces of code inserted around reference assignments, to track changes made during collection. Incremental collectors break up the collection work into small steps interleaved with application execution.

ZGC is designed to perform all expensive collection work concurrently, targeting sub-millisecond pause times even on very large heaps. This is an official design goal stated in the OpenJDK documentation. Shenandoah takes a similar concurrent approach, aiming for consistent short pauses that don't grow with heap size, with a stated goal of keeping pauses under 10ms for large heaps. These are engineering achievements of real sophistication. But the fundamental problem they're solving, making sure the collector and the application agree on the state of the object graph, remains the same.

7. Python, Java, and JavaScript

These three runtimes handle memory management differently, and the differences matter for understanding their behavior.

CPython uses reference counting as its foundation. Most objects are freed the moment their reference count drops to zero, without any collection pause. On top of this, CPython's cyclic garbage collector periodically looks for reference cycles among container objects (lists, dicts, classes, and similar types). The cyclic collector can be tuned or disabled for specific use cases.

Java provides several tracing garbage collectors with different performance characteristics. Objects are typically allocated in a region called the young generation and promoted to older regions if they survive. Modern collectors include G1 (the default since Java 9), ZGC, and Shenandoah, each making different trade-offs between throughput, latency, and pause behavior. Which one is best depends on the application's requirements.

JavaScript is a language with multiple engines. V8, used by Node.js and Chrome, uses a generational collector. Its young generation ("new space") is collected frequently and cheaply. Its old generation is collected by a more comprehensive mark-compact collector. V8 also uses incremental and concurrent collection techniques to reduce pauses during page interactions. Other JavaScript engines may use different implementations.

The common thread across all three: no runtime automatically prevents memory leaks. If your code holds a reference to an object it no longer logically needs, the collector cannot reclaim it. The object is still reachable. From the collector's perspective, it's still alive. This is one of the most important things to understand about garbage collection: reachability and usefulness are not the same thing.

8. Let's Build a Garbage Collector

Enough theory. Let's build one.

We'll implement a simple mark-and-sweep collector in Python. This is an educational implementation: it manages a simulated object graph, not Python's actual memory. Think of it as the algorithm running in plain sight, without any runtime internals in the way.

Step 1: Represent Objects

class GCObject:
    def __init__(self, name):
        self.name = name
        self.references = []  # Other GCObjects this one points to
        self.marked = False

    def __repr__(self):
        return f"GCObject({self.name})"

Each object has a name for identification, a list of outgoing references, and a marked flag that the collector will use.

Step 2: Build an Object Graph

# Create objects
root = GCObject("Root")
a    = GCObject("A")
b    = GCObject("B")
c    = GCObject("C")
d    = GCObject("D")
e    = GCObject("E")

# Connect them
root.references.append(a)   # Root --> A
a.references.append(b)      # A --> B
b.references.append(c)      # B --> C

d.references.append(e)      # D --> E (unreachable from Root)

# All known objects
heap = [root, a, b, c, d, e]

The graph looks like this:

Root --> A --> B --> C      (reachable)

D --> E                     (unreachable, garbage)

Step 3: Define the Roots

roots = [root]

In a real runtime, roots include all local variables in active stack frames and global variables. Here we keep it simple: just one root object.

Step 4: Mark

We start from the roots and visit every reachable object. If we've already marked an object, we don't visit it again (which prevents infinite loops in cyclic graphs).

def mark(obj):
    if obj.marked:
        return  # Already visited
    obj.marked = True
    for ref in obj.references:
        mark(ref)  # Recursively mark everything reachable

This recursive approach works well for our small example. A production implementation would use an explicit stack or queue instead, to avoid hitting Python's recursion limit on deeply nested object graphs.

We call this for each root:

for root_obj in roots:
    mark(root_obj)

After marking, root, a, b, and c all have marked = True. d and e are still False.

Step 5: Sweep

Now we walk the entire heap and reclaim anything that wasn't marked.

def sweep(heap):
    reachable = []
    collected = []

    for obj in heap:
        if obj.marked:
            obj.marked = False  # Reset for the next collection cycle
            reachable.append(obj)
        else:
            collected.append(obj)
            print(f"Collecting: {obj}")

    return reachable

heap = sweep(heap)

Running this prints:

Collecting: GCObject(D)
Collecting: GCObject(E)

Our collector has removed d and e from the simulated heap. Python itself has not reclaimed those objects, because the variables d and e still reference them. This is intentionally a simulation of the algorithm, not a real Python memory manager. root, a, b, and c remain in the heap and are considered live.

The Complete Collector

class GCObject:
    def __init__(self, name):
        self.name = name
        self.references = []
        self.marked = False

    def __repr__(self):
        return f"GCObject({self.name})"


def mark(obj):
    if obj.marked:
        return
    obj.marked = True
    for ref in obj.references:
        mark(ref)


def sweep(heap):
    reachable = []
    for obj in heap:
        if obj.marked:
            obj.marked = False
            reachable.append(obj)
        else:
            print(f"Collecting: {obj}")
    return reachable


def collect(roots, heap):
    for root_obj in roots:
        mark(root_obj)
    return sweep(heap)


# Build the graph
root = GCObject("Root")
a, b, c = GCObject("A"), GCObject("B"), GCObject("C")
d, e    = GCObject("D"), GCObject("E")

root.references.append(a)
a.references.append(b)
b.references.append(c)
d.references.append(e)

heap = [root, a, b, c, d, e]
roots = [root]

print("Before collection:", heap)
heap = collect(roots, heap)
print("After collection:", heap)

Output:

Before collection: [GCObject(Root), GCObject(A), GCObject(B), GCObject(C), GCObject(D), GCObject(E)]
Collecting: GCObject(D)
Collecting: GCObject(E)
After collection: [GCObject(Root), GCObject(A), GCObject(B), GCObject(C)]

That's mark-and-sweep. Mark everything reachable. Sweep everything that isn't. Around forty lines of Python to express the core idea.

9. Our Tiny Collector vs Real Garbage Collectors

What we built captures the essential logic. What it doesn't capture is everything that makes production garbage collectors genuinely hard to build.

Concurrency. Our collector runs while nothing else is happening. A real collector has to contend with application threads that are constantly creating and discarding references, requiring careful synchronization or concurrent traversal techniques.

Compaction. After sweeping, the heap can become fragmented: reachable objects interspersed with holes where garbage used to be. Real collectors often compact the heap, moving live objects together to eliminate fragmentation. This requires updating every reference that points to a moved object.

Generations. Our collector treats all objects equally. Generational collectors maintain separate regions and collect the young generation far more often than the old, because that's where most of the garbage is.

Allocation. We didn't implement object allocation at all. Real runtimes maintain complex free lists or bump-pointer allocators to carve up heap memory efficiently.

Pause reduction. Our collector stops everything while it works. Reducing or eliminating those pauses is one of the central challenges of modern collector design.

The gap between our forty-line implementation and a production garbage collector is vast. But the conceptual foundation is the same: find what's reachable, reclaim what isn't.

Conclusion

We started with a question: if the program doesn't manually free memory, how does the runtime know what it can safely delete?

The answer is reachability. Tracing collectors start from a set of known roots, determine which objects remain reachable, and reclaim those that don't. Mark-and-sweep is one concrete way to implement that idea, and that's what we built.

Reference counting is a simpler approach that works immediately and locally, freeing memory the moment it becomes unreachable. But it fails on circular references, which is why CPython supplements it with a cyclic collector.

Generational collection makes the practical observation that most objects die young and builds the collector around that pattern, spending most of its energy where most of the garbage is.

GC pauses exist because the collector needs a consistent view of the object graph, and modern runtimes invest heavily in making those pauses shorter and less frequent.

And memory leaks can still happen in garbage-collected languages, not because the collector failed, but because the program still holds a reference to something it no longer needs. Reachability and usefulness aren't the same thing. The collector can only see the first one.

The next time you see a GC pause in production, or wonder why a long-running service's memory keeps growing, you now have the mental model to start asking the right questions.

The garbage collector has been doing its job invisibly this whole time. Now you know how.

Why Do Computers Need So Many Ways to Sort?

Aditya Sharma — Fri, 14 Aug 2026 05:16:11 +0000

Introduction

Here's a question that sounds like it should have a simple answer.

We have a list of one million numbers, and we want them in order. Sorting is one of the most studied problems in computer science. Researchers have spent decades on it. We have entire textbooks dedicated to it.

So why don't we just have one sorting algorithm? The best one. Use it everywhere. Done.

The fact that we don't is genuinely interesting. Not because computer scientists couldn't agree, but because "the best sorting algorithm" turns out to be a question that can't be answered without first asking several others.

Best for what data? Best under what constraints? Best when you care about speed, or memory, or predictability, or maintaining the original order of equal elements?

The answer changes every time.

Section 1: The Obvious Answer Isn't Always the Best Answer

Suppose you've never heard of sorting algorithms before and someone hands you a shuffled deck of cards and asks you to sort them. What do you naturally do?

Most people pick up the deck, find the lowest card, pull it out, and start building a new pile. Then they find the next lowest, and the next, until the original deck is empty and the new one is sorted.

This is called selection sort. It's intuitive, easy to implement, and thoroughly mediocre in practice.

The problem is that for every card you place, you have to scan through all the remaining cards to find the minimum. If you have a hundred cards, you scan a hundred, then ninety-nine, then ninety-eight. For a million numbers, that adds up to roughly half a trillion comparisons. It's an O(n²) algorithm, meaning the work grows with the square of the input size.

Double the input, quadruple the work. For large inputs, this becomes genuinely painful.

Okay, so selection sort is slow. Is there a faster approach? Yes. Several, in fact. And each one makes a different bet about what kind of data it's going to see.

Section 2: When the Data Changes, the Best Algorithm Changes

Consider three different lists of numbers:

Random:         [8, 2, 9, 1, 5, 3, 7, 4, 6]
Nearly sorted:  [1, 2, 3, 4, 6, 5, 7, 8, 9]
Already sorted: [1, 2, 3, 4, 5, 6, 7, 8, 9]

These three lists require fundamentally the same result: put the numbers in order. But the most efficient path to that result is different for each one.

An algorithm well-designed for random data might do needless work on data that's already mostly sorted. An algorithm that's brilliant at handling nearly-sorted data might have a catastrophic failure mode when the data is random and adversarial.

This is the core insight of the entire article. Sorting algorithms are not just mathematical curiosities. They are strategies. And like all strategies, they have conditions under which they excel, and conditions under which they struggle.

Section 3: Why Quicksort Is So Fast Despite Its Worst Case

Quicksort is one of the most influential sorting algorithms in computer science, which is surprising when you first hear its story.

The algorithm works by picking a value from the list called a pivot, then rearranging the list so everything smaller than the pivot ends up on its left, and everything larger ends up on its right. Now the pivot is in its final position. Repeat this process recursively on the two halves, and the whole list sorts itself.

[3, 7, 1, 8, 2, 5, 4, 6]
               ^
         pivot = 5 (for example)

After partitioning:
[3, 1, 2, 4] [5] [7, 8, 6]

Recursively sort each half.

When this works well, each partition step cuts the problem roughly in half. Halving the problem repeatedly leads to O(n log n) performance: very fast for large inputs.

But here's the uncomfortable part. Quicksort has an O(n²) worst case. If you consistently pick a bad pivot, say, always the largest or smallest element, the partition step barely reduces the problem. Instead of splitting a thousand-element list into two groups of roughly five hundred, you split it into one of nine hundred ninety-nine and one of one. You've done the work of a partitioning step but made almost no progress.

This is not theoretical. If you feed a sorted list to a naive quicksort that always picks the first element as the pivot, you get exactly this worst case. Historically, some real-world systems have been brought down by adversarial inputs that deliberately trigger quicksort's worst case.

So why does everyone still use it?

Because on random data, the worst case almost never occurs. The average behavior of quicksort is O(n log n), and the constant factors involved are very small. It makes efficient use of memory, works mostly in-place without needing a separate copy of the data, and has excellent cache behavior because it accesses memory in a reasonably sequential pattern.

Modern implementations address this with smarter pivot-selection strategies or by switching to a different algorithm when the recursion depth suggests a worst case is developing. The specific approach varies, but the goal is the same: make pathological inputs far harder to trigger accidentally, without sacrificing the speed advantage on typical data.

Quicksort is fast in practice because it's designed around the reality of what most data looks like, not the pathological extremes.

Section 4: Why Insertion Sort Refuses to Die

If quicksort is the workhorse of general-purpose sorting, insertion sort is the algorithm that everyone learns first and assumes they've left behind. It's simple, it's O(n²) in the general case, and surely any serious application would never use it.

Except serious applications use it all the time, just not on large random lists.

Insertion sort works the way you might organize a hand of playing cards. You take one card at a time from the unsorted pile and slide it into the correct position in the sorted portion of your hand. Each insertion requires scanning backward through the sorted portion until you find where the new card belongs.

Start: [5, 2, 8, 1, 9]

Take 2: scan left, 5 > 2, shift 5 right → [2, 5, 8, 1, 9]
Take 8: 5 < 8, stop → [2, 5, 8, 1, 9]
Take 1: shift 8, 5, 2 right → [1, 2, 5, 8, 9]
Take 9: 8 < 9, stop → [1, 2, 5, 8, 9]

On random data with millions of elements, insertion sort is genuinely slow. On a ten-element list, it's extremely fast and the implementation overhead of a more sophisticated algorithm would cost more time than it saves. On a nearly-sorted list, insertion sort is remarkable: if most elements are already close to their final positions, each insertion only requires moving backward a step or two. In the best case, an already-sorted list, insertion sort is O(n). It simply confirms each element is in the right place and moves on.

This adaptive behavior, where the algorithm naturally speeds up when the data is already partially ordered, is a property that more sophisticated algorithms often lack or have to work hard to achieve.

Section 5: When Predictability Matters: Merge Sort

Quicksort's average case is excellent. Its worst case is a problem. If you're building software where you need to guarantee behavior regardless of input, that unpredictability is uncomfortable.

Merge sort offers a different bargain. Its worst case and its best case are the same: O(n log n). Always. No matter what the input looks like.

Merge sort works by dividing the list in half, recursively sorting each half, and then merging the two sorted halves back together.

[8, 2, 9, 1, 5, 3]

Split:  [8, 2, 9]   [1, 5, 3]
Sort:   [2, 8, 9]   [1, 3, 5]

Merge: compare fronts, take smaller each time
[1, 2, 3, 5, 8, 9]

The merge step is elegant. Two sorted lists can be merged into one sorted list in a single linear pass: compare the front elements of both lists, take whichever is smaller, and repeat. This is what gives merge sort its reliable O(n log n) behavior.

But that reliability comes with a cost. To merge two halves, you need somewhere to put the merged result while you work. Merge sort requires O(n) additional memory, proportional to the size of the input. For a list of one million numbers, you need memory for roughly another million numbers as working space.

That's not always acceptable. On memory-constrained systems, or when sorting very large datasets, the extra allocation is a real concern.

Merge sort is also notably stable. Stability means that when two elements compare as equal, they keep their original relative order. Whether stability matters depends entirely on what you're sorting.

If you're sorting a list of integers, stability is irrelevant. 5 is 5.

But suppose you're sorting a list of customer records, first by purchase amount, then by customer name. After sorting by name, you sort by purchase amount. If the sorting algorithm is stable, customers with the same purchase amount remain in alphabetical order within their group. If it's unstable, that secondary ordering gets scrambled.

The difference between stable and unstable sorting becomes very concrete, very quickly, when your data has structure.

Section 6: The Real World Is Messy: Meet Timsort

Here's what real-world data actually looks like, more often than computer science textbooks suggest.

It's not purely random. It's not neatly ordered. It's somewhere in between. Data often arrives in chunks that are already partially sorted, maybe a new batch appended to an existing sorted list, or records that were imported in rough order. Within the chaos, there are pockets of order.

Tim Peters noticed this in 2002 while working on Python and designed an algorithm to exploit it. He called it Timsort.

The core idea is to first scan the input for runs: sequences of elements that are already in order (or in reverse order, which can be flipped cheaply). Real data tends to have these. Then, instead of discarding that existing order and sorting from scratch, Timsort preserves the runs and merges them together using merge sort's reliable merging strategy. For any run that's too short to be useful, Timsort uses insertion sort to extend it, because insertion sort is fast on small and nearly-sorted data.

Input: [1, 3, 5, 2, 4, 6, 7, 8, 9]

Timsort identifies runs:
  Run 1: [1, 3, 5]   (already ascending)
  Run 2: [2, 4, 6, 7, 8, 9]   (already ascending)

Merge the runs:
  [1, 2, 3, 4, 5, 6, 7, 8, 9]

The result is an algorithm that has O(n log n) worst-case performance like merge sort, but on data with existing order, it approaches O(n). It's stable. And it exploits the patterns that actually appear in real programs.

Timsort became Python's default sorting algorithm from version 2.3 through 3.11, and Java has used it for sorting object arrays via Arrays.sort() since Java 7. Java's sorting of primitive arrays uses a different approach altogether, so "Java uses Timsort" is only part of the picture. Python later replaced Timsort with Powersort in Python 3.12, but the philosophy remains the same: exploit existing order rather than ignoring it.

Timsort is a useful illustration of a broader point. The algorithm that wins in the real world isn't necessarily the one with the most beautiful theoretical properties. It's the one that accurately models how data actually behaves.

Section 7: What "Fast" Actually Means

We've been casually throwing around phrases like O(n log n) and O(n²). These are useful for comparing how algorithms scale, but they don't tell the complete story of practical performance.

Big O notation describes asymptotic behavior: how an algorithm's cost grows as the input grows toward infinity. It intentionally ignores constant factors. An algorithm that does one comparison per element is O(n), and so is one that does a hundred comparisons per element. The notation treats them identically.

In practice, those constants matter.

An algorithm with O(n log n) complexity but high constant factors can be slower than an O(n²) algorithm on small inputs. That's one reason insertion sort is competitive for small lists even though its asymptotic behavior is worse than quicksort's.

There's also the question of how algorithms interact with hardware. Modern CPUs don't access memory uniformly. Accessing data that's already loaded in a nearby cache is dramatically faster than fetching data from main memory. Algorithms that access memory in a sequential, predictable pattern tend to benefit from this; algorithms that jump around the data structure unpredictably can suffer badly.

Quicksort's in-place partitioning tends to access memory in a way that hardware caches handle well. Merge sort's behavior depends on the implementation. Algorithms designed without regard to memory access patterns can perform worse than their Big O suggests on real hardware.

None of this means Big O notation is useless. It's the essential first filter. You eliminate clearly bad options, O(n²) for large random inputs, before worrying about these finer points. But after that first cut, the difference between good algorithms and great ones often lies in the details that asymptotic analysis ignores.

Conclusion

The question this article started with was: if sorting just means putting things in order, why do we need so many ways to do it?

The answer is that "sorting" is not one problem. It's a family of problems that share the same goal but differ in everything that matters for choosing a strategy.

The size of the data changes the picture. The existing order of the data changes the picture. Memory constraints change the picture. Whether you need stable sorting changes the picture. Whether you need guaranteed worst-case behavior or merely good average behavior changes the picture.

Insertion sort looks naive until you realize it's faster than everything else on small or nearly-sorted inputs. Quicksort looks fragile because of its worst case until you see how rarely that worst case occurs on real data, and how well its constants compare to the alternatives. Merge sort looks wasteful because of its memory requirement until you need a stable sort with guaranteed performance. Timsort looks complicated until you appreciate that it's not trying to be theoretically elegant: it's trying to win on the data that real programs actually produce.

When you call sorted() in Python or Arrays.sort() on objects in Java, you're not invoking one of these algorithms in isolation. You're invoking years of careful thinking about the actual distribution of real-world data, and a hybrid strategy designed to perform well across all of it.

The deeper lesson isn't about sorting specifically. It's about how good algorithms are designed. Not by finding the most elegant mathematical solution in a vacuum, but by understanding the environment in which the algorithm will actually run, and then building something that fits that environment.

The textbook answer and the real-world answer are often different. The best engineers know which one they need.

High Availability

Aditya Sharma — Thu, 13 Aug 2026 06:19:38 +0000

This is Part 13 and Finale part of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.

We've spent twelve articles asking the same question in different forms.

How do we make the system handle more?

More traffic. More data. More reads. More writes. More users, spread across more of the world, asking for more things, all at once.

Every time we hit a limit, we found what was too concentrated and we distributed it. One server became many. One database became a primary with replicas. One giant dataset became shards. One application became services. Content moved from one origin to edge locations near the users who needed it.

The architecture we've built can scale. It can absorb traffic spikes. It can serve millions of users simultaneously across multiple continents.

But Part 12 ended by asking something we've never asked before.

We've built a system that can handle scale. But what happens when part of it simply stops working?

That's the question this article answers. And it turns out, it changes how you think about everything.

Section 1: What Happens When a Server Dies?

Let's start with the most ordinary failure imaginable.

An application server dies. Not a slow leak, not a gradual degradation. It just disappears. The process stops. The machine goes dark.

If that server is the only one running the application, the answer is simple and bad.

User
  |
  v
Server ❌

The application is down.
Users get errors.
Nothing works until someone notices and restarts it.

But this is actually a problem we already solved, back in Part 4. We added more application servers and put a load balancer in front of them. We weren't thinking about failure at the time. We were thinking about traffic.

              .-- App Server 1 ❌
User --> Load Balancer --> App Server 2
              '-- App Server 3

Server 1 dies. The load balancer, which periodically checks whether each server is healthy, notices that Server 1 isn't responding. It stops sending traffic there. Requests go to Server 2 and Server 3 instead.

The users connected to Server 1 at the moment it failed might experience a brief error or a dropped request. But the overall system keeps running. A few seconds later, everything looks normal again.

This is the most basic form of redundancy: having more than one of something, so that when one fails, the others carry on.

We added redundancy to the application layer for performance reasons. It turns out that same redundancy also makes the system more resilient to failure. The two benefits came together for free.

But redundancy only helps if the thing that fails isn't the only path through the system.

Section 2: The Single Point of Failure

Now let's make the failure harder.

The load balancer is routing traffic across three healthy application servers. The cache is working. The message queue is draining normally.

Then the database fails.

App Server 1 --.
App Server 2 --+--> Database ❌
App Server 3 --'

It doesn't matter that we have three application servers. Every single one of them depends on that database. Without it, they can't read user data. They can't save orders. They can't authenticate logins. They can receive requests just fine. They have nowhere to send them.

Three healthy servers. Zero working application.

This is what engineers call a single point of failure: any component whose failure alone is enough to bring the entire system down.

The question that should now become second nature is this: if this component disappeared right now, would the entire system disappear with it?

Ask it about every component in your architecture.

The load balancer: if there's only one, and it fails, every user is disconnected. Single point of failure.

The database: if there's only one, and it fails, every application server is blind. Single point of failure.

The message queue: if there's only one and it fails, background work stops accumulating and workers have nothing to process.

The cache: less critical, since a failing cache is survivable if the database can absorb the extra reads. But still worth considering.

Single points of failure are everywhere in early architectures, often invisible because we build for the happy path. Everything is fine until it isn't, and then the single point that failed takes everything with it.

Section 3: Don't Let One Failure Take Everything Down

The response to a single point of failure is the same idea as the response to a traffic bottleneck: don't let one instance carry all the weight.

For application servers, we already have this. Multiple servers share the load.

For the database, we also already have this, in a different form. Read replicas were introduced for performance. But a replica is also a copy of the data. If the primary database fails, and there's a replica that's been kept in sync, the situation isn't as permanent as it first appears.

The key question is: what happens to that replica when the primary disappears?

If nothing automated happens, the replica just sits there, receiving no new data, while the application falls over. Engineers have to manually promote the replica to primary, update connection strings, restart services, and redirect traffic. That can take minutes or hours. For most applications, that's an unacceptable outage.

But if the system is designed to handle this automatically, something different happens.

The primary fails. Within seconds, a monitoring process detects the failure. The replica is automatically promoted to become the new primary. Application servers are redirected to the new primary. Traffic resumes.

Users might experience a brief interruption, a few seconds of errors, while the switchover happens. But the application comes back on its own, without anyone being paged at 3am to manually fix it.

This automatic process of moving from a failed component to a healthy replacement is called failover.

Primary Database fails:

Before failover:
  App Servers --> Primary ❌
                  Replica (idle, waiting)

After failover:
  App Servers --> New Primary (promoted from replica)

Failover only works if there's something to fail over to. That means redundancy has to be built in before the failure happens, not scrambled together after. You can't provision a replica in the middle of an outage and expect it to help immediately. The replica has to already exist, already be in sync, and already be ready to take over.

This is what it means to design for failure: not fixing things after they break, but building in the capacity to survive breakage before it occurs.

Section 4: High Availability and Failover

There's a concept that captures this design philosophy in a phrase: high availability.

A system is highly available when it continues to serve users even when individual components fail. Not because those components never fail. Because the system was built assuming they would.

High availability doesn't mean zero downtime. That's an unrealistic standard. Hardware fails. Software has bugs. Networks have hiccups. Deployments go wrong. Claiming a system will never have a moment of unavailability is almost never true.

What high availability does mean is that failures are absorbed. The system detects them, routes around them, and keeps working. Not every failure mode can be absorbed, but the common ones can be designed for.

The practical mechanisms that make this work are ones we've already touched on:

Health checks let the load balancer and other components know which servers are responding and which aren't. A server that stops answering health checks gets marked as unhealthy and taken out of rotation before users are routed to it.

Automatic failover means that when a primary database goes down, a replica takes its place without waiting for human intervention.

Multiple availability zones mean that if an entire data center loses power or network connectivity, services running in other locations continue serving traffic. Your application doesn't have to be in one physical place.

None of these are exotic techniques. They're standard practice in systems designed to stay up. What makes them effective isn't any individual mechanism. It's the mindset that produces them: the deliberate assumption that any given component can fail at any given moment, and the architecture built around that assumption.

Section 5: When the Network Fails Too

So far we've talked about components disappearing entirely. But there's a category of failure that's often harder to handle: partial failure.

A server isn't dead. It's just slow. A network isn't down. It's dropping some packets. A downstream service isn't unavailable. It's responding, but taking five seconds per request instead of fifty milliseconds.

These partial failures are treacherous because they don't trigger the clean detection that a complete outage does. The health check passes. The server responds. But every request that touches it takes five seconds, and those slow requests start backing up, consuming threads, and eventually making the service that depends on it look sick too.

A few techniques exist to contain this kind of failure.

Timeouts are the simplest defense. Instead of waiting indefinitely for a response from a slow service, set a limit. If the response doesn't arrive within 500 milliseconds, give up and return an error. This prevents one slow dependency from holding every request hostage.

Retries can help when a failure is likely to be temporary. A network hiccup that drops one packet is often resolved by trying again. But retries have a catch that matters enough to spend a moment on.

Section 6: Retries Can Make Things Worse

Imagine a user clicks "Pay Now." The Order Service calls the Payment Service. The Payment Service processes the payment successfully and sends a response. That response gets lost somewhere in the network. The Order Service never receives it.

From the Order Service's perspective, the payment request timed out. Should it retry?

If it retries, the Payment Service receives a second request to process the same payment. This time it might succeed and send a response that arrives. The payment is processed and the order goes through. But the user's card was charged twice.

This isn't a theoretical edge case. It happens. And the solution requires thinking carefully about what it means to run an operation more than once.

An operation is idempotent if running it multiple times produces the same result as running it once. Some operations are naturally idempotent. Reading a user's profile twice returns the same profile. But charging a payment card twice doesn't produce the same result as charging it once.

Designing retries safely means designing the operations being retried to be idempotent where possible: using unique identifiers for payment requests so the Payment Service can detect duplicates and refuse to process the same payment twice, even if it receives the request multiple times.

This is the same complexity that Part 12 introduced when services started communicating over a network. The failure-handling layer inherits those problems and has to solve them deliberately.

Recovering from failure is itself a design problem.

A related technique is the circuit breaker. If a downstream service is failing consistently, retrying rapidly can make things worse: flooding an already struggling service with repeated requests. A circuit breaker tracks how often calls to a service are failing. Once failures exceed a threshold, it stops sending requests to that service entirely for a period of time, letting it recover rather than hammering it. When the cooldown expires, it tries again cautiously.

These aren't exotic patterns. They're standard tools for building systems that survive the messiness of real-world distributed operation.

Section 7: Designing for Failure

At this point it might seem like the answer is: add redundancy everywhere, retry everything, set timeouts on every call, add circuit breakers to every dependency.

But redundancy isn't free.

Every replica is another machine to pay for, another machine to keep in sync, another machine to monitor. Running services across multiple availability zones doubles the infrastructure cost. The operational complexity of managing failover, tracking health across many components, and debugging distributed failures across redundant systems is real and significant.

The right question isn't "how do we achieve maximum redundancy?" The right question is "what failure modes does this specific system need to survive, and what's the cost of not surviving them?"

A small internal tool used by twenty people during business hours can tolerate several hours of downtime. The cost of building high availability into it is almost certainly greater than the cost of the occasional outage. A basic health check and a single database is probably fine.

A global e-commerce platform processing payments around the clock cannot afford minutes of downtime without losing significant revenue and user trust. Multi-region failover, replica databases, circuit breakers, and automated recovery are worth the cost because the alternative is worse.

These are two genuinely different answers to the same question, and both are correct for their context.

Designing for failure means honestly assessing what failures your system needs to survive, how much downtime is acceptable, how much data loss is acceptable, and then building exactly enough resilience to meet those requirements. Not less. But not reflexively more, either.

Conclusion

Let's go all the way back to the beginning.

One user. One server. One database.

That was the starting point of this series. A perfectly reasonable starting point for any new application. Simple to understand, simple to build, simple to deploy.

Then the users multiplied, and we started hitting limits.

One server couldn't handle the traffic, so we added more and put a load balancer in front of them.

The database was answering the same questions thousands of times, so we added a cache and stopped making it repeat itself.

The cache needed to stay accurate as data changed, so we built invalidation strategies to keep it honest.

One database couldn't handle all the reads, so we added read replicas and distributed the load.

One database couldn't hold all the data, so we added sharding and split the data across many machines.

Some user requests triggered too much background work, so we added message queues and moved that work off the critical path.

Users in distant regions were waiting too long for content, so we added CDN edge locations and brought the content closer to them.

The application itself became too large and too tightly coupled for many teams to develop independently, so we split it into services that could be deployed and scaled on their own.

And finally, with all those pieces in place, we asked the question that reframes everything: what happens when something fails?

The answer wasn't a new technology. It was a mindset.

Every technique in this series was a response to a specific bottleneck. Load balancers responded to traffic. Caching responded to repeated work. Sharding responded to data volume. Message queues responded to latency. CDNs responded to distance. Microservices responded to organizational coupling.

High availability responds to failure. And it does so by applying the same fundamental principle as every solution before it: don't let any single thing be the only thing standing between your users and a working application.

The complete journey:

One server          -->  Load Balancers
One query, repeated -->  Caching + Invalidation
One database reads  -->  Read Replicas
One database too large --> Sharding
Users waiting       -->  Message Queues
Content too far     -->  CDNs
One codebase        -->  Microservices
Single point of failure --> Redundancy + Failover

Each column on the right isn't a technology to memorize. It's an answer to a specific question that the growing system forced someone to ask.

That's what system design actually is. Not a catalog of tools. Not a collection of patterns to be applied uniformly. It's the practice of asking what happens when this component can't handle what's being asked of it, and then finding the right response.

The questions change as the system grows. But the habit of asking them stays the same.

What happens when one server isn't enough?
What happens when one database can't keep up?
What happens when data outgrows one machine?
What happens when users are too far from the server?
What happens when one application becomes too hard to change?
And finally: what happens when something fails?

Every one of those questions led somewhere. Not to a perfect system, because perfect systems don't exist. But to a better one, more capable of surviving the pressures that scale inevitably brings.

The goal was never to build a system that never fails.

It was to build one that keeps working when it does.

That's where this series ends. And if you've been reading since Part 1, that's also where the real work begins.

Microservices

Aditya Sharma — Wed, 12 Aug 2026 04:39:46 +0000

This is Part 12 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.

Look at what we've built around the application.

The infrastructure distributes traffic, data, background work, and content across dozens of machines. Load balancers route requests across application servers. Caches absorb repeated reads. Read replicas share the database load. Shards divide the data. Message queues decouple slow work from fast responses. CDN edge locations serve static content from nearby.

Every time a new bottleneck appeared, we distributed something. Traffic. Reads. Storage. Work over time. Content over geography.

But Part 11 ended by pointing at something we haven't touched yet.

The application code itself is still one thing.

While the infrastructure around it grew more distributed and sophisticated, the application running on those servers remained a single, unified codebase. One repository. One deployment process. One running process serving every feature the product offers.

For a long time, that was fine. Now it's becoming the problem.

Section 1: When the Application Was Small

It helps to go back to the beginning of the application's life, because the architecture that's now causing problems was originally a perfectly sensible choice.

When the application launched, it had a handful of features: user accounts, order management, payments, search, and notifications. A team of five engineers built it all in one codebase. They deployed it as one unit. When something broke, anyone on the team could find it. When a new feature needed to be added, one person could make the change and understand how it fit into the whole.

The early application:

[ One Codebase ]
  - Users
  - Orders
  - Payments
  - Search
  - Notifications

One team. One deployment. One database.

This is called a monolith: a single application that contains all the business logic and runs as one deployable unit.

The monolith isn't a mistake. It's often exactly the right choice early on. There's nothing to coordinate between services, no network calls between components, no distributed systems problems to manage. A developer can run the entire application on their laptop and understand the whole thing in a few days. Changes are fast, deployments are simple, and debugging is straightforward.

For a small team building a product that's still finding its footing, a monolith is the pragmatic choice.

The problems don't come from the architecture being wrong. They come from the application growing far beyond the scale it was originally designed for.

Section 2: When One Codebase Becomes Too Much

The company grows. Users grow. Features multiply. The engineering team expands from five people to fifty, then to a hundred and fifty.

Now there are separate teams for each major product area: one team working on the payment system, another focused on search, another on notifications, another on the core ordering flow. Each team has its own roadmap, its own priorities, its own release schedule.

But they all share the same codebase.

This is where the friction begins, and it's worth being specific about what that friction actually looks like in practice.

A developer on the payments team makes a change to how the application processes refunds. Before that change can go to production, the entire application has to be tested. Not just the payments code. Everything. Because in a single codebase, a change in one part can have unexpected effects in another. The test suite for the entire application has to run. It takes forty minutes.

Then the deployment. To push the refund fix, the company deploys the entire application. The search features go with it. The notification system goes with it. The user account code goes with it. All of it is deployed as one unit, even though only a few lines of payment code changed.

That deployment carries risk. If something goes wrong with the refund change, it can affect search results, or notification delivery, or the order flow. Features that had nothing to do with refunds can break. The more code that ships in one deployment, the larger the blast radius of anything that goes wrong.

And as the codebase grows, something else happens: it becomes harder to understand. A developer who joins the team to work on search has to navigate a codebase that also contains the entire payment system, the complete notification logic, and everything else. None of that is relevant to their work, but it's all there, and the boundaries between different areas of the code become increasingly blurry.

The dependency problem grows with it. The search code might depend on a utility function in the payments module, not because search needs payments, but because someone took a shortcut years ago and it was never cleaned up. Now a change to payments requires verifying that search still works.

The application isn't necessarily slow. The users might be getting fast responses. By every performance metric, the system might look healthy.

But the teams are slowing down. Deployments are getting more dangerous. The codebase is getting harder to change confidently. The bottleneck is no longer in the infrastructure.

The application itself has become the bottleneck.

Section 3: What If the Application Didn't Have to Be One Thing?

At some point, someone on the team asks the question that reframes the whole problem.

"What if payments didn't have to live in the same codebase as search? What if they didn't have to be deployed together? What if the team working on notifications could release a change without touching anything related to orders?"

Think about what that would require.

The payment functionality would have to be separated into its own independent unit. It would have its own codebase, its own deployment process, its own team ownership. The search functionality would be another independent unit. Notifications another. Orders another. Users another.

Each unit would be responsible for one specific area of the product. And when those units needed to talk to each other, they'd communicate through a well-defined interface, the same way any two separate systems communicate.

BEFORE: One application

[ Monolith ]
  Users + Orders + Payments + Search + Notifications
  |
  Deployed as one unit

AFTER: Separated responsibilities

[ Users     ]   [ Orders    ]   [ Payments  ]
[ Service   ]   [ Service   ]   [ Service   ]

[ Search    ]   [ Notifications ]
[ Service   ]   [ Service       ]

Each deployed independently.

The word for this idea, when taken to a deliberate architectural pattern, is microservices.

Section 4: Meet Microservices

A microservice is a relatively small, independently deployable service responsible for a specific business capability.

Notice what that definition doesn't say. It doesn't specify a maximum file size or line count. It doesn't prescribe exactly how many services an application should have. Those details vary enormously from one company to another.

What the definition does say is that the service is independently deployable and responsible for a specific capability. Those two properties are the point.

Independently deployable means the payments team can ship a change to the Payment Service on Tuesday afternoon without needing to coordinate with the search team, wait for the notification team's release window, or risk breaking the order flow. The deployment is scoped to one service.

Responsible for a specific capability means the service has clear ownership. The Payment Service handles payments. It knows its own data, its own logic, its own dependencies. Other services don't reach into its internals. They request things from it through its interface, and it handles the rest.

In practice, an API gateway often sits in front of the services, acting as the single entry point for incoming requests and routing them to the appropriate service.

                     [ API Gateway ]
                           |
         .-----------------+-----------------.
         |         |           |             |
    [ Users    [ Orders    [ Payments    [ Search
     Service]   Service]    Service]     Service]
                                |
                       [ Notifications
                          Service ]

A user request comes in through the gateway. The gateway determines which service needs to handle it and routes accordingly. The services handle their own domains.

Section 5: The Benefit: Independent Scaling and Deployment

The most immediate practical benefit of this separation is that things which used to be coupled can now move independently.

Consider what happens when the application runs a flash sale. Search traffic spikes dramatically as users browse and compare products. In a monolith, handling that spike means scaling the entire application: more instances of the whole thing, including the payment code, the notification logic, and everything else that isn't under any additional load.

With separate services, the Search Service can be scaled independently. More instances of search spin up to handle the spike. The Payment Service, which isn't receiving unusual traffic, stays as it is. The Notification Service doesn't change. You're not wasting resources scaling code that doesn't need it.

Flash sale traffic spike:

Monolith approach:
  Scale everything x5 (payments, orders, search, notifications, users)
  Most of that capacity is wasted on code that isn't under load.

Microservices approach:
  Scale Search Service x5
  Everything else unchanged.
  Resources go exactly where the demand is.

The deployment story changes too. The payments team can release a fix on Wednesday. The search team can release an improvement on Thursday. The notification team can release a new feature on Friday. Each team operates on its own schedule. A bug in one service doesn't block another team's release. A problem discovered in the search code doesn't delay a critical payment fix.

Over time, this compounds. Teams move faster when their work is truly independent. Ownership becomes clearer. Codebases become smaller and easier to reason about. Onboarding a new engineer to the Payment Service means learning one service, not understanding the entire product.

Section 6: The Trade-off: Now the Network Is Part of Your Application

If microservices were purely beneficial, every application would use them from day one. They're not.

In a monolith, when the order logic needs to call the payment logic, it's a function call. It happens in memory, in the same process, in microseconds. It either works or it doesn't.

In a microservices architecture, that same interaction is a network request.

Monolith:

Order logic --> calls --> Payment function
(in memory, same process, microseconds)

Microservices:

Order Service --> network request --> Payment Service
(crosses a network, takes time, can fail)

And networks fail. That's not a design flaw or an implementation problem. It's a fundamental property of distributed systems. A service can be slow. It can be temporarily unavailable. It can process a request and send a response that never arrives. It can be overloaded and start dropping requests.

Think through what this means for a simple order placement.

A user places an order. The Order Service receives the request. It needs to charge the user, so it calls the Payment Service. The Payment Service is slow right now because of unrelated load. The Order Service waits. How long should it wait before giving up? If it gives up too quickly, it might cancel a payment that was actually being processed. If it waits too long, the user's request hangs.

The Payment Service processes the payment successfully. Before the response travels back to the Order Service, a network hiccup drops the packet. The Order Service never receives confirmation. Does it retry? If it retries, and the payment already went through, the user might be charged twice.

The Order Service calls the Inventory Service to reserve the item. The Inventory Service is down. The order is paid but the inventory isn't reserved. The state across services is now inconsistent.

None of these scenarios existed in the monolith, because none of them could exist. There was no network between the order logic and the payment logic. They lived in the same process.

Microservices don't eliminate this complexity. They introduce it, deliberately, in exchange for the independence they provide. The distributed systems problems of latency, partial failure, retries, and consistency now belong to the application layer, not just the infrastructure layer.

This is the trade-off stated plainly:

Monolith:
  Simpler communication (function calls, not network calls)
  Simpler deployments (one unit)
  Simpler local development (run one thing)
  Coupled scaling (can't scale one part independently)
  Coupled deployments (one change ships everything)
  Coupled teams (changes in one area risk others)

Microservices:
  Independent scaling (scale what needs it)
  Independent deployments (ship one service at a time)
  Independent teams (own your service end to end)
  Network communication (with all its failure modes)
  Distributed consistency challenges
  More complex observability and debugging

Neither column is inherently better. The right choice depends on the specific situation.

Section 7: When Should You Actually Split?

The honest answer is: later than most teams think.

Microservices are often presented as a modern best practice, something sophisticated engineering teams do. That framing leads teams to adopt them too early, before the problems they solve have actually materialized.

A team of five engineers building an early-stage product almost certainly doesn't need microservices. The overhead of coordinating deployments across multiple services, setting up inter-service communication, managing distributed failures, and operating multiple independent codebases will slow them down more than the monolith ever would. The monolith is a feature, not a liability, at that stage.

The signals that suggest a split might be worth the complexity are specific.

Different parts of the application have genuinely different scaling needs, and the cost of scaling everything together is becoming real. Different teams own different areas, and shared deployments are creating real coordination friction. A clear business boundary exists between two areas of the application, and crossing that boundary requires constant negotiation. The risk surface of a single deployment has grown large enough that changes feel genuinely dangerous.

When those things are true, and when the team is large and experienced enough to manage distributed systems complexity, separation starts to pay off.

When those things aren't yet true, a well-organized monolith, with clear internal boundaries and disciplined code ownership, is often the better choice. The goal was never to have microservices. The goal was always to be able to move fast and build reliable software. Microservices are one way to achieve that, in the right circumstances, at the right scale.

Conclusion

Here's the complete picture of what this series has built.

We started with one server. We progressively distributed every part of the system that became a bottleneck.

Traffic was concentrated at one server, so we added load balancers and distributed it across many.
Repeated database work was concentrated at one database, so we added caching and eliminated it.
Read traffic was concentrated at one database, so we added replicas and spread it.
Data was concentrated in one database, so we added sharding and partitioned it.
Slow background work was concentrated in the user's request, so we added queues and moved it out.
Static content was concentrated at one origin server far from users, so we added CDNs and distributed it geographically.
And now, the application logic was concentrated in one codebase and one deployment, so we split it into services that could be developed, deployed, and scaled independently.

The lesson across all twelve parts has been the same. Find what's concentrated. Understand why. Distribute it in the way that fits the problem.

But there's one question this series hasn't asked yet, and it might be the most important one.

We've spent every article asking: how do we make the system handle more?

More traffic. More data. More requests. More users.

Now look at what we've built. Multiple services. Multiple databases. Read replicas. Shards. Queues. Workers. CDN edge nodes. Cache layers.

Dozens of moving pieces, spread across many machines, connected by a network.

What happens when one of those pieces fails?

A database server loses power. A service crashes under unexpected load. A data center loses network connectivity. A disk fills up. A deployment goes wrong and takes a service offline.

We've built a system that can scale. But can it survive?

How do you build a system that keeps working even when parts of it fail?

That question is what Part 13 is about.

Content Delivery Networks

Aditya Sharma — Mon, 10 Aug 2026 04:56:15 +0000

This is Part 11 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.

We've spent the last several articles making things faster by reducing work.

We stopped the database from answering the same question twice. We distributed reads across multiple replicas. We split data across shards so no single machine ever had to hold everything. We moved slow background tasks off the critical path so users don't have to wait for them.

Each of these solutions attacked the same underlying problem: the system was doing more work than it needed to, or concentrating too much work in one place.

But Part 10 ended by pointing at a different kind of problem entirely. One that has nothing to do with work.

A user in Bengaluru, sending a request to a server in Virginia, has to wait for that request to travel halfway around the world and come back. The server might be perfectly healthy. The database might be responding instantly. The cache might be working exactly as intended. And the user still waits.

Because the content is simply too far away.

That's the problem this article is about.

Section 1: When the Server Isn't the Problem

Imagine the application is running well. It's hosted in a data center in Virginia. Most of the early users are in the United States, and for them, the experience is fast. Pages load quickly. Images appear immediately. Everything feels responsive.

Then the application grows. Users in India start signing up. Then Germany. Then Brazil, Singapore, Japan.

And the complaints start coming in.

"The website is slow."
"Images take forever to load."
"It feels sluggish."

The engineering team checks everything they know to check.

CPU usage is normal. The application servers aren't overloaded.

RAM is fine. No memory pressure.

The database is responding quickly. Query times look healthy.

The cache hit rate is high. Most repeated queries aren't even reaching the database.

The message queue is draining normally. Background work isn't backed up.

Nothing is broken. Nothing is overloaded. The system, by every metric the team knows how to measure, looks fine.

And yet users in India are waiting three or four seconds for the page to load.

So the question becomes: if the server is healthy, why does the application still feel slow?

Section 2: Distance Has a Cost

The answer is something that no amount of server optimization can fix: physical distance.

When a user in Bengaluru opens a web page hosted in Virginia, their request has to travel thousands of kilometers across the internet. It hops through cables, routers, undersea fiber lines, and data centers spread across the globe. The server in Virginia receives it, processes it, and sends a response back. That response travels the same distance in reverse.

The server's processing time might be 10 milliseconds. But the round-trip across the network might add another 200 or 300 milliseconds on top of that. Load a page with twenty images, each making its own round trip, and those milliseconds compound into seconds.

User in Bengaluru                         Server in Virginia
      |                                          |
      |--- request (travels ~13,000 km) -------> |
      |                                          | (processes in 10ms)
      |<-- response (travels ~13,000 km) ------- |
      |
      Total experienced latency: 250-350ms per request
      Page with 20 resources: potentially 2-4 seconds

This is network latency, and it is governed by physics. Data can only travel so fast. Light through fiber-optic cable moves at roughly two thirds the speed of light in a vacuum. The distance between Bengaluru and Virginia is real, and it takes real time to cross it.

No code change, no database optimization, no caching strategy, and no queue will change this. You cannot make light travel faster.

Server processing time is only part of the total time a user experiences. The network matters too, and when users are geographically far from the server, the network dominates everything else.

This means that every technique we've discussed so far, as valuable as it is, solves the wrong problem for these users. Their problem isn't that the server is slow. Their problem is that the server is far away.

If you can't bring the user closer to the server, the only remaining option is to bring the server closer to the user.

Section 3: Move the Content Closer

That phrase sounds obvious once you say it, but it has a real implication worth sitting with.

Most of what a web application serves is the same for everyone. A product image is the same image whether it's requested by someone in Berlin or someone in São Paulo. A company's logo is the same file regardless of who's asking for it. The JavaScript that makes the page interactive, the CSS that makes it look right, the fonts that make the text readable: all of that is identical for every single user.

When a user in India requests a product page, the application sends HTML, CSS, JavaScript, a dozen images, a font file, and more. The actual personalized piece of that response, the part that's different for each user, is often a small fraction of the total data being transferred.

The rest is identical content, being sent from the same server in Virginia, to millions of different users spread around the world.

That's where the waste lives. Not in the server working too hard, but in the same static content making the same long journey over and over again, in every direction, to every corner of the globe.

If a thousand users in Germany are all requesting the same product image, why is that image traveling from Virginia to Germany a thousand times? What if it only had to make that trip once, and then could be served from somewhere much closer to Germany for all subsequent requests?

That's the question that leads naturally to the solution.

Section 4: Meet the CDN

A Content Delivery Network, or CDN, is a network of servers distributed around the world. Instead of every user's request traveling all the way to the origin server, users can receive content from a server that's geographically nearby.

These distributed servers are often called edge locations or Points of Presence, sometimes abbreviated as PoPs. A CDN provider might operate hundreds of them, in cities across every continent. The idea is that no matter where a user is in the world, there's an edge location within a short network distance of them.

WITHOUT CDN:

User in India      -->  Internet  -->  Origin Server (Virginia)
User in Germany    -->  Internet  -->  Origin Server (Virginia)
User in Brazil     -->  Internet  -->  Origin Server (Virginia)

(Same content traveling the same long distances, repeatedly)

WITH CDN:

User in India      -->  Edge Location (Mumbai)
User in Germany    -->  Edge Location (Frankfurt)
User in Brazil     -->  Edge Location (São Paulo)

(Content served from nearby, origin server rarely involved)

The edge location has a cached copy of the content. When the user in India requests a product image, they get it from Mumbai, not Virginia. The round trip is a fraction of what it was. The image loads in milliseconds instead of hundreds of milliseconds.

This should feel familiar, because it's an idea you've seen before.

In Part 6, we learned not to make the database answer the same question repeatedly. We put a cache in front of it. Instead of recalculating an answer over and over, we stored it once and handed it out.

The CDN is that exact idea, applied to geography.

Instead of sending the same content across the world over and over, you store it at locations closer to the people who need it. The origin server answers once. The edge location serves the rest.

The content doesn't get faster to generate. It gets faster to receive.

Section 5: What Happens When the Content Isn't There?

The first time a user in Frankfurt requests a file that the Frankfurt edge location has never seen, the CDN doesn't have it yet. That's a cache miss.

When this happens, the edge location does what any cache does on a miss: it goes to fetch the content from the origin server, delivers it to the user, and stores a copy locally for next time.

First request to an edge location (Cache Miss):

User in Frankfurt
      |
      v
Frankfurt Edge Location: "I don't have this."
      |
      v
Origin Server (Virginia): returns the file
      |
      v
Frankfurt Edge Location: stores a copy
      |
      v
User in Frankfurt: receives the file
      (this request was still slow, but it seeded the edge)

All subsequent requests (Cache Hit):

User in Frankfurt
      |
      v
Frankfurt Edge Location: "I have this." --> serves immediately

Every request after the first one is served from Frankfurt. The origin server doesn't get involved again until the cached copy expires.

The CDN sits between users and the origin server, absorbing the majority of requests for static content. The origin server, which might be doing a lot of other work, suddenly has far fewer requests to handle. Not because the content became less popular, but because the edge locations are handling most of it.

This matters especially during traffic spikes. A viral moment where millions of users suddenly request the same image or video doesn't flood the origin server. The edge locations absorb it. Each edge location serves its local cluster of users from its own cache, and the origin server sees only a fraction of the total traffic.

Section 6: The Trade-off: Freshness vs Distance

By this point in the series, you already know what's coming.

Every caching strategy introduces the same fundamental tension: the faster you serve content, the more you risk serving content that's no longer current.

CDNs are no different.

Imagine a company updates its website's logo. The new logo file is uploaded to the origin server. But edge locations around the world still have the old logo cached. Users in Tokyo, Lagos, and Buenos Aires continue to see the old version until the cached copy at each edge location expires and the new one is fetched.

For a logo change, that's a minor inconvenience. For a critical bug fix in a JavaScript file, it could mean users in some regions are running broken code for hours after the fix was deployed.

This is the same cache invalidation problem from Part 7, now playing out at a global scale across dozens of edge locations instead of a single cache layer.

CDN providers offer ways to handle this. You can set expiration times on content, telling the edge location how long to hold onto a cached copy before checking for a fresher version. You can issue a purge command that forces edge locations to drop their cached copies immediately, so the next request triggers a fresh fetch from the origin.

But there's a more fundamental limit to where a CDN helps.

Everything we've discussed so far assumes the content is the same for everyone. Static files, images, shared web pages: these are natural fits for edge caching, because serving one cached copy satisfies any user who requests it.

Personalized content is a different story.

A user's bank balance is unique to them. Their private messages are theirs alone. Their personalized dashboard reflects their specific account, preferences, and history. You cannot cache these at an edge location and serve them to other users, because they belong to one person.

Good candidates for CDN caching:
  Product images, company logos, marketing pages,
  CSS stylesheets, JavaScript files, video content,
  font files, public documentation.

Poor candidates for CDN caching:
  Bank balances, private messages, personalized feeds,
  account settings, real-time inventory, session data.

The CDN is most powerful when many users want the same thing. When every user wants something different, the edge location can't help, and the request has to travel all the way to the origin anyway.

This is the honest picture of what a CDN provides: a dramatic improvement for the substantial portion of web traffic that is static and shared, and no improvement at all for requests that are inherently personal.

Conclusion

Let's look at the full map of what we've built across this series.

We started with one server. We've been adding layers ever since, each one solving a specific kind of problem.

The full picture:

Users around the world
      |
      v
[ CDN Edge Locations ]      <-- static content served nearby
      |
      v (dynamic requests only)
[ Load Balancer ]           <-- traffic distributed across servers
      |
      v
[ Application Servers ]     <-- requests handled in parallel
      |              |
      v              v
[ Cache ]        [ Message Queue ]   <-- repeated work absorbed,
                       |                  slow work moved to background
                       v
                  [ Workers ]
      |
      v
[ Read Replicas ]           <-- read traffic distributed
      |
      v
[ Database Shards ]         <-- data distributed across machines

Each layer in that diagram exists because one specific problem demanded it.

Load balancers appeared because one server couldn't handle the traffic.
Caching appeared because the database kept answering the same questions.
Read replicas appeared because one database couldn't handle all the reads.
Sharding appeared because one database couldn't hold all the data.
Message queues appeared because users shouldn't wait for work that can happen later.
CDNs appeared because content was too far from the people who needed it.

In every case, the pattern was the same. A bottleneck appeared. We found the specific thing that was being concentrated too much in one place or one moment. Then we distributed it.

Traffic. Repeated work. Reads. Data. Background work. Content.

All of it, distributed.

But look at the application itself for a moment. We've been scaling the infrastructure around it, but the application code that runs on those servers has been growing too. Features get added every week. Teams grow. Engineers who joined recently struggle to understand parts of the codebase that were written before they arrived.

Deploying a change to fix a bug in the payment system requires redeploying the entire application, including the parts that handle user profiles, notifications, search, and everything else. A failure in one part can bring down the whole thing.

The infrastructure scales. The team scales. But the application, as a single unified codebase, starts to become a problem of its own.

What happens when one application becomes too big for one team, one codebase, and one deployment to handle?

That's the question Part 12 takes on.

Your Users Shouldn't Have to Wait: Learn Message Queues

Aditya Sharma — Sat, 08 Aug 2026 18:11:31 +0000

This is Part 10 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.

In Part 9, we solved the problem of data that had grown too large for a single database. We split it across multiple shards, each holding its piece of the whole, so that no single machine ever had to carry everything.

At that point, the architecture could scale in almost every direction we'd tried to push it. Traffic was distributed across application servers. Repeated database work was absorbed by the cache. Read traffic was spread across replicas. Data itself was partitioned across shards.

And yet.

We ended Part 9 by noticing something that none of those solutions addressed. Some user requests trigger a lot of downstream work. Saving an order is one thing. But saving the order, sending a confirmation email, generating an invoice, updating inventory, firing off a notification, recording an analytics event, triggering the recommendation engine: that's an entirely different conversation.

Right now, all of that happens before the user gets a response.

The question we left with was this: what if they didn't have to wait for all of it?

Section 1: The User Doesn't Need Everything Right Now

Before we look at any solution, it's worth asking a simpler question.

When a user places an order, what do they actually need to know before they can move on?

They need to know the order was received. They need confirmation that the important thing happened: their money was accepted, their items are reserved, the transaction is real. That's it. That's what they're waiting for.

They do not need to wait for the confirmation email to land in their inbox. They do not need to wait for the invoice to be generated and stored somewhere. They do not need to wait for the analytics system to record that this purchase happened. They certainly don't need to wait for the recommendation engine to update its model based on what they just bought.

All of that will happen. It should happen. But none of it needs to happen before the user gets their confirmation screen.

This seems obvious when you say it directly. Of course the user doesn't need to wait for the analytics event. Of course they don't need to hold their breath while the recommendation system recalculates.

But the way most applications are initially built, that's exactly what happens.

Section 2: Doing Everything Synchronously

Here's what a typical order flow looks like when everything is wired together the simple, obvious way.

User clicks "Place Order"
        |
        v
Application receives request
        |
        v
Save order to database        (50ms)
        |
        v
Send confirmation email       (200ms)
        |
        v
Generate invoice              (120ms)
        |
        v
Update inventory              (80ms)
        |
        v
Record analytics event        (90ms)
        |
        v
Trigger recommendations       (150ms)
        |
        v
Return response to user

Total wait: ~690ms

The user clicked a button and waited almost 700 milliseconds for a confirmation screen. More than half a second, for a response to something they did in an instant.

And that's assuming nothing goes wrong. What if the email service is having a slow moment and takes two seconds instead of 200 milliseconds? The user waits two seconds. What if the analytics system is down entirely? The request fails, and the user gets an error, even though the order itself was saved perfectly.

The application has tied its response time and its reliability to every piece of downstream work it performs. Every slow step makes the user wait longer. Every failing step makes the whole request fail.

That's not a hardware problem. It's not a database problem. It's an architectural problem. The application is doing work in a sequence when most of that work doesn't actually depend on the steps before it.

The confirmation email doesn't need the invoice to be generated before it can be sent. The analytics event doesn't need the email to succeed before it can be recorded. These tasks are all independent of each other. They're only sequential because that's how they got wired together.

So the question becomes: what would it look like to stop treating them as sequential?

Section 3: Put the Work in a Queue

The insight is simple once you see it.

Not all work needs to happen now. Some work needs to happen eventually, but the user doesn't need to wait for it.

If that's true, then instead of doing all that work before responding, the application can do this:

Save the order. That's the important part, and it must happen now.
Make a note that a confirmation email needs to be sent.
Make a note that an invoice needs to be generated.
Make a note that an analytics event needs to be recorded.
Return a response to the user. Then, separately, something else comes along, reads all those notes, and does the work.

User clicks "Place Order"
        |
        v
Application receives request
        |
        v
Save order to database        (50ms)
        |
        v
Leave notes for background work
        |
        v
Return response to user

Total wait: ~60ms

The user gets their confirmation in 60 milliseconds instead of 690. Then, in the background, the email goes out, the invoice is generated, the analytics are recorded. The user never waits for any of it.

This is the idea behind a message queue. Instead of doing background work inline, the application puts a message into a queue. Each message is a piece of work that needs to happen: "send this email," "generate this invoice," "record this event." The queue holds onto those messages. Something else picks them up and does the actual work.

The application that creates messages is called the producer. It produces work to be done.

The queue is the place that holds work waiting to be processed. It doesn't do the work itself. It holds the work.

The thing that picks up messages and processes them is called a consumer or worker. It consumes messages from the queue and does the actual task.

BEFORE:
User --> Application --> [save] --> [email] --> [invoice] --> [analytics] --> Response

AFTER:
User --> Application --> [save] --> Queue --> Response

                                    Queue
                                      |
                         .------------+-------------.
                         |            |             |
                     [Worker]     [Worker]      [Worker]
                       email      invoice      analytics

The user's request touches the queue for a moment, hands off the work, and returns a response. The workers operate completely independently. They process messages at their own pace, without the user having to wait.

Section 4: Let Workers Do the Work Later

This separation between "the work that happens now" and "the work that happens later" is called asynchronous processing. The user's request doesn't wait for every piece of work to complete before finishing. It hands off what it can, responds quickly, and trusts that the rest will happen.

It's worth being clear about what changes for the user.

From their perspective, they clicked a button and got a confirmation almost instantly. The confirmation email arrives a second or two later, while they're already looking at the confirmation screen. The invoice appears in their account shortly after. They experience everything they expected to experience. They just didn't have to wait for it all at once.

And from the system's perspective, the work is now separated. The application server that handled the user's request is free to handle the next request. It's not blocked, waiting for an email to send. The workers that process the queue can be scaled independently, separate from the application servers. If there's a backlog of emails to send, you add more email workers. The application servers that handle user requests don't need to change at all.

It also makes the system more resilient. Before, if the email service went down, user requests would fail entirely, even though the order itself was being saved correctly. Now, if the email service goes down, the messages just accumulate in the queue. When the email service recovers, the workers process the backlog. Orders are never affected. Users get their emails a little late, but the system doesn't break.

Section 5: The Queue as a Buffer

There's a second, equally important reason queues exist that has nothing to do with user experience.

Imagine a flash sale. A limited-edition product goes on sale at noon, and 50,000 users try to place an order at exactly the same moment.

Without a queue, 50,000 requests arrive simultaneously. Every single one of them tries to do all the work at once. The email service receives 50,000 requests in the same second. The invoice system receives 50,000 requests in the same second. The analytics system receives 50,000 requests in the same second. Systems that were designed for a normal pace of traffic suddenly face a spike that's fifty times larger than usual.

Without a queue:

50,000 simultaneous orders
        |
        v
Email Service    <-- overwhelmed
Invoice System   <-- overwhelmed
Analytics        <-- overwhelmed

Some of those systems buckle. Response times explode. Errors start appearing. The spike that should have been a celebration turns into an incident.

Now add a queue.

With a queue:

50,000 simultaneous orders
        |
        v
      Queue         <-- absorbs the spike instantly

      Queue
        |
        v
Workers process at steady pace
  Email: one at a time, as fast as they can
  Invoice: one at a time, as fast as they can
  Analytics: one at a time, as fast as they can

The 50,000 orders arrive in a second. They all get saved. They all drop their background work into the queue. Users all get their confirmation. Then the queue drains over the next few minutes as workers process messages at a sustainable rate.

The queue acts as a buffer between the rate at which work arrives and the rate at which it can be processed. Producers and consumers don't have to run at the same speed. The queue absorbs the difference.

This property is what makes message queues so valuable during traffic spikes. The burst of user activity doesn't translate directly into a burst of load on every downstream system. It translates into a larger queue, which then drains steadily. The downstream systems see a smooth, consistent workload regardless of how spiky the incoming traffic was.

Section 6: The Trade-off: Work Can Fail

It would be easy at this point to conclude that queues solve everything. They reduce user latency. They isolate failures. They smooth out traffic spikes. What could go wrong?

Quite a bit, actually. Moving work to the background introduces a new category of problems that synchronous systems don't have to worry about.

Workers can fail. A worker picks up a message and crashes halfway through processing it. The email was never sent. The invoice was never generated. If nothing else intervenes, the user never gets their email, and nobody knows.

Most queue systems handle this by keeping a message in the queue until a worker explicitly confirms it's done. If the worker crashes without confirming, the message goes back into the queue and another worker picks it up. That's a good default, but it creates the next problem.

Work can happen twice. If a worker processes a message, sends the email, and then crashes before confirming completion, the queue puts the message back. Another worker picks it up and sends the email again. The user receives the same confirmation email twice.

This is called duplicate processing, and handling it correctly requires extra care. Either the worker has to be designed so that doing the same work twice causes no harm (the technical term for this is idempotency), or the system has to track which messages have already been completed.

Some messages keep failing. Imagine an email address is invalid. Every time a worker tries to send to it, it fails. The message goes back in the queue. Another worker picks it up, tries again, fails again. This can loop indefinitely, taking up queue space and worker time while never making progress.

Most queue systems have a mechanism for this: after a message fails a certain number of times, it gets moved to a separate place called a dead-letter queue. The dead-letter queue holds messages that couldn't be processed, so that engineers can inspect them, understand why they failed, and decide what to do.

None of these are unsolvable problems. They're the trade-offs that come with asynchronous processing. You gain speed, resilience, and the ability to absorb bursts. You take on the responsibility of handling failures gracefully, ensuring work isn't lost, and being careful when the same work might happen more than once.

That's the contract. Synchronous work is simple: it either works or it doesn't, and the user knows immediately. Asynchronous work is more powerful, but it requires the system to think carefully about what happens when individual pieces go wrong in the background where nobody is watching.

Conclusion

Let's look at what we've now built, from the very beginning of this series.

We started with a single server. We scaled traffic across many application servers behind a load balancer. We reduced unnecessary database work with caching, and kept that cache accurate with invalidation strategies. We distributed database reads across replicas. We split the data itself across shards when a single database became too large.

And now we've changed when work happens, not just how it's distributed.

The complete picture so far:

Users
  |
  v
[ Load Balancer ]
  |
  v
[ Application Servers ]
  |              |
  v              v
[ Cache ]    [ Message Queue ]
               |
               v
          [ Workers ]
  |
  v
[ Read Replicas ]     [ Database Shards ]

At each step in this series, scaling meant something slightly different.

Load balancers let us scale traffic. Caching let us scale by reducing work. Read replicas let us scale reads. Sharding let us scale storage and writes. Message queues let us scale by changing when work happens.

That last idea is the most unusual one. Scaling doesn't always mean making something faster. Sometimes it means moving work out of the user's way entirely, letting them continue while the system handles the rest at its own pace.

But we've been making one quiet assumption this entire time.

We've assumed that users are somewhere near our servers. That when a user in Bengaluru or Berlin or São Paulo sends a request, the server receives it quickly and the response arrives quickly.

That assumption starts to break down at global scale.

A server sitting in a data center in Virginia is fast for users in Virginia. For a user in Singapore, that same request has to travel thousands of kilometers across the internet and back. No amount of caching, sharding, or queueing changes the speed of light.

What happens when the physical distance between your users and your servers becomes the bottleneck?

That's the problem Part 11 is about.

One Database Can't Hold Everything: Learn Database Sharding

Aditya Sharma — Fri, 07 Aug 2026 05:46:43 +0000

This is Part 9 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.

Look at what we've built so far.

We started with a single server. We added more application servers when one wasn't enough, spreading traffic horizontally across many machines. We added a cache to stop the database from answering the same question thousands of times. We taught that cache to stay honest as data changed underneath it. We added read replicas to distribute the remaining read traffic across multiple database copies, so the primary could focus on writes.

At each step, the system got more capable. At each step, a new bottleneck appeared just past the solution we'd just built.

For a while after adding read replicas, things were genuinely good. Reads scaled. The primary handled writes. The application felt fast.

But the application kept growing. And something that had nothing to do with traffic started becoming a problem.

The data itself.

Section 1: When One Database Becomes Too Big

Traffic is one dimension of growth. Data is another. And they don't always move together.

Every time a new user signs up, their record gets written to the database. Every order, every message, every transaction, every notification, every log entry accumulates. Most of it never gets deleted. Regulations sometimes require keeping it for years. Business needs require querying it at any time.

For a while, this is fine. A database that holds ten million user records is manageable. One that holds fifty million is still fine with the right hardware. But at some point, something shifts.

Backups start taking hours instead of minutes. A backup that used to finish before anyone noticed is now still running when the morning traffic surge begins.

Indexes grow large enough that keeping them in memory becomes difficult. An index that once fit comfortably in RAM now spills to disk, and suddenly queries that were instant start slowing down, not because the query is wrong, but because the index that makes it fast is too big to hold in memory all at once.

Maintenance operations that used to be routine become risky. Adding a column to a table with two billion rows isn't a quick command anymore. It locks the table, it takes hours, and the application has to work around it.

And writes keep coming. Every new user, every new transaction, every new event. The write load on the primary database grows steadily, not because users are doing anything unusual, but simply because there are more of them every day.

The engineers look at their monitoring dashboards. The read replicas are handling reads beautifully. But the primary database is under increasing strain, and not just from write volume. It's the sheer size of everything it has to manage.

So they do what engineers usually try first: they upgrade the hardware.

Section 2: Why Bigger Servers Stop Helping

More RAM, faster disks, more CPU cores. For a while, vertical scaling works. The database gets more room to breathe and the slowdowns ease.

But vertical scaling has a ceiling, and that ceiling is physical.

There are only so many CPU cores you can put in one machine. There is only so much RAM a single server can hold, and that limit is measured in terabytes, not infinite amounts. The fastest possible disk is still a single disk attached to a single machine. And even before you hit the absolute physical limits, you hit the economic ones. The price of a server doesn't scale linearly with its capability. A machine with twice the RAM doesn't cost twice as much. It costs three or four times as much, and at a certain point the cost becomes unreasonable.

More importantly, a bigger machine doesn't solve the fundamental problem. The data is all in one place. Every query, no matter how fast the hardware running it, still has to navigate the same enormous dataset. Every backup still has to copy the same enormous amount of data. Every index still has to track the same enormous number of rows.

You're not reducing the problem. You're just buying a bigger container for it.

Vertical Scaling:

Small Server    →    Medium Server    →    Large Server    →    ???
[ 64GB RAM  ]       [ 256GB RAM  ]        [ 1TB RAM    ]
[ 1TB disk  ]       [ 8TB disk   ]        [ 64TB disk  ]

Each jump costs significantly more.
Eventually, there is no larger option.
The problem is still the same size.

At some point, an engineer asks the question that changes the direction of the whole conversation.

What if we stopped trying to make the database bigger, and started splitting the data across multiple databases instead?

Section 3: One Database Can't Hold Everything: Database Sharding

Let's think through that idea slowly, because it's a significant shift from everything we've done before.

With read replicas, we created multiple copies of the same database. Every replica had all the data. The benefit was that reads could be spread across many machines. But the data itself still lived in one place, fully, on the primary.

This new idea is different. Instead of copying the data, we divide it.

Imagine your application has 90 million users. Right now, all 90 million live in a single users table in a single database. What if instead, you split them across three separate databases?

Database 1: Users with IDs 1 to 30 million
Database 2: Users with IDs 30 million to 60 million
Database 3: Users with IDs 60 million to 90 million

Each database now holds only one third of the total data. Each one is a third of the size. Each one has smaller indexes. Each one can be backed up in a third of the time. Each one receives only the writes that belong to its slice of users.

When a user logs in, the application looks at their user ID, figures out which database holds their data, and talks to that one.

Without Sharding:

All 90 million users
        |
        v
[ Single Database ]   <-- enormous, under pressure

With Sharding:

Users 1-30M    →  [ Database Shard 1 ]
Users 30-60M   →  [ Database Shard 2 ]
Users 60-90M   →  [ Database Shard 3 ]

Each individual database is now a manageable size. As the total number of users grows past 90 million, you add a fourth shard. The existing databases don't have to grow. Only the new shard fills up with new data.

This is Database Sharding. Each individual database is called a shard, and together they hold the complete dataset that used to belong to one database. No shard has everything. Every shard has its piece.

The effect on writes is immediate and direct. Instead of all 90 million users' writes going to one primary, they're now distributed. User activity for the first 30 million users hits one database. Activity for the next 30 million hits another. The write load is divided along with the data.

And each shard can have its own read replicas, if needed. The full architecture composes cleanly with everything we've already built.

Section 4: Choosing a Shard

Once you accept the idea of splitting data across multiple databases, a very practical question appears: how does the application know which shard to look in?

This is where the shard key comes in. A shard key is the piece of information you use to decide where a particular piece of data lives. The application uses it to route every query to the right shard.

The example we just used divided users by their ID range. User ID is the shard key, and the range it falls into determines the destination.

User ID 4,523,901    →  falls in range 1-30M   →  Shard 1
User ID 47,112,008   →  falls in range 30-60M  →  Shard 2
User ID 83,400,551   →  falls in range 60-90M  →  Shard 3

This works, and it's easy to reason about. But ID range isn't the only way to divide data. Different applications shard in different ways depending on what makes sense for their data.

A global application might shard by geography. Users in North America go to one database. Users in Europe go to another. Users in Asia go to a third. This keeps data physically close to the users it belongs to, which can reduce how far queries have to travel across a network.

A multi-tenant business application might shard by customer. Every piece of data belonging to Company A lives in one shard. Company B lives in another. Each customer's data is completely isolated from every other customer's.

In every case, the idea is the same: pick a property of the data that lets you divide it predictably, and make sure the application can always determine which shard holds what it needs.

Sharding by User ID range:
  User 4M     →  Shard 1
  User 47M    →  Shard 2

Sharding by Geography:
  User in Germany   →  EU Shard
  User in Brazil    →  Americas Shard

Sharding by Tenant:
  Company A data    →  Shard A
  Company B data    →  Shard B

The application now carries a small piece of routing logic it didn't have before. For every database operation, it has to answer the question: which shard does this belong to? Most of the time this is straightforward, as long as the shard key is always available. If you're looking up a user, you have the user ID. If you're looking up an order, the order is associated with a user, so you have the user ID. The routing decision is quick.

But the new complexity doesn't stop there. Not every problem is about where data lives. Some problems are about whether the data ends up in the right places to begin with.

Section 5: The Hidden Trade-off: Uneven Shards

Dividing data by ID range seems clean on paper. Users 1 to 30 million here, 30 to 60 million there. Equal thirds. But numbers don't tell the whole story.

Not all users are equally active.

Imagine your application has been around for several years. The oldest users, those with low user IDs from the early days, tend to be the most engaged. They've built up years of activity, thousands of posts, millions of interactions. They log in every day.

The newest users, those with high user IDs who joined recently, might have just signed up and haven't done much yet. Some of them will never come back.

Now look at what that means for your shards.

Shard 1: Users 1-30M
  Old, active users. High write volume.
  Large amount of historical data per user.
  Frequent queries, heavy index usage.
  Running hot.

Shard 3: Users 60-90M
  New users, low engagement.
  Minimal data per user.
  Infrequent queries.
  Barely doing anything.

The data is divided evenly by count, but the work is not divided evenly at all. Shard 1 is overloaded. Shard 3 has capacity to spare. You've added complexity to the system, but you haven't actually distributed the pressure evenly.

This problem has a name: a hotspot. A shard that receives disproportionately more traffic or stores disproportionately more active data than the others.

Hotspots happen when the shard key you chose doesn't distribute real-world activity evenly, even if it looks balanced on paper. ID ranges can create hotspots based on user age. Geographic sharding can create hotspots if one region has far more users than others. Tenant sharding can create hotspots if one company is ten times larger than every other.

Dealing with hotspots requires either choosing a better shard key, splitting the overloaded shard into smaller pieces, or distributing data using a more sophisticated strategy that mixes up the routing so no single shard ends up holding all the busiest users.

There are well-established approaches to solving the hotspot problem, but each one adds more complexity. The application's routing logic becomes more involved. Moving data between shards, if you decide the current division is wrong, is a large and careful operation. Cross-shard queries, where you need data that spans multiple shards, require the application to query several databases and assemble the result.

None of these problems are unsolvable. They're the trade-offs that sharding introduces in exchange for the problems it solves. And the same rule that has applied to every technique in this series applies here: sharding solves the problems of scale, and it introduces new problems in return.

Conclusion

Let's look at how far the architecture has come since Part 1.

We started with a single server handling a single user. We scaled application servers horizontally so that traffic could be spread across many machines. We added a cache so the database didn't have to keep answering the same questions. We built invalidation strategies to keep that cache accurate. We added read replicas to distribute reads so the primary database could focus on writes.

And now, in Part 9, we've taken the data itself and spread it horizontally across multiple databases. Each shard holds its piece of the whole. Storage scales because each database only grows with the users assigned to it. Writes scale because they're distributed across shards. Maintenance becomes manageable because no single database ever has to hold everything.

The full picture looks something like this.

                        [ Load Balancer ]
                               |
             .-----------------+-----------------.
             |                 |                 |
       [ App Server ]    [ App Server ]    [ App Server ]
             |                 |                 |
             v                 v                 v
          [ Cache ]         [ Cache ]         [ Cache ]
             |
    .--------+--------.
    |                 |
[ Shard 1 ]      [ Shard 2 ]      [ Shard 3 ]
[ Primary ]      [ Primary ]      [ Primary ]
[ Replica ]      [ Replica ]      [ Replica ]
[ Replica ]      [ Replica ]      [ Replica ]

It's a long way from where we started.

But notice something that this architecture still doesn't handle gracefully.

Some operations don't belong to a single shard. Imagine a user sends a money transfer to someone who lives on a different shard. Or the application needs to generate a report that pulls data from every shard simultaneously. Or a new user signs up, and the welcome email, the onboarding notification, the recommendation engine seed, and the analytics event all need to happen as part of the same flow.

These are tasks that are either slow, span multiple shards, or shouldn't block the user from moving on.

Right now, the user has to wait for all of it.

What if they didn't have to?

What if the application could say: "I've saved your order. The rest will happen in the background." And then actually make that happen, reliably, even if it takes a few seconds?

That idea, of separating the work that has to happen immediately from the work that can happen whenever the system gets to it, is where Part 10 begins.

One Database Isn't Enough: Learn Read Replicas

Aditya Sharma — Thu, 06 Aug 2026 09:47:49 +0000

This is Part 8 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.

We've come a long way.

We taught our application to stop asking the database the same question over and over. We added a cache to absorb repeated reads, and we built strategies to keep that cache honest as data changed underneath it. At each step, the database had less to do.

And for a while, that was enough.

But something interesting kept happening. Even after caching was in place, even after the most common queries were being served from memory without touching the database at all, engineers would look at their database and find it still running hot. Still busy. Still the slowest part of the system.

The natural reaction is to wonder if something is broken. Did the cache fail? Are there too many cache misses? Is the invalidation strategy wrong?

Not necessarily. Sometimes the database is busy for a completely different reason.

To understand that reason, we need to think carefully about the kind of work caching actually eliminates, and the kind it can't touch.

Section 1: Caching Solved One Problem, Not All of Them

Think back to why caching made sense in the first place.

The database was being asked the same questions, by thousands of different users, over and over again. "What are the trending articles right now?" "What products are on sale?" "What does the homepage look like?" The answers didn't change between requests, so computing them repeatedly was pure waste. The cache absorbed that waste by storing the answer once and handing it out to everyone who asked.

The key word there is same. Caching works beautifully when many users want the same thing.

But as an application grows, it doesn't just serve more users. It serves more kinds of requests. And a lot of those requests aren't the same at all.

Consider what happens when a real application is running at scale. Someone opens their email inbox. Someone else checks their bank balance. A third user scrolls through a feed of posts from the specific people they follow. A fourth checks the status of an order they placed three days ago.

Every one of those requests reaches the database. None of them can be served from a shared cache, because the answer is different for every single user. Your inbox is not my inbox. Your transaction history is not mine. Your feed is assembled from a different set of accounts than anyone else's.

This is the category of work that caching cannot help with, and it turns out this category is enormous. In most applications that have grown to real scale, the majority of database reads fall into exactly this bucket: personal, unique, and impossible to pre-store.

So after you've deployed a cache and reduced the repetitive work as much as possible, what's left is not waste. What's left is legitimate work.

The database isn't doing unnecessary work anymore. It's just doing too much legitimate work.

That's a different problem than the one caching solved. And it needs a different solution.

Section 2: Why Some Reads Can Never Be Cached

It helps to be precise about this, because the instinct when a database is struggling is always to reach for the cache. Before we move on, let's close that door properly.

A cache works by storing the result of a query under a key, and returning that stored result to anyone who asks the same question. The efficiency comes entirely from sharing one stored result across many requesters.

Now imagine trying to cache Aisha's notification feed. You store it under the key notifications:aisha. The next time Aisha checks her notifications, the cache serves it instantly.

But Aisha is one user. That cached result is used by exactly one person. It gets stale the moment she receives a new notification. And there are ten million users on this platform, each with their own notifications, each with their own key in the cache, each requiring their own database query to populate.

You haven't reduced the work. You've just moved it, and added the overhead of maintaining a cache on top.

Shared cached data (works well):
  trending_articles  →  served to 50,000 users from one cache entry

Personal cached data (doesn't help):
  notifications:aisha   →  served to 1 user
  notifications:ravi    →  served to 1 user
  notifications:chen    →  served to 1 user
  ...ten million more

Caching personal data can make sense in specific, narrow situations. But it doesn't solve the underlying problem, which is that the database has to compute ten million different answers for ten million different people, and it only has one machine's worth of computing power to do it with.

Most web applications receive far more reads than writes. People browse far more than they post. They view far more than they update. They check their feeds, their histories, their dashboards, constantly, while writes happen occasionally.

This means that a database under heavy load is spending most of its time answering read queries. And many of those read queries are the kind that can't be shared.

So if one database is overwhelmed by read requests, and caching can't absorb them, what do you actually do?

Section 3: One Database Isn't Enough: Meet the Read Replica

Here's the simplest version of the question.

If the database is struggling because too many reads are arriving at once, and you can't reduce the reads, what's the only other option?

Spread them across more than one database.

Think about what a read actually requires. The database needs a copy of the data, and it needs the ability to run a query against it. That's it. A read doesn't change anything. It doesn't modify any rows. It doesn't need to coordinate with other writes. It just looks something up and returns an answer.

That means, in principle, if you had two identical copies of the database, you could answer twice as many reads simultaneously. Half the reads go to the first copy, half to the second. Neither one has to work as hard.

This is the idea behind a Read Replica.

A Read Replica is an additional database server that holds a copy of all the data from your main database. It exists specifically to serve read queries, sharing the load so the primary database doesn't have to answer every single one.

Without Read Replicas:

App Servers
    |
    | (all reads AND writes)
    |
    v
[ Primary Database ]   <-- doing everything, overwhelmed

With Read Replicas:

App Servers
    |
    |-- reads -->  [ Read Replica 1 ]
    |-- reads -->  [ Read Replica 2 ]
    |-- reads -->  [ Read Replica 3 ]
    |
    |-- writes --> [ Primary Database ]

The primary database handles all writes. The replicas handle reads. Each replica has a full copy of the data, so any of them can answer any read query, just as the primary would.

And if traffic keeps growing, you add more replicas. The primary database stays the same. The read load gets distributed across however many replicas you need.

That's the whole idea. One database becomes several. Reads get shared across all of them. The primary is free to focus on writes.

Section 4: Sharing Reads Between Databases

The moment you have multiple databases that can serve reads, a practical question appears: how does the application decide which one to use?

This is called read/write splitting, and it's exactly what it sounds like. The application is configured to send writes to the primary database and reads to the replicas.

In practice, this can be handled in a few ways. Some applications make the decision in their own code, explicitly choosing which database connection to use depending on the operation. Some use a proxy layer that sits between the application and the databases, routing queries automatically based on whether they're reads or writes.

Application
    |
    v
[ Database Proxy / Router ]
    |                   |
    | writes            | reads
    v                   v
[ Primary ]        [ Replicas ]

From the perspective of the application developer, this is mostly transparent. You write a read query, it goes to a replica. You write an insert or update, it goes to the primary. The routing logic handles the rest.

There's one important thing to understand about the replicas, though: they're not independent databases that you manage separately. They're kept in sync with the primary through a process called replication.

Replication means that whenever something is written to the primary database, that change is propagated to the replicas automatically. You don't manually copy data across. The database system does it for you, continuously, in the background.

Think of the primary as the source of truth, and the replicas as mirrors that are constantly trying to reflect it accurately. Every write that lands on the primary eventually flows out to every replica.

Eventually. That word is doing a lot of work, and it leads us to the most important nuance in this entire article.

Section 5: The Hidden Trade-off: Replication Lag

Replication isn't instant.

When Aisha updates her profile picture, the change is written to the primary database immediately. But the replicas don't receive that change at the exact same millisecond. The primary has to propagate the update out to each replica, and that takes time. Usually a very small amount of time, often measured in milliseconds. But time nonetheless.

During that brief window, something quietly uncomfortable is true.

The primary database has the new profile picture. The replicas still have the old one. If a request for Aisha's profile is routed to a replica before the update has arrived there, the user gets the old photo.

Timeline after Aisha's update:

t=0ms    Primary receives write: new profile picture stored.
t=0ms    Replicas: still have old picture.
t=12ms   Replication completes: replicas now have new picture.

If someone reads Aisha's profile at t=5ms from a replica:
  → They see the old picture.

If someone reads Aisha's profile at t=20ms from a replica:
  → They see the new picture.

This gap between what the primary knows and what the replicas know is called replication lag. In healthy systems under normal conditions, it tends to be very small. Milliseconds. Users rarely notice.

But replication lag becomes significant under heavy write load, when the primary is processing many updates quickly and the replicas are struggling to keep up. The lag can stretch from milliseconds to seconds. In extreme cases, longer.

The question you have to ask for each application is: what happens if a user sees slightly stale data from a replica?

For many reads, the answer is: nothing significant. If Aisha's follower count shows 4,821 instead of 4,822 for a few milliseconds while replication catches up, the world doesn't end. That's a tolerable inconsistency.

For other reads, it matters more. Imagine Aisha updates her own profile and is immediately redirected to a page that shows her profile. If that read hits a replica that hasn't caught up yet, Aisha sees her own old photo staring back at her, even though she just changed it. That's confusing and feels broken, even if technically nothing went wrong.

This is the trade-off that Read Replicas introduce: you gain the ability to handle far more read traffic, but you accept that replicas might be slightly behind the primary at any given moment.

Different teams handle this in different ways. Some applications route certain sensitive reads, like "show me my own profile immediately after I edit it," to the primary rather than a replica, accepting slightly higher load on the primary for those specific operations. Others build in a brief delay before redirecting, giving replication a moment to catch up. Others simply accept that brief inconsistency for non-critical reads, and let replica lag resolve itself naturally.

There is no single right answer. The right answer depends on what your application is doing and what your users will actually notice.

This is the same lesson that appeared in cache invalidation: distributing data across multiple places gives you performance, and it introduces the possibility that those places disagree. Any time data lives in more than one location, those locations can fall out of sync. Every strategy for scaling a database has some version of this trade-off embedded inside it.

Conclusion

Let's take stock of where our architecture stands.

We started this series with a single server handling a single user. We scaled application servers horizontally, spreading traffic across many machines. We added a cache to eliminate repeated database reads and taught it to stay accurate as data changed.

Now we've taken the same horizontal scaling idea and applied it to the database layer itself, but only to reads. The primary database handles writes, the source of truth that everything else flows from. The read replicas distribute the vast majority of traffic across multiple machines, each one a full copy of the data, each one answering queries that the primary no longer has to touch.

The result is a system that can serve an enormous volume of reads without the database becoming the ceiling.

But notice something.

Today we taught one database to become many. Reads are now distributed across replicas, each one a full mirror of the primary, each one sharing the load.

But every one of those databases contains exactly the same data.

And every write still has to reach the same primary database. The replicas help enormously with reads, and reads dominate most applications, but they do nothing for writes. Adding ten replicas doesn't make writes any faster. The primary is still the only machine allowed to accept new data.

For a long time, that's fine. Writes are less frequent than reads, and a single well-provisioned primary can handle a lot. But applications keep growing.

Eventually, the problem is no longer reading the data.

The problem is storing it.

What happens when a single database can no longer hold everything, no matter how powerful the machine running it is? When the data itself outgrows the box?

That's where Part 9 begins.

Fast... But Wrong? Meet Cache Invalidation

Aditya Sharma — Wed, 05 Aug 2026 09:20:39 +0000

This is Part 7 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.

Last time, we ended on a question that sounded simple but isn't.

Aisha updated her profile picture. Her new photo is now saved in the database. But the cache is still holding onto the old one, completely unaware that anything changed. So every request for Aisha's profile gets served the old data. Confidently. Instantly. Incorrectly.

How does a cache know when the data it's holding is no longer correct?

Think about what we've actually built at this point. We have an application that responds fast, scales horizontally, and avoids hammering the database with repeated identical queries. From a performance standpoint, it looks great.

But Aisha's friends are loading her profile and seeing a photo she replaced five minutes ago. The system isn't slow anymore. It's wrong.

Speed and correctness are two different things. We optimized hard for one, and quietly broke the other.

Engineers have a name for this problem: cache invalidation. It refers to the challenge of keeping the data in your cache consistent with the data in your database, as that underlying data changes over time.

It turns out to be one of the genuinely hard problems in building software systems. Not hard in a complicated-algorithm way. Hard in the way that every solution has a catch, and the right answer always depends on what you're willing to accept.

Let's think through it together.

Section 1: When Cached Data Lies

It's worth sitting with the problem a little longer before rushing to fix it, because the damage stale data can cause varies enormously depending on what's being cached.

Consider a few examples.

Your application caches the list of trending articles. An hour later, the list has changed. New articles have risen, old ones have faded. But the cache still serves the original list. Users see slightly outdated trending content. That's mildly annoying, but nobody gets hurt.

Now imagine your application caches a product's price. A flash sale begins and the price drops by 40%, but the cache still confidently serves the old price. Users add the item to their cart expecting a discount they aren't going to get. That's a real problem: frustrated users, support tickets, potential refund requests.

Now imagine your application caches a user's account permissions. A user is suspended by an admin, but the cache still serves the old permissions. That suspended user continues accessing parts of the system they should be locked out of. That's a security issue.

The underlying problem is the same in all three cases. The cache is serving data that no longer matches what's in the database. But the consequences range from "mildly stale" to "genuinely dangerous."

This is the tax that caching collects. The cache made your system faster, but it did so by creating a second place where data lives. And here is a rule worth remembering, because it applies far beyond caching:

Any time data lives in two places, those two places can disagree.

The question is never whether they will disagree. Given enough time and enough writes, they will. The question is always: how long are you willing to let them disagree, and what happens to your users when they do?

There's a famous saying among engineers, sometimes credited to Phil Karlton:

"There are only two hard things in computer science: cache invalidation and naming things."

It's been repeated enough to become a cliché. But the reason it's stuck around is that it's true. Let's see why.

Section 2: The First Instinct: Delete It When It Changes

The most obvious approach is also the most direct one. If the cached data might be wrong because the underlying data changed, then when you change the underlying data, just remove the cached version.

The logic is simple: an absent cache entry is honest. It says "I don't know." A stale cache entry lies. So if something changes, delete the cached copy, and let the next request go back to the database for a fresh answer.

This pattern is called Cache Aside, and it works like this:

READ path:
  1. Check the cache.
  2. If hit → return cached data.
  3. If miss → fetch from database, store in cache, return data.

WRITE path:
  1. Write the updated data to the database.
  2. Delete the corresponding entry from the cache.

Let's trace through Aisha's profile update.

Aisha updates her profile picture:

App Server → Database: "Update profile for aisha"
App Server → Cache:    "Delete profile:aisha"

Next request for Aisha's profile:

App Server → Cache: "Do you have profile:aisha?"
Cache: "No." → Cache Miss

App Server → Database: "Get profile for aisha"
Database returns the new, updated profile.

App Server → Cache: "Store this as profile:aisha"
App Server → User: here's the updated profile

The next person who visits Aisha's profile gets a cache miss, goes to the database, and the cache gets repopulated with the correct, up-to-date data. From that point on, subsequent requests are hits again.

It's clean. It's honest. And for many situations, it works well.

But notice what this approach requires: the application has to know, every single time it writes data, which cache entries to delete.

For something like a user profile, that's manageable. Write to the user table, delete the profile:aisha key. One write, one delete.

But systems get complicated. Imagine a product page that displays the product name, its current price, its average rating, and a list of the five most recent reviews. That data might come from four different database tables. It might be composed by three different parts of your application. When a new review gets posted, which cache keys need to be deleted? When a price is updated, what about any cached search results that also displayed that price? When a product is renamed, does the cache for every page that ever mentioned it need to be cleared?

Suddenly "just delete it when it changes" requires an exhaustive mental map of every cache entry that depends on every piece of data. Miss one entry, and the cache serves a lie.

The harder the system, the harder it is to maintain that map without mistakes.

Section 3: Letting Time Do the Work

Looking at the mess that Cache Aside can become in complex systems, you start wishing the cache would just clean itself up automatically, without requiring the application to track every dependency.

And there is a way to do exactly that. Instead of actively deleting cache entries when data changes, you give each cache entry a lifespan. After that lifespan expires, the cache entry is gone, regardless of whether anyone told it to leave.

This is called a TTL, which stands for Time-To-Live.

When you store something in the cache, you attach an expiration time to it. The cache holds onto it for that duration, serves it freely to whoever asks, and then discards it automatically when the time is up. The next request after expiration becomes a cache miss and fetches fresh data from the database.

SET trending_articles "[...data...]" WITH TTL = 60 seconds

→ For the next 60 seconds: cache hits, database never touched.
→ After 60 seconds: entry expires, next request is a miss.
→ Fresh data is fetched from database and cached again for another 60 seconds.

The beauty of TTL is that the application no longer needs to track which cache entries to invalidate. It just sets a reasonable expiration and lets time handle it. Every piece of cached data has a built-in expiry date. Nothing lives forever.

For data that changes infrequently and where brief staleness is acceptable, TTL is elegant. Trending articles that get recalculated every few minutes? Cache them with a 60-second TTL and stop worrying about it. Homepage banners that a marketing team updates once a day? A 5-minute TTL works fine.

But TTL introduces its own uncomfortable question.

How long should a cache entry live?

That question sounds like a detail, but it's actually where a real tension lives.

Set the TTL too short, and you're sending requests to the database constantly. The cache barely helps because entries expire before they have a chance to absorb much traffic. You're essentially paying the overhead of maintaining a cache without getting much of the benefit.

Set the TTL too long, and you risk serving stale data for an extended window. If a product price is cached with a 24-hour TTL and a flash sale starts, users could be seeing the wrong price for hours before the cache naturally expires and corrects itself.

There's no universal right answer. A TTL is always a tradeoff between how fresh the data needs to be and how much database load you're willing to accept. You pick a number that balances those two concerns for your specific use case, knowing that you can't fully optimize for both at once.

And here's the deeper issue. Even with a well-chosen TTL, there's still a window where the cache is wrong. Maybe just 30 seconds. Maybe just 5. But in those 30 seconds, thousands of users could be served stale data.

For some applications, a 30-second window of stale trending articles is completely fine. For a stock trading platform serving real-time prices, 30 seconds of stale data is a disaster.

The right TTL is the one that fits what your users actually need, not the one that happens to be technically convenient.

Section 4: Keeping Cache and Database in Step

Cache Aside requires the application to manually delete entries on every write. TTL lets entries expire on a schedule but tolerates a window of staleness. Both involve some moment in time where the cache and the database disagree.

What if the requirement is stricter than that? What if the cache must never serve data that doesn't match the database?

That's where Write-Through caching comes in.

The idea is a small but significant shift in how writes work. Instead of writing to the database and then figuring out what to do with the cache separately, the application updates both at the same time, in the same operation.

WRITE path with Write-Through:
  1. Write the updated data to the database.
  2. Write the updated data to the cache.
  3. (Done. Cache and database now agree.)

Let's trace through Aisha again.

Aisha updates her profile picture:

App Server → Database: "Update profile for aisha"  ✓
App Server → Cache:    "Update profile:aisha"       ✓

Next request for Aisha's profile:

App Server → Cache: "Do you have profile:aisha?"
Cache: "Yes, here it is." → Cache Hit (with the *correct* new data)

No staleness. No window of disagreement. The cache is updated at the same moment as the database, so every read after a write immediately sees the correct data.

This sounds ideal, and in some ways it is. Applications where incorrect data is genuinely costly (financial balances, inventory counts, access permissions) often lean toward Write-Through for exactly this reason. The cache and database stay synchronized by construction.

But Write-Through has its own cost, and it's worth being honest about it.

Every single write now has to update two places instead of one. That adds some latency to write operations. More importantly, it means the cache now contains entries for things that might almost never be read. When you write-through, you're pre-populating the cache on every write, regardless of whether anyone is going to read that data soon. You're paying the cost of caching data that might sit there unused.

Compare that to Cache Aside, where the cache only fills up with data that someone actually asked for. Cache Aside is demand-driven: things enter the cache because they were requested. Write-Through is write-driven: things enter the cache because they were modified, whether or not anyone will read them next.

Neither is wrong. They have different shapes.

Cache Aside:
  Reads are slightly slower on first miss.
  Writes are fast (just the database).
  Cache contains what people read.
  Risk: stale data between write and next read.

Write-Through:
  Reads are always fast (cache is always up to date).
  Writes are slightly slower (two destinations).
  Cache may contain things nobody ever reads.
  Risk: wasted cache space on infrequently-read data.

And even Write-Through isn't a complete solution. What happens if the write to the database succeeds but the write to the cache fails? Or the write to the cache succeeds but something goes wrong before the database write commits? Now the two places are out of sync again, through failure rather than design.

Systems where data lives in more than one place have to reason about failure. Always. That's a topic big enough for its own conversation, but it's worth flagging now: even a well-designed Write-Through strategy requires thinking about what happens when individual steps fail.

Section 5: There Is No Perfect Answer

By now, you've seen three approaches to cache invalidation. Each one appeared because the previous one had a flaw:

Cache Aside was honest and simple, but required the application to track every cache dependency. Miss one, and you serve stale data. Scale the system, and tracking those dependencies becomes a maintenance burden.

TTL removed that burden by letting time handle invalidation automatically, but introduced a guaranteed window of staleness, and forced an uncomfortable question about how long that window should be.

Write-Through eliminated the staleness window by synchronizing writes, but slowed down write operations and populated the cache with data that might never be read.

There is no fourth option that fixes all three problems at once. Every approach is a different answer to the same underlying tension: the cache exists to avoid work, but avoiding work means tolerating some risk that what you're serving isn't perfectly current.

The choice between these strategies isn't about which one is correct. It's about which tradeoffs you can live with, given your specific application.

A social media platform caching post counts can tolerate a few seconds of staleness. If a post shows 1,042 likes instead of 1,043, no one is harmed. A payment system caching a user's account balance cannot tolerate the same thing. If the balance is wrong even for a second, someone could be overcharged or allowed to spend money they don't have.

The same engineer, working on both systems, would make different choices. Not because one engineer is more experienced than the other, but because the data is different, the consequences are different, and therefore the acceptable tradeoffs are different.

This is what makes cache invalidation hard. Not the implementations themselves, which are learnable in an afternoon. What's hard is developing the judgment to know which approach fits which situation, and being honest about what you're giving up either way.

Phil Karlton was right. It really is one of the hard problems.

Conclusion

Let's trace the path of this article.

We started where Part 6 left us: Aisha's profile picture had been updated in the database, but the cache was still confidently serving the old one. That was the problem. Not that caching is broken, but that a cache holding onto old data isn't just slow to update. It's actively wrong.

We looked at three ways engineers have learned to handle this.

Cache Aside keeps things simple by deleting cache entries when the underlying data changes, trusting the next read to fetch fresh data and repopulate. It works, but it puts the burden on the application to know what to delete, a burden that grows with the system's complexity.

TTL offloads that burden to time. Set an expiration, let it self-clean. The tradeoff is accepting that stale data will exist for the duration of that window, and that choosing the right window is a judgment call with no universally correct answer.

Write-Through keeps the cache and database synchronized on every write, so reads are always accurate. The tradeoff is slower writes and a cache that may fill with data nobody ever requests.

Every one of these strategies is in active use in production systems today. Often, the same system uses all three, with different strategies applied to different types of data depending on how fresh that data needs to be.

Now step back and look at how far this series has come.

We started with a single server handling a single user. We added more servers when one wasn't enough. We added a load balancer to distribute the traffic across them. We added a cache to stop the database from answering the same question thousands of times. And in this article, we learned how to keep that cache honest as data changes underneath it.

At each step, we made the system handle more load. And at each step, a new problem appeared just past the solution we'd just built.

That pattern continues.

Even with a well-designed cache, there are requests that can never be served from one. Personalized content. Real-time transaction histories. Queries that are unique to each user and can't be pre-stored. Every one of those requests goes straight to the database, every single time.

As the application grows from thousands to millions of users, that one database starts receiving millions of different questions simultaneously. No amount of caching can absorb that. The database itself becomes the ceiling.

So what do you do when the database is the bottleneck, caching can't help, and there's simply more traffic than one machine can handle?

That's where we're headed in Part 8.

Why Ask Twice? Meet the Cache

Aditya Sharma — Tue, 04 Aug 2026 09:13:44 +0000

This is Part 6 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.

Let's pick up exactly where we left off.

In Part 5, we solved the server bottleneck by adding more application servers behind a load balancer. Traffic that used to crush a single server now gets spread across many. On paper, that feels like a win.

But we ended on an uncomfortable observation.

No matter how many application servers we add, they all talk to the same database. The servers stopped being the problem. The database didn't go anywhere.

And as more servers send more requests, all of them still end up in the same place the database's doorstep.

So we asked a question and left it hanging:

If thousands of users request the same information repeatedly, do we really need to ask the database every single time?

Today, we answer that question properly. Not by naming a technology. By actually thinking it through.

Section 1: The Question We Left With

Let's make this concrete, because "database bottleneck" is still a vague phrase until you see it happen.

Imagine you're running a news website. It's a slow Tuesday morning, nothing unusual, until one article about a major event starts trending. Within the next five minutes, fifty thousand people open your homepage.

Every single one of them is asking your application the same question:

"What are the top 10 trending articles right now?"

Your application, being obedient, does exactly what it was built to do. It takes that question and forwards it to the database.

User 1  → App Server → Database: "Give me top 10 trending articles"
User 2  → App Server → Database: "Give me top 10 trending articles"
User 3  → App Server → Database: "Give me top 10 trending articles"
...
User 50,000 → App Server → Database: "Give me top 10 trending articles"

Fifty thousand separate trips to the database. Fifty thousand times the database has to scan through rows, sort by trending score, and assemble a result.

Now here's the part that should genuinely bother you.

The answer was identical every single time.

Nothing changed between request #1 and request #50,000. The database wasn't computing fifty thousand different answers. It was doing the exact same work, fifty thousand times in a row, to produce the exact same result.

If a coworker asked you the same question fifty thousand times in five minutes, you wouldn't recalculate the answer from scratch each time. You'd remember what you said the first time, and just repeat it. Your database doesn't get that luxury , it has no concept of "I already answered this." Every query looks brand new to it, even if it's identical to the one from half a second ago.

So the problem was never that the database is slow, or poorly designed. The problem is that we keep making it repeat work it has already done.

Section 2: Do We Really Need To Ask Again?

Let's slow down and really sit with this, because the instinct to "just ask the database" is so automatic that it's worth questioning directly.

Why does the application ask the database every time?

Because that's the default behavior we build into applications. A request comes in, the application has no memory of anything, so it goes and fetches the data fresh. It doesn't matter if the same data was fetched one second ago. The application doesn't remember that. It just knows how to ask.

But think about what "asking the database" actually costs.

The database has to:

Receive the query over the network
Parse and plan how to execute it
Read data from disk or its own memory
Filter, sort, or join tables if needed
Package the result and send it back over the network Every one of those steps takes time. Even a "fast" query that takes 20 milliseconds is still 20 milliseconds of real work , CPU cycles, disk reads, network round trips.

Now multiply that by 50,000 identical requests. That's 50,000 times the CPU work, 50,000 times the disk activity, 50,000 times the network traffic , for a result that never changed.

This is the moment where the solution should start to feel obvious, even before we name it. If the answer doesn't change, and we already computed it once, why compute it again?

Right now, every request takes the same path, no matter how many times it's been taken before:

Without a cache:

User → Application → Database → Response

Every single request, without exception, walks all the way down to the database. There's no shortcut, no memory of what was already asked.

Now picture inserting one extra stop along that path , something that can answer instantly if it already knows the answer, and only lets the request continue to the database if it doesn't:

With a cache:

User → Application → Cache → Database (only if needed) → Response

The database's job is to be the source of truth. It should be trusted to hold the correct data. But that doesn't mean it has to be asked every single time someone wants to read that data. What if, somewhere between the user and the database, we kept a copy of recent answers? A copy that's fast to check, and that we hand out instead of bothering the database again?

That idea — keeping a copy of an answer so you don't have to redo the work to get it again — is the entire foundation of caching. And it's worth being precise about what's actually being saved here. Caching isn't really about storing data. It's about avoiding expensive work. The data is just the byproduct of that work , a database scan, a sort, a computation. The cache exists so we never have to pay for that work twice when the answer hasn't changed.

Section 3: Meet The Cache

Let's define it the way it actually emerges, not the way a textbook would state it upfront.

A cache is a place where we store the result of an expensive operation, so that the next time someone asks for the same thing, we can hand them the stored result instantly , without redoing the expensive operation. The point was never the storage itself. The point is dodging the work that produced it.

That's it. That's the whole idea. Everything you'll learn about caching in the coming weeks - Redis, TTLs, invalidation strategies, cache layers is just refinement on top of this one sentence.

Let's walk through how this changes our trending-articles example.

The first user opens the homepage. The application has never answered this question before, so it goes to the database, gets the top 10 trending articles, and sends them back to the user.

But this time, before responding, the application does one more thing. It takes that result and stores it somewhere fast to access — let's just call it "the cache" for now with a label like trending_articles.

User 1 → App Server → Database: "Give me top 10 trending articles"
                     ← Database returns result
        App Server → Cache: "Save this as trending_articles"
        App Server → User 1: here's your result

Now user 2 shows up, asking for the same thing. This time, the application does something different. Before going anywhere near the database, it checks the cache first.

User 2 → App Server → Cache: "Do you have trending_articles?"
                     ← Cache: "Yes, here it is."
        App Server → User 2: here's your result

No database involved. No query planning, no disk reads, no sorting. Just a quick lookup in a place designed to answer fast.

User 3, user 4, all the way to user 50,000 , same story. As long as the cached result is still considered valid, they all get served from the cache. The database, which used to handle 50,000 identical queries, now handles just one.

That single shift checking a fast, temporary storage location before going to the slow, authoritative one — is what a cache does. It sits between your application and your database (or any expensive computation, really) and intercepts repeat requests before they become repeat work.

Section 4: Cache Hit vs Cache Miss

Once you accept the basic idea, two very natural situations start to matter, and it's worth naming them clearly because you'll see these terms everywhere from now on.

Cache hit: the application checks the cache, and the data it's looking for is already there. It hands that data back immediately, without touching the database.

Cache miss: the application checks the cache, and the data isn't there. Maybe nobody has asked for it yet, or maybe it was stored earlier but has since been removed. In this case, the application has no shortcut. It has to go to the database, get the answer the normal way, and importantly store it in the cache before responding, so that the next request becomes a hit.

Let's trace through both cases with a slightly different example: user profile pages.

Imagine a user named Aisha visits her own profile page for the first time today.

Request: GET /profile/aisha

App Server checks cache for key "profile:aisha"
Cache: "I don't have that." → Cache Miss

App Server → Database: "Get profile data for aisha"
Database → App Server: returns profile data

App Server → Cache: "Store this as profile:aisha"
App Server → User: here's the profile

That was a miss. Slower path, database involved, but the cache is now "warmed up" with Aisha's data.

Now, thirty seconds later, Aisha refreshes the page, or a friend visits her public profile.

Request: GET /profile/aisha

App Server checks cache for key "profile:aisha"
Cache: "Yes, here it is." → Cache Hit

App Server → User: here's the profile

That was a hit. No database involved at all. Just a lookup.

This is the pattern that repeats constantly in real systems: the first request for something is always a miss, because nothing has been cached yet. But every subsequent request for that same thing as long as it stays in the cache becomes a hit, and hits are dramatically cheaper than misses.

This is also why caching helps most for data that's read far more often than it changes. Trending articles, product listings, user profiles, popular search results these are all things that thousands of people might request in a short window, while the underlying data itself barely changes minute to minute. That's exactly the kind of workload where caching turns a database from "constantly overwhelmed" into "occasionally consulted."

Section 5: Why Is Cache So Fast?

At this point you might be wondering something completely reasonable: if a cache is just "storage" and a database is also "storage," why is one so much faster than the other?

The answer comes down to where the data physically lives.

A traditional database, especially for data that doesn't all fit comfortably in memory, keeps a lot of its data on disk. Even fast solid-state disks are still slower than the alternative we're about to talk about, because reading from disk involves physically locating and retrieving data through a storage controller.

A cache, on the other hand, almost always stores its data in RAM the computer's main memory.

Here's the difference that matters: RAM is designed for extremely fast access by the CPU, while persistent storage is optimized for keeping data safely over time, not for handing it back instantly. That's why reading from RAM is significantly faster than reading from disk nanoseconds instead of fractions of a millisecond. That might sound like a small difference on paper, but at the scale of thousands of requests per second, it's the difference between a database that's gasping for breath and one that barely notices the traffic.

There's a rough mental model worth keeping:

CPU register access     →  fastest (fractions of a nanosecond)
RAM access               →  very fast (nanoseconds)
SSD disk access          →  fast, but much slower than RAM
Network round trip       →  slower still
Traditional spinning disk → slowest of the common options

A cache deliberately trades one thing for the speed it gains: RAM is more expensive per gigabyte than disk, and it's volatile if the machine loses power, whatever's in RAM disappears. That's precisely why we don't use a cache as our source of truth. The database still holds the real, permanent data on disk, safely persisted. The cache just holds a temporary, disposable copy of the most frequently requested answers, sitting in memory where it can be handed out almost instantly.

This is also why a cache being "wrong" is a survivable problem. If the cache is lost a server restarts, the cache is cleared, whatever — nothing is actually lost. The application just experiences a wave of cache misses, falls back to the database, and starts rebuilding the cache from scratch. The database is still there as the safety net.

That single property cache is fast but temporary, database is slower but permanent is the core tradeoff that everything else in caching design is built around.

Section 6: Redis — The Most Popular Cache

So far we've talked about "the cache" as a concept, without naming any specific technology. That was deliberate. The idea needed to make sense on its own before attaching a name to it.

In practice, when engineers build this "fast, in-memory storage layer sitting in front of the database" idea, one of the most popular tools they reach for is Redis.

Redis is, at its core, an in-memory data store. It keeps data in RAM, which as we just established is exactly what makes it fast enough to serve as a cache. It's typically used as a separate service that your application servers talk to, sitting between them and the database.

Here's roughly how our trending-articles flow looks with Redis in the picture:

                 ┌───────────────┐
   Request  ───► │  App Server   │
                 └──────┬────────┘
                        │
                   check cache first
                        │
                 ┌──────▼────────┐
                 │     Redis     │  ← fast, in-memory
                 └──────┬────────┘
                        │ (only on a miss)
                 ┌──────▼────────┐
                 │   Database    │  ← slower, on disk
                 └───────────────┘

Redis supports simple commands that map naturally to what we've been describing storing a value under a key, and retrieving a value by that key. Something conceptually like:

SET trending_articles "[...serialized list of articles...]"
GET trending_articles

That SET is what happens on a cache miss, right after the application gets fresh data from the database. That GET is what happens on every request afterward, checking whether a cached answer already exists.

It's worth being honest here: Redis isn't the only way to build a cache, and it does a lot more than just basic caching once you dig into it. But for a beginner building a mental model of System Design, the important thing isn't memorizing Redis's full feature set. It's understanding that Redis is simply one popular, well-built tool for doing the thing we just spent this entire article reasoning our way toward — storing answers in fast memory so you don't have to keep recomputing them.

Once you understand why caching exists, learning any specific caching tool becomes a matter of syntax, not concept.

Conclusion

Let's retrace the path we took today.

We started with a database drowning under repeated, identical queries. We asked whether it made sense to keep asking the same question over and over, and realized it didn't the answer wasn't changing, so the work was wasted.

That led us to the idea of a cache: a fast, temporary storage layer that holds onto previously computed answers so future requests can be served instantly, without touching the database at all.

We learned to tell a cache hit from a cache miss, and saw why the very first request for something is always a miss, while everything after tends to become a hit as long as the cached data is still there.

We looked at why caches are fast in the first place: RAM instead of disk, nanoseconds instead of milliseconds, at the cost of being temporary rather than permanent.

And finally, we named Redis as one of the most common tools engineers reach for to build exactly this kind of caching layer in real systems.

But notice something we conveniently avoided this entire article.

We stored trending_articles in the cache. We stored profile:aisha in the cache. But what happens when Aisha updates her profile picture five minutes later? The cache still confidently hands out the old profile data, because as far as it knows, nothing changed. It has no idea the underlying data in the database was just updated.

The cache doesn't automatically know when the real data changes. It just keeps serving whatever it was told to remember until something tells it otherwise.

So the question we're left with is this:

How does a cache know when the data it's holding is no longer correct?

That question is exactly where Part 7 begins.