This is Part 8 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.
We've come a long way.
We taught our application to stop asking the database the same question over and over. We added a cache to absorb repeated reads, and we built strategies to keep that cache honest as data changed underneath it. At each step, the database had less to do.
And for a while, that was enough.
But something interesting kept happening. Even after caching was in place, even after the most common queries were being served from memory without touching the database at all, engineers would look at their database and find it still running hot. Still busy. Still the slowest part of the system.
The natural reaction is to wonder if something is broken. Did the cache fail? Are there too many cache misses? Is the invalidation strategy wrong?
Not necessarily. Sometimes the database is busy for a completely different reason.
To understand that reason, we need to think carefully about the kind of work caching actually eliminates, and the kind it can't touch.
--
Section 1: Caching Solved One Problem, Not All of Them
Think back to why caching made sense in the first place.
The database was being asked the same questions, by thousands of different users, over and over again. "What are the trending articles right now?" "What products are on sale?" "What does the homepage look like?" The answers didn't change between requests, so computing them repeatedly was pure waste. The cache absorbed that waste by storing the answer once and handing it out to everyone who asked.
The key word there is same. Caching works beautifully when many users want the same thing.
But as an application grows, it doesn't just serve more users. It serves more kinds of requests. And a lot of those requests aren't the same at all.
Consider what happens when a real application is running at scale. Someone opens their email inbox. Someone else checks their bank balance. A third user scrolls through a feed of posts from the specific people they follow. A fourth checks the status of an order they placed three days ago.
Every one of those requests reaches the database. None of them can be served from a shared cache, because the answer is different for every single user. Your inbox is not my inbox. Your transaction history is not mine. Your feed is assembled from a different set of accounts than anyone else's.
This is the category of work that caching cannot help with, and it turns out this category is enormous. In most applications that have grown to real scale, the majority of database reads fall into exactly this bucket: personal, unique, and impossible to pre-store.
So after you've deployed a cache and reduced the repetitive work as much as possible, what's left is not waste. What's left is legitimate work.
The database isn't doing unnecessary work anymore. It's just doing too much legitimate work.
That's a different problem than the one caching solved. And it needs a different solution.
--
Section 2: Why Some Reads Can Never Be Cached
It helps to be precise about this, because the instinct when a database is struggling is always to reach for the cache. Before we move on, let's close that door properly.
A cache works by storing the result of a query under a key, and returning that stored result to anyone who asks the same question. The efficiency comes entirely from sharing one stored result across many requesters.
Now imagine trying to cache Aisha's notification feed. You store it under the key notifications:aisha. The next time Aisha checks her notifications, the cache serves it instantly.
But Aisha is one user. That cached result is used by exactly one person. It gets stale the moment she receives a new notification. And there are ten million users on this platform, each with their own notifications, each with their own key in the cache, each requiring their own database query to populate.
You haven't reduced the work. You've just moved it, and added the overhead of maintaining a cache on top.
Shared cached data (works well):
trending_articles → served to 50,000 users from one cache entry
Personal cached data (doesn't help):
notifications:aisha → served to 1 user
notifications:ravi → served to 1 user
notifications:chen → served to 1 user
...ten million more
Caching personal data can make sense in specific, narrow situations. But it doesn't solve the underlying problem, which is that the database has to compute ten million different answers for ten million different people, and it only has one machine's worth of computing power to do it with.
Most web applications receive far more reads than writes. People browse far more than they post. They view far more than they update. They check their feeds, their histories, their dashboards, constantly, while writes happen occasionally.
This means that a database under heavy load is spending most of its time answering read queries. And many of those read queries are the kind that can't be shared.
So if one database is overwhelmed by read requests, and caching can't absorb them, what do you actually do?
Section 3: One Database Isn't Enough: Meet the Read Replica
Here's the simplest version of the question.
If the database is struggling because too many reads are arriving at once, and you can't reduce the reads, what's the only other option?
Spread them across more than one database.
Think about what a read actually requires. The database needs a copy of the data, and it needs the ability to run a query against it. That's it. A read doesn't change anything. It doesn't modify any rows. It doesn't need to coordinate with other writes. It just looks something up and returns an answer.
That means, in principle, if you had two identical copies of the database, you could answer twice as many reads simultaneously. Half the reads go to the first copy, half to the second. Neither one has to work as hard.
This is the idea behind a Read Replica.
A Read Replica is an additional database server that holds a copy of all the data from your main database. It exists specifically to serve read queries, sharing the load so the primary database doesn't have to answer every single one.
Without Read Replicas:
App Servers
|
| (all reads AND writes)
|
v
[ Primary Database ] <-- doing everything, overwhelmed
With Read Replicas:
App Servers
|
|-- reads --> [ Read Replica 1 ]
|-- reads --> [ Read Replica 2 ]
|-- reads --> [ Read Replica 3 ]
|
|-- writes --> [ Primary Database ]
The primary database handles all writes. The replicas handle reads. Each replica has a full copy of the data, so any of them can answer any read query, just as the primary would.
And if traffic keeps growing, you add more replicas. The primary database stays the same. The read load gets distributed across however many replicas you need.
That's the whole idea. One database becomes several. Reads get shared across all of them. The primary is free to focus on writes.
--
Section 4: Sharing Reads Between Databases
The moment you have multiple databases that can serve reads, a practical question appears: how does the application decide which one to use?
This is called read/write splitting, and it's exactly what it sounds like. The application is configured to send writes to the primary database and reads to the replicas.
In practice, this can be handled in a few ways. Some applications make the decision in their own code, explicitly choosing which database connection to use depending on the operation. Some use a proxy layer that sits between the application and the databases, routing queries automatically based on whether they're reads or writes.
Application
|
v
[ Database Proxy / Router ]
| |
| writes | reads
v v
[ Primary ] [ Replicas ]
From the perspective of the application developer, this is mostly transparent. You write a read query, it goes to a replica. You write an insert or update, it goes to the primary. The routing logic handles the rest.
There's one important thing to understand about the replicas, though: they're not independent databases that you manage separately. They're kept in sync with the primary through a process called replication.
Replication means that whenever something is written to the primary database, that change is propagated to the replicas automatically. You don't manually copy data across. The database system does it for you, continuously, in the background.
Think of the primary as the source of truth, and the replicas as mirrors that are constantly trying to reflect it accurately. Every write that lands on the primary eventually flows out to every replica.
Eventually. That word is doing a lot of work, and it leads us to the most important nuance in this entire article.
--
Section 5: The Hidden Trade-off: Replication Lag
Replication isn't instant.
When Aisha updates her profile picture, the change is written to the primary database immediately. But the replicas don't receive that change at the exact same millisecond. The primary has to propagate the update out to each replica, and that takes time. Usually a very small amount of time, often measured in milliseconds. But time nonetheless.
During that brief window, something quietly uncomfortable is true.
The primary database has the new profile picture. The replicas still have the old one. If a request for Aisha's profile is routed to a replica before the update has arrived there, the user gets the old photo.
Timeline after Aisha's update:
t=0ms Primary receives write: new profile picture stored.
t=0ms Replicas: still have old picture.
t=12ms Replication completes: replicas now have new picture.
If someone reads Aisha's profile at t=5ms from a replica:
→ They see the old picture.
If someone reads Aisha's profile at t=20ms from a replica:
→ They see the new picture.
This gap between what the primary knows and what the replicas know is called replication lag. In healthy systems under normal conditions, it tends to be very small. Milliseconds. Users rarely notice.
But replication lag becomes significant under heavy write load, when the primary is processing many updates quickly and the replicas are struggling to keep up. The lag can stretch from milliseconds to seconds. In extreme cases, longer.
The question you have to ask for each application is: what happens if a user sees slightly stale data from a replica?
For many reads, the answer is: nothing significant. If Aisha's follower count shows 4,821 instead of 4,822 for a few milliseconds while replication catches up, the world doesn't end. That's a tolerable inconsistency.
For other reads, it matters more. Imagine Aisha updates her own profile and is immediately redirected to a page that shows her profile. If that read hits a replica that hasn't caught up yet, Aisha sees her own old photo staring back at her, even though she just changed it. That's confusing and feels broken, even if technically nothing went wrong.
This is the trade-off that Read Replicas introduce: you gain the ability to handle far more read traffic, but you accept that replicas might be slightly behind the primary at any given moment.
Different teams handle this in different ways. Some applications route certain sensitive reads, like "show me my own profile immediately after I edit it," to the primary rather than a replica, accepting slightly higher load on the primary for those specific operations. Others build in a brief delay before redirecting, giving replication a moment to catch up. Others simply accept that brief inconsistency for non-critical reads, and let replica lag resolve itself naturally.
There is no single right answer. The right answer depends on what your application is doing and what your users will actually notice.
This is the same lesson that appeared in cache invalidation: distributing data across multiple places gives you performance, and it introduces the possibility that those places disagree. Any time data lives in more than one location, those locations can fall out of sync. Every strategy for scaling a database has some version of this trade-off embedded inside it.
--
Conclusion
Let's take stock of where our architecture stands.
We started this series with a single server handling a single user. We scaled application servers horizontally, spreading traffic across many machines. We added a cache to eliminate repeated database reads and taught it to stay accurate as data changed.
Now we've taken the same horizontal scaling idea and applied it to the database layer itself, but only to reads. The primary database handles writes, the source of truth that everything else flows from. The read replicas distribute the vast majority of traffic across multiple machines, each one a full copy of the data, each one answering queries that the primary no longer has to touch.
The result is a system that can serve an enormous volume of reads without the database becoming the ceiling.
But notice something.
Today we taught one database to become many. Reads are now distributed across replicas, each one a full mirror of the primary, each one sharing the load.
But every one of those databases contains exactly the same data.
And every write still has to reach the same primary database. The replicas help enormously with reads, and reads dominate most applications, but they do nothing for writes. Adding ten replicas doesn't make writes any faster. The primary is still the only machine allowed to accept new data.
For a long time, that's fine. Writes are less frequent than reads, and a single well-provisioned primary can handle a lot. But applications keep growing.
Eventually, the problem is no longer reading the data.
The problem is storing it.
What happens when a single database can no longer hold everything, no matter how powerful the machine running it is? When the data itself outgrows the box?
That's where Part 9 begins.
Top comments (0)