What You'll Learn
By the end of this article, you'll understand:
- Why applications can still become slow even after adding more servers.
- Why the database often becomes the next bottleneck.
- Why scaling one part of a system doesn't automatically scale everything.
- How every application eventually discovers its next bottleneck.
- Why caching exists and the problem it solves.
In the previous article, we solved one of the biggest challenges in System Design.
Instead of relying on a single server, we introduced a Load Balancer that distributes incoming requests across multiple application servers. By sharing the workload, we were able to increase the application's capacity without forcing a single machine to handle every user.
Our architecture now looks something like this.
Users
│
▼
+------------------+
| Load Balancer |
+------------------+
│ │ │
▼ ▼ ▼
+--------+ +--------+ +--------+
|Server A| |Server B| |Server C|
+--------+ +--------+ +--------+
\ | /
\ | /
\ | /
+-------------+
| Database |
+-------------+
At first, everything works exactly as expected.
Instead of one server handling every request, the workload is now divided across multiple machines. CPU usage drops, memory pressure decreases, and users once again experience fast response times.
For a while, it feels like the scaling problem has finally been solved.
Then your application becomes even more popular.
More users sign up.
More requests begin arriving every second.
Traffic continues increasing.
Naturally, you open your monitoring dashboard expecting to see one of your servers struggling under the additional load.
But something surprising happens.
The application servers are healthy.
CPU utilization is comfortably below its limit.
Memory usage looks normal.
The Load Balancer is distributing requests exactly as it should.
Everything appears to be working perfectly.
Yet users are telling a different story.
Pages that used to load instantly are becoming slower.
Searching for products takes longer than before.
Refreshing the home page feels noticeably less responsive.
Some requests even begin timing out during busy periods.
At this point, many developers become confused.
If the application servers aren't overloaded anymore, where is the delay coming from?
The answer lies in a component that every request still depends on.
No matter which application server receives a request, they all have one thing in common.
Sooner or later, almost every request needs information from the same database.
This introduces one of the most important lessons in System Design.
Solving one bottleneck rarely solves the entire system.
More often than not, it simply reveals the next one.
Let's find out why the database becomes that next bottleneck.
--
One Database. Many Servers.
Let's take another look at our architecture.
Users
│
▼
+------------------+
| Load Balancer |
+------------------+
│ │ │
▼ ▼ ▼
+--------+ +--------+ +--------+
|Server A| |Server B| |Server C|
+--------+ +--------+ +--------+
\ | /
\ | /
\ | /
+-------------+
| Database |
+-------------+
At first glance, this architecture appears to solve our scaling problem.
Instead of relying on a single server, requests are now distributed across multiple machines. If one server becomes busy, another one can continue processing incoming requests.
From the perspective of the application servers, everything seems perfectly balanced.
However, if you look closely, you'll notice something interesting.
Although we now have three application servers, we still have only one database.
Every request that needs data eventually arrives at the same place.
Imagine three users visiting your application at exactly the same moment.
The Load Balancer distributes their requests perfectly.
User 1 ─────► Server A
User 2 ─────► Server B
User 3 ─────► Server C
So far, everything looks great.
Now let's see what happens next.
Server A ─────► Database
Server B ─────► Database
Server C ─────► Database
Suddenly, all three servers are asking the same database for information at almost exactly the same time.
If one hundred users are using your application, this usually isn't a problem.
If one thousand users are active, the database has more work to do.
Now imagine one hundred thousand users.
Your Load Balancer continues distributing requests perfectly.
Your application servers continue processing requests efficiently.
But every one of those servers is still competing for the attention of a single database.
This is an important shift in the way we think about scaling.
Earlier in the series, we focused on the application server because that was the component struggling to keep up with incoming requests.
After introducing multiple servers, we solved that bottleneck.
However, we didn't reduce the number of database queries being made.
In fact, as our application became more popular, the total number of database queries increased dramatically.
The bottleneck didn't disappear.
It simply moved.
This is one of the most important ideas in System Design.
When you solve one performance problem, you often expose the next one.
Today, the application servers are no longer limiting your system.
The database is.
But why does the database become slow in the first place?
After all, isn't it just another computer running software?
To answer that question, we first need to understand what actually happens every time your application asks the database for information.
--
Why the Database Eventually Becomes the Slowest Part
Earlier in this series, we learned that every request passing through your application consumes CPU time and RAM.
However, many requests don't end there.
Most applications can't answer a user's request using only the information already available in memory. They need to retrieve data that has been stored permanently.
That's where the database comes in.
Consider something as simple as opening your Instagram home feed.
When you refresh the page, your application doesn't already know which posts should appear on your screen.
Instead, it has to ask the database several questions.
- Which user is currently logged in?
- Which accounts does this user follow?
- What are the latest posts from those accounts?
- Which posts should appear first?
Only after receiving those answers can the application prepare the response and send it back to your phone.
The same pattern exists in almost every modern application.
When you browse products on Amazon, the application asks the database for product information.
When you search for a movie on Netflix, the application asks the database for matching titles.
When you open your inbox, Gmail asks the database for your emails.
The database becomes the source of truth for almost everything your application needs.
This is what makes the database different from your application server.
The application server is responsible for processing requests.
The database is responsible for storing and retrieving data.
At first, this division of responsibility works extremely well.
The application performs its calculations, the database provides the required information, and the user receives a response within milliseconds.
The challenge begins as traffic increases.
Imagine that one user opens your application.
The database receives a few queries and responds almost instantly.
Now imagine one thousand users doing the same thing.
The application servers are no longer the limiting factor because we've distributed requests across multiple machines.
The database, however, still has to answer thousands of queries every second.
Every login request needs user information.
Every profile page needs profile data.
Every search requires looking through stored records.
Every product page needs product details.
The workload grows with every new user.
Unlike the application servers, which we can easily multiply by adding more machines behind the Load Balancer, our architecture still depends on a single database.
As more queries arrive, the database has to process an increasing amount of work.
Some queries complete immediately.
Others begin waiting.
Eventually, response times start increasing.
The application servers aren't waiting for the CPU anymore.
They're waiting for the database.
This is a subtle but extremely important change.
Earlier in the series, users experienced delays because the application server couldn't process requests fast enough.
Now the application server is often sitting idle, waiting for the database to return the information it needs.
From the user's perspective, the application still feels slow.
But the cause of the slowdown has completely changed.
This is why experienced engineers spend so much time identifying the true bottleneck before trying to optimize a system.
Optimizing a component that isn't causing the delay rarely improves performance.
Before you can make an application faster, you first need to identify what it's waiting for.
And in large applications, that answer is very often the database.
--
Do We Really Need to Ask the Database Every Time?
At this point, we've identified the problem.
Every request eventually reaches the same database, and as more users arrive, the number of database queries increases dramatically.
This raises an interesting question.
Do we really need to ask the database every single time?
Think about a typical e-commerce website.
Imagine you're browsing the homepage of an online store.
The list of "Today's Best Sellers" is displayed to every visitor.
Now suppose one thousand people open the homepage within the next minute.
If the application follows the same process for every request, the sequence might look like this.
Request 1
│
▼
Database
Request 2
│
▼
Database
Request 3
│
▼
Database
...
Request 1000
│
▼
Database
The interesting part is that every request is asking for exactly the same information.
The best-selling products haven't changed within those few seconds.
Yet the application keeps asking the database the same question over and over again.
The database faithfully executes every query, even though the answer is identical.
From the database's perspective, there is nothing unusual happening.
It simply receives a request, processes it, and returns the result.
However, from a system design perspective, this feels inefficient.
We're repeatedly asking for information that we already retrieved only moments ago.
Let's look at another example.
Suppose you open a news website.
The headline article is visible to every visitor.
If ten thousand people open the website within the next few minutes, should the application perform the exact same database query ten thousand times?
Probably not.
The data hasn't changed.
The answer is already known.
So why keep asking the database?
This simple observation introduces one of the most powerful ideas in System Design.
If some data is requested repeatedly and doesn't change very often, perhaps we shouldn't retrieve it from the database every single time.
Instead, we could temporarily keep a copy of that information somewhere much faster to access.
The next time another user asks for the same data, the application could simply reuse that copy instead of sending another query to the database.
Notice what we've done.
We haven't made the database faster.
We haven't upgraded the hardware.
We haven't changed our application's business logic.
We've simply reduced the number of times the database needs to do the same work.
This idea is known as caching.
Caching doesn't replace your database.
Your database still remains the source of truth for your application's data.
Instead, a cache acts as a temporary storage area for information that is requested frequently.
By answering repeated requests from the cache instead of the database, we reduce the workload on the database and improve the overall performance of the application.
Caching is one of the most widely used optimization techniques in modern software systems.
Whether you're using YouTube, Instagram, Amazon, Netflix, or countless other applications, chances are that caching is helping deliver information much faster than repeatedly querying the database.
But introducing a cache also creates new questions.
Where should this cached data be stored?
How long should it remain there?
And what happens when the original data changes?
Those are exactly the questions we'll answer in the next part of this series.
--
Wrapping Up
At the beginning of this article, we solved the problem of overloaded application servers by introducing a Load Balancer and distributing requests across multiple machines.
For a while, everything improved.
The servers had enough CPU power.
Memory usage was under control.
Users once again experienced fast response times.
But as our application continued to grow, we discovered something important.
Scaling one part of a system doesn't automatically scale the entire system.
Although we added more application servers, every one of those servers still depended on the same database. As the number of users increased, the number of database queries increased as well, eventually making the database the next bottleneck.
This teaches us one of the most valuable lessons in System Design.
Every time you solve a bottleneck, another one eventually appears.
The goal isn't to build a system that never has bottlenecks.
The goal is to identify the current bottleneck, solve it, and prepare for the next one.
In this article, the database became that bottleneck.
We also asked an important question.
If thousands of users are requesting the same information, do we really need to ask the database thousands of times?
The answer, as we discovered, is no.
Instead of repeatedly performing the same expensive work, we can temporarily store frequently accessed data somewhere much faster to retrieve.
That simple idea forms the foundation of one of the most important optimization techniques in modern software engineering:
Caching.
Almost every large-scale application on the internet relies on caching in one form or another. Whether you're scrolling through Instagram, browsing Amazon, watching YouTube, or reading articles online, caches are constantly working behind the scenes to reduce database load and deliver information more quickly.
In the next part of this series, we'll explore how caching works, where cached data is stored, and why technologies like Redis have become an essential part of modern System Design.
See you in Part 6.
Top comments (0)