This is Part 3 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.
What You'll Learn
By the end of this article, you'll understand:
- Why every request consumes server resources.
- What the CPU actually does when your application runs.
- Why RAM is just as important as the CPU.
- Why applications become slower as more users arrive.
- Why upgrading to a bigger server isn't always a permanent solution.
In the previous article, we stepped inside a server and followed the complete lifecycle of a request. We saw how the browser sends an HTTP request, how the server passes that request to the application, how the application retrieves information from the database, and finally how the response makes its way back to the browser.
At first glance, the entire process appears surprisingly straightforward. A request arrives, the application performs its work, and a response is returned to the user. For a handful of users, this happens so quickly that we rarely stop to think about what is happening behind the scenes.
This naturally raises an interesting question.
If a server can process a request in just a few milliseconds, why do applications become slow as they grow? Why does an application that performs perfectly during development begin struggling once thousands of people start using it? More importantly, why do some websites stop responding altogether during flash sales, ticket bookings, or product launches?
The answer isn't that the server suddenly becomes less efficient. It also isn't because your code mysteriously changed overnight. In most cases, the application is running exactly the same code that it was running yesterday.
What has changed is the number of requests competing for the server's attention.
To understand why this matters, imagine you've built a simple note taking application. A single user logs in, creates a note, edits another one, and logs out. Each of these actions sends a request to the server, and the server processes them almost instantly. The user enjoys a smooth experience because the server has very little work to do.
Now imagine that one hundred people are using the application at the same time.
Although the workload has increased, modern servers are designed to handle hundreds or even thousands of requests every second. The application may still feel just as responsive because the available resources are more than enough for the current demand.
Now increase that number again.
Imagine ten thousand users opening the application within the same minute.
Interestingly, the requests themselves haven't become more complicated. Every user is still performing the same actions logging in, creating notes, reading notes, and editing them. The application isn't doing anything fundamentally different.
The only difference is that the server now has far more work to complete in the same amount of time.
This is one of the most important ideas in System Design.
Applications rarely become slow because individual requests are difficult to process. They become slow because thousands of ordinary requests compete for the same limited resources.
At this point, another question naturally appears.
What are these resources?
Every request that reaches your server needs several things before it can be completed. It needs processing power to execute your application's code. It needs memory to temporarily store information while the request is being handled. It also needs network bandwidth to receive incoming requests and send responses back to users. If the application needs information from a database, accessing that data also consumes time.
None of these resources are unlimited.
Every server has a finite amount of processing power, memory, storage, and network capacity. As more requests arrive, those resources have to be shared among a growing number of users. Eventually, one of them becomes the bottleneck, preventing the server from processing requests as quickly as they arrive.
This is the moment when users begin noticing slower response times. Pages that once loaded instantly now take several seconds. Some requests begin waiting in line while others are still being processed. If the workload continues increasing, requests may eventually start timing out or fail altogether.
Many beginners see this situation and immediately reach for what seems like the obvious solution.
"Why not just buy a more powerful server?"
It's a reasonable question, and for some applications it works for a while.
However, upgrading to a larger server only postpones the problem. It doesn't eliminate it.
Before we can understand why, we first need to understand what actually happens every time your application executes a single line of code.
That journey begins with one of the most important components inside any computer:
The CPU.
--
The CPU: Who Actually Executes Your Code?
Every request that reaches your server eventually asks your application to do some work.
Sometimes that work is as simple as returning a homepage. Other times, it involves validating a password, calculating the total price of a shopping cart, generating recommendations, or preparing a dashboard filled with data.
Regardless of how simple or complex the request is, one thing remains true.
Somebody has to execute your code.
That "somebody" is the CPU.
The CPU, or Central Processing Unit, is often described as the brain of a computer. While that analogy is useful, it doesn't fully explain its role in a web application.
A better way to think about the CPU is this:
The CPU is the component that executes every instruction your application asks the computer to perform.
Whenever a request reaches your application, your code doesn't magically produce a result. Every if statement, every loop, every function call, and every mathematical operation is eventually translated into instructions that the CPU executes.
Consider this simple example.
def login(email, password):
user = find_user(email)
if verify_password(password, user.password_hash):
return "Login Successful"
return "Invalid Credentials"
At first glance, this looks like ordinary Python code.
However, when this function runs, the CPU performs the actual work behind every line.
It executes the function call.
It evaluates the if condition.
It performs the password verification.
It decides which value should be returned.
Although we write code using programming languages like Python, JavaScript, or Java, computers ultimately understand only machine instructions. The CPU is responsible for executing those instructions one after another until the request is complete.
This means that every request consumes CPU time.
For a single user, the amount of CPU time required is usually so small that it feels insignificant. A login request might take only a few milliseconds to process, and the server immediately becomes available for more work.
The situation changes when many users arrive at the same time.
Imagine that one person clicks the Login button.
The CPU spends a few milliseconds executing the necessary instructions, finishes the request, and moves on.
Now imagine one thousand people clicking Login at almost exactly the same moment.
Nothing about the code has changed.
The CPU still has to execute the same instructions for every request.
The only difference is that it now has one thousand times more work to complete.
This is an important idea to understand.
The CPU doesn't become slower because more people are using your application.
Instead, it simply has more instructions waiting to be executed.
A helpful way to think about this is to imagine a chef working in a restaurant.
When a single customer walks in, preparing one meal is straightforward.
If one hundred customers arrive together, the chef doesn't suddenly lose the ability to cook. The problem is that there are now far more orders than one person can prepare at the same time.
The CPU behaves in much the same way.
It continues executing instructions exactly as before, but as more requests arrive, the amount of work waiting to be completed grows rapidly.
This is why CPU usage becomes one of the most important metrics when monitoring a server.
When CPU usage is low, the server has plenty of processing power available for new requests.
As CPU usage increases, the server has less capacity to respond immediately.
Eventually, the CPU reaches a point where it is busy almost all the time. New requests still arrive, but they now have to wait for processing time before they can be completed.
From the user's perspective, this appears as a slow application.
The website hasn't changed.
The code hasn't changed.
The CPU is simply busy serving everyone else.
Understanding this helps explain an important principle in System Design.
A server doesn't become slow because requests are inherently difficult.
It becomes slow because every request requires CPU time, and CPU time is a limited resource.
Of course, the CPU isn't the only resource involved in processing a request.
While the CPU is busy executing your application's code, it also needs a place to temporarily store the data it is working with.
That temporary workspace is called RAM, and understanding its role is just as important as understanding the CPU itself.
--
RAM: Where Your Request Lives While It's Being Processed
Now that we understand the CPU's role, let's return to our login request.
When the CPU begins executing your application, it immediately encounters a new problem.
Where should it keep the information it's currently working with?
Suppose a user submits the following request:
{
"email": "john@example.com",
"password": "mypassword123"
}
The application now has several pieces of information that it needs while processing the request.
It needs to remember the user's email address. It needs to store the password received from the browser. After querying the database, it also needs to keep the user's information available while verifying the password. Finally, it has to prepare the response before sending it back to the browser.
All of this information needs to be stored somewhere while the request is being processed.
This is where RAM comes in.
RAM, or Random Access Memory, is the server's short-term working memory. Unlike the database, which stores information permanently, RAM only holds the data that the server needs right now.
A useful way to think about RAM is to imagine an office desk.
Suppose you're solving a complicated maths problem.
Your notebook, calculator, pen, and reference books are all spread across your desk while you're working. Once you've finished, you put everything away so the desk is ready for the next task.
RAM works in a very similar way.
While a request is being processed, all of the information required for that request is temporarily placed into RAM. Once the request is complete, most of that memory becomes available again for future requests.
This is why RAM is often called working memory.
It isn't designed to store your application's data forever.
Its purpose is to provide a fast workspace where the CPU can access the information it needs without constantly reading it from slower storage devices.
Let's look at our login example again.
As the request moves through the application, RAM temporarily stores information such as:
- The email address submitted by the user.
- The password received from the browser.
- The user record retrieved from the database.
- Variables created by your application's code.
- The final response before it's sent back to the browser.
These pieces of information may exist for only a fraction of a second, but during that time they occupy memory.
This isn't a problem when only a few users are using your application.
If ten people log in at the same time, the server simply allocates enough memory for each request and continues working.
Now imagine ten thousand users trying to log in simultaneously.
Each request still needs its own temporary workspace.
Each request still creates variables.
Each request still stores database results while your application is processing them.
Individually, these requests don't require much memory.
Collectively, they can consume several gigabytes of RAM.
As the available memory begins to fill up, the server has fewer resources available for new requests.
Eventually, it reaches a point where memory becomes another bottleneck.
Just like CPU time, RAM is a limited resource that every request competes for.
This is why adding more users to an application doesn't simply increase the amount of work the CPU performs. It also increases the amount of memory the server needs to keep thousands of requests active at the same time.
At this point, we've identified two important resources that every request depends on.
The CPU executes your application's instructions.
RAM stores the information required while those instructions are being executed.
Neither resource is unlimited.
As the number of users grows, the server must divide both CPU time and memory among an increasing number of requests.
Understanding how these two resources work together explains why applications eventually begin to slow down, even when the code itself hasn't changed.
--
When Thousands of Requests Arrive Together
By this point, we've identified two of the most important resources a server depends on.
The CPU executes your application's instructions, while RAM temporarily stores the data required to process each request.
As long as both of these resources are available, the server can continue processing requests smoothly.
Let's see what happens when traffic starts increasing.
Imagine your application is receiving only five requests every second. The CPU has more than enough time to execute your code, and there is plenty of memory available for every active request. Users experience fast response times because the server is comfortably handling the workload.
Now suppose your application becomes popular overnight.
Instead of five requests every second, it suddenly starts receiving five thousand.
At first, nothing looks different from the outside.
Every request is still asking the server to perform the same tasks.
The login request still verifies a password.
The profile page still fetches user information.
The dashboard still retrieves data from the database.
The requests haven't changed.
The number of requests has.
This creates an interesting situation.
While the CPU is busy processing one request, hundreds of other requests continue arriving.
Each of those requests also needs CPU time.
Each of them also needs memory.
However, the server doesn't suddenly receive more CPU cores or additional RAM simply because more users joined the application.
Instead, the existing resources have to be shared among everyone.
This is where response times begin to increase.
Imagine you're standing in a supermarket with only one billing counter.
If there are two customers in front of you, you'll probably be out within a few minutes.
If there are two hundred customers, the cashier hasn't become slower.
The billing process hasn't changed.
The only difference is that many more people are waiting for the same limited resource.
Servers behave in exactly the same way.
When the CPU is busy executing existing requests, newly arriving requests often have to wait before they can be processed.
Similarly, if a large number of requests are already occupying memory, the server has fewer resources available for incoming requests.
From the user's perspective, none of this is visible.
They simply notice that the application feels slower than usual.
A page that normally loads in 200 milliseconds might now take two or three seconds.
A login request that usually feels instant suddenly keeps showing a loading spinner.
Eventually, if the server becomes overwhelmed, some requests may even fail completely.
This is an important observation.
The application isn't slow because your algorithm suddenly became inefficient.
It isn't slow because the programming language changed.
In many cases, the code is exactly the same code that worked perfectly yesterday.
The difference is that your server is now trying to divide a fixed amount of CPU time and memory among a much larger number of users.
This idea lies at the heart of System Design.
When applications grow, the challenge isn't just writing correct code.
The challenge is ensuring that limited resources can continue serving an ever-increasing number of requests.
Naturally, this leads to another question.
If the problem is that the server doesn't have enough CPU power or memory, why not simply upgrade the machine?
At first, that sounds like the perfect solution.
And in many situations, it actually is.
But only for a while.
--
Can't We Just Buy a Bigger Server?
At this point, the solution might seem obvious.
If the server is running out of CPU power and memory, why not simply replace it with a more powerful machine?
In fact, this is exactly what many companies do when their applications begin experiencing higher traffic.
Suppose your application is currently running on a server with:
- 4 CPU cores
- 8 GB of RAM
As your user base grows, you notice that CPU usage is constantly close to 100%, and the server is beginning to respond more slowly.
One option is to upgrade the server.
You move your application to a machine with:
- 16 CPU cores
- 64 GB of RAM
Without changing a single line of code, your application can now handle significantly more traffic.
This approach is known as Vertical Scaling.
Vertical scaling simply means increasing the resources of an existing server instead of adding new ones.
If your application needs more processing power, you upgrade the CPU.
If it needs more working memory, you add more RAM.
From the application's perspective, nothing changes.
It is still running on a single server.
The only difference is that the server has become more powerful.
For many applications, this is an excellent solution.
It is relatively simple to implement because there is no need to redesign the architecture or distribute requests across multiple machines.
In fact, most applications spend the early stages of their life relying entirely on vertical scaling.
As traffic grows, the server is upgraded whenever additional resources are needed.
However, this approach has an important limitation.
No matter how powerful a server becomes, it will always have finite resources.
There will always be a maximum number of CPU cores.
There will always be a limit to how much RAM can be installed.
Eventually, you reach a point where upgrading the hardware becomes either technically impossible or financially impractical.
Even before reaching those limits, larger servers become increasingly expensive.
Doubling the available CPU power or memory doesn't always mean the application can handle twice as many users. At some point, the cost of buying bigger machines grows much faster than the performance benefits they provide.
This is why companies such as Netflix, Amazon, Google, and Instagram don't rely on a single extremely powerful server.
It's not because powerful servers don't exist.
It's because one server—no matter how powerful—is still just one server.
This realization changes the way engineers think about scaling.
Instead of asking,
"How can we make this server bigger?"
they begin asking a completely different question.
"What if one server didn't have to handle every request?"
That single question marks the beginning of modern System Design.
And it leads us directly to the next chapter of this series.
--
Wrapping Up
In this article, we explored why servers eventually slow down as applications grow.
We learned that every request consumes valuable resources, particularly CPU time and RAM. While a single request requires very little of either, thousands of simultaneous requests force the server to divide those limited resources among many users.
We also discovered that upgrading to a more powerful server—known as Vertical Scaling—can significantly improve performance, but only up to a point. Every machine has physical and financial limits, and eventually a single server is no longer enough to keep up with growing demand.
This realization introduces one of the most important ideas in System Design.
Sometimes the solution isn't to build a bigger server.
Sometimes the solution is to use more than one server.
But introducing multiple servers creates a new challenge.
If two identical servers are running your application, how does a user's request know which server it should go to?
We'll answer that question in the next part of this series as we explore one of the most fundamental components of modern web architecture:
The Load Balancer.
See you in Part 4.
Top comments (0)