An API can look perfectly fine during development.
You test it locally, everything responds quickly, and a single server seems more than enough.
Then traffic grows.
Suddenly, the server is handling thousands of requests at the same time. CPU usage increases, database queries pile up, response times get worse, and eventually requests start failing.
At that point, the solution usually isn't just a bigger server.
You need to change how the system is designed.
A common architecture looks like this:
Users
|
v
Load Balancer
/ | \
v v v
Server Server Server
\ | /
\ | /
Cache
|
v
Database
For work that doesn't need to happen during the request, a queue can be added:
API Server
|
v
Queue
|
+----> Worker
+----> Worker
+----> Worker
This is the point where backend development starts becoming system design.
The First Problem: One Server
Start with the simplest architecture:
Users
|
v
Server
|
v
Database
Every request reaches the same server.
That server handles authentication, business logic, database queries, API responses, and potentially background work too.
As traffic increases, its resources become limited:
CPU → High
Memory → High
Latency → Increasing
Errors → Increasing
You can move to a larger machine.
That's vertical scaling.
But eventually, a single machine still has a ceiling.
The other option is to add more servers.
Horizontal Scaling
Instead of:
Users
|
v
One Server
we can have:
Load Balancer
/ | \
v v v
Server Server Server
This is horizontal scaling.
The load balancer distributes incoming requests across available servers.
For example, if three servers are handling traffic:
10,000 requests/sec
|
v
Load Balancer
/ | \
3333 3333 3334
The actual distribution won't always be perfectly equal. It depends on the load-balancing strategy and the workload.
Common strategies include:
Round Robin
Least Connections
Weighted Routing
IP Hash
The important idea is simple:
Spread the work instead of depending on one machine.
Keeping Servers Stateless
Multiple servers introduce another problem.
Suppose a user logs in through Server 1.
If the login state exists only in Server 1's memory, the next request could reach Server 3.
Login
↓
Server 1
↓
Session stored locally
Next request
↓
Server 3
↓
Session not found
This is one reason scalable applications generally avoid keeping important shared state only in a server's local memory.
Instead, shared state can be stored in something accessible to all instances:
Server 1 ─┐
Server 2 ─┼──> Redis / Database
Server 3 ─┘
Now any server can handle the next request.
The Database Becomes the Next Bottleneck
Adding more backend servers doesn't automatically solve the whole problem.
Consider:
Load Balancer
/ | \
Server Server Server
\ | /
Database
Now three servers can process requests, but all of them are still querying the same database.
The database may become the bottleneck.
This leads to an important system-design principle:
When you remove one bottleneck, another one often becomes visible.
So instead of sending every request to the database, we can reduce unnecessary database work.
Caching
Suppose thousands of users request the same product:
GET /api/products/123
Without caching:
Request
↓
Server
↓
Database
↓
Response
With a cache:
Request
↓
Server
↓
Cache
|
+---- Hit → Response
|
+---- Miss
↓
Database
↓
Cache
↓
Response
If the requested data is already in the cache, the database doesn't need to process another query.
A simple example:
1,000 requests
↓
Cache
↓
900 cache hits
↓
100 database requests
The numbers are just an example. The actual benefit depends on the application's traffic and cache-hit rate.
Redis is commonly used for this kind of workload.
But Caching Creates Another Problem
What happens when the database changes?
Suppose the cache contains:
product:123 → ₹999
Then the database is updated:
₹999 → ₹899
The cache may still contain the old value.
Now the application can return stale data.
This is why caching isn't simply:
"Put Redis in front of the database."
You also need to think about:
TTL
Invalidation
Cache-aside
Write-through
Stale data
Caching can reduce database load significantly, but it introduces another system that needs to be designed and maintained.
Not Every Task Belongs in the Request
Some operations are expensive but don't need to finish before the user gets a response.
For example:
Upload video
↓
Process video
↓
Generate thumbnails
↓
Send notifications
Doing all of this inside the API request can make the endpoint slow.
Instead:
Client
|
v
API Server
|
v
Queue
|
+----> Worker 1
+----> Worker 2
+----> Worker 3
The API creates a job and puts it into the queue.
Workers process those jobs separately.
This works well for things such as:
Emails
Notifications
Image processing
Video processing
Reports
Data exports
The queue acts as a buffer between incoming requests and background processing.
A More Complete Architecture
Putting the pieces together:
Users
|
v
Load Balancer
|
+------------+------------+
| | |
v v v
Server 1 Server 2 Server 3
| | |
+------------+------------+
|
+------+------+
| |
v v
Cache Database
|
v
Read Replica
Server
|
v
Queue
|
+------+------+
| | |
v v v
Worker Worker Worker
Each component has a different responsibility:
Load Balancer
→ Distributes requests
Servers
→ Run application logic
Cache
→ Avoids repeated expensive reads
Database
→ Stores persistent data
Queue
→ Holds asynchronous work
Workers
→ Process background jobs
That's the important part of this architecture.
You're not adding technologies randomly.
You're adding them because a specific part of the system needs help.
What Happens During a Traffic Spike?
Imagine your application normally receives:
10,000 requests/minute
Then traffic suddenly jumps:
100,000 requests/minute
A scalable architecture might handle the increase like this:
Traffic Spike
↓
Load Balancer
↓
Multiple Servers
↓
Cache handles repeated reads
↓
Database handles remaining queries
↓
Queue absorbs background work
↓
Workers process jobs
If the infrastructure supports autoscaling, more application instances can be started as demand increases.
But simply adding servers isn't enough.
You have to understand where the actual bottleneck is.
The Database Has Its Own Scaling Problems
At larger workloads, the database may need additional techniques:
Indexing
Connection Pooling
Query Optimization
Read Replicas
Partitioning
Sharding
For example:
Database
/ \
v v
Primary Read Replica
Writes Reads
But these aren't default requirements for every application.
If a database handles your workload comfortably, adding replicas or sharding can add unnecessary complexity.
The architecture should follow the workload.
One Million Requests Doesn't Tell the Whole Story
The number 1 million requests sounds huge, but request count alone doesn't tell us how difficult the system is.
Consider:
1 million requests
→ Simple cached reads
versus:
1 million requests
→ Complex queries
→ Multiple database writes
→ External API calls
→ File processing
These are very different workloads.
When designing for scale, ask:
How many requests per second?
How expensive is each request?
How much data does each request access?
How much traffic can be cached?
How many requests reach the database?
Which operations can run asynchronously?
What happens when a server fails?
These questions are much more useful than simply asking:
"Can my backend handle 1 million requests?"
The Mental Model
When traffic increases, don't immediately think:
"I need a bigger server."
Think:
"Where is the bottleneck?"
Then solve that specific problem.
More Traffic
↓
Load Balancer
↓
More Servers
↓
Cache
↓
Database
↓
Queue + Workers
The load balancer distributes traffic.
Multiple servers increase application capacity.
The cache reduces repeated work.
The database stores persistent data.
The queue moves expensive work out of the request path.
Workers process that work independently.
That's the transition from building an API to engineering a system.
A backend that works for 100 users and a backend that works for millions of users can use the same basic code.
What changes is how you design the system around that code.
Top comments (0)