What Happens When Your API Suddenly Gets 10 Million Requests?
One of the most interesting things about software engineering is that success creates new problems.
A small application has simple problems.
The database is slow.
A query needs optimization.
A page takes too long to load.
A user reports a bug.
Then something unexpected happens.
People actually start using your product.
Thousands become hundreds of thousands.
Hundreds of thousands become millions.
Suddenly, the problems change.
Your code didn't suddenly become worse.
The system simply entered a different world.
An API that worked perfectly yesterday might collapse tomorrow because the number of requests has changed.
This is where system design becomes real.
Because scaling isn't about adding more servers.
It's about understanding where pressure appears, how systems fail, and how we design software that can continue serving users even when demand grows beyond our expectations.
So let's imagine something simple.
We built a social media API.
Users can:
- Create accounts
- View profiles
- Publish posts
- Like posts
- Follow other users
Everything works perfectly.
The API handles 10,000 requests per day.
Then a famous creator shares the app.
Suddenly, millions of people arrive.
The API receives 10 million requests.
What happens next?
Let's find out.
Step 1: The First Problem Is Usually Not The Server
When developers hear "10 million requests," they often immediately think:
"We need more servers."
Sometimes that's true.
Often, it isn't the first problem.
The first question should be:
Where is the bottleneck?
A request travels through many layers.
User
↓
Load Balancer
↓
Application Server
↓
Database
↓
Cache
↓
External Services
Every layer has limits.
Your API server might handle millions of requests.
But your database might not.
Your database might be fast.
But an external payment API might be slow.
Scaling starts with observation.
Not assumptions.
Step 2: Understanding Request Volume
10 million requests sounds massive.
But let's break it down.
10 million requests per day:
10,000,000 / 86,400 seconds
Approximately:
116 requests per second
That is very different from:
10 million requests per minute.
Numbers matter.
System design is about understanding the actual workload.
Average traffic is important.
Peak traffic is even more important.
A product might average 100 requests per second but experience 5,000 requests per second during a viral event.
Systems usually fail during peaks.
Not averages.
Step 3: The Application Server
Initially, we might have one backend server.
Something like:
User
↓
Django / Node.js / Laravel API
↓
Database
For a small application, this is perfectly fine.
But now thousands of users arrive simultaneously.
The server has limited resources.
CPU.
Memory.
Network connections.
Threads.
Database connections.
Eventually requests begin waiting.
Response times increase.
Users see:
"Loading..."
Then:
"Something went wrong."
The first scaling step is usually horizontal scaling.
Step 4: Horizontal Scaling
Instead of one server, we create many.
Load Balancer
|
---------------------
| | |
API 1 API 2 API 3
|
Database
Now traffic is distributed.
One server no longer carries everything.
The load balancer becomes the traffic controller.
It decides:
"Which server should handle this request?"
This is one of the simplest and most powerful ideas in scalable systems.
Don't make one machine stronger.
Make the system capable of using many machines.
Step 5: The Stateless Principle
Horizontal scaling introduces a new challenge.
Imagine a user logs in.
Server 1 stores their session.
Then the next request goes to Server 2.
Server 2 doesn't know who they are.
The solution is making services stateless.
The server shouldn't remember important information locally.
Instead:
Authentication data goes into tokens.
Sessions go into shared storage.
Files go into object storage.
The database becomes the source of truth.
Stateless systems are easier to scale because every server is interchangeable.
Step 6: The Database Becomes The Real Problem
Eventually, the database becomes the bottleneck.
This happens often.
Applications scale horizontally.
Databases are harder.
Imagine millions of users loading profiles.
Every request does:
SELECT * FROM users WHERE id = ?
The database starts struggling.
Now we need optimization.
Step 7: Database Indexing
The simplest database optimization is often an index.
Without an index:
The database scans every row.
One million users?
Check one million records.
With an index:
The database can quickly find the information.
Like looking up a word in a dictionary.
Indexes are one of the most powerful examples of data structures in real systems.
Behind many database indexes are structures like B-trees.
Computer science becomes practical very quickly.
Step 8: Caching
Now imagine millions of users viewing the same popular profile.
Do we need to ask the database every time?
Probably not.
We introduce caching.
Request
↓
Cache
↓
Database (if missing)
A cache stores frequently requested information.
Examples:
Popular profiles.
Trending posts.
Product information.
Configuration.
Redis is commonly used for this.
But caching introduces new problems.
When does data expire?
What happens when information changes?
How do we avoid stale data?
Caching is powerful because it trades complexity for speed.
Step 9: The Cache Problem
Imagine someone changes their username.
The database updates.
But the cache still has the old value.
Now users see inconsistent information.
This is the famous cache invalidation problem.
One of the hardest problems in computer science is not creating caches.
It's knowing when to remove information from them.
Sometimes the simplest solution is the best.
Cache less.
Expire data quickly.
Only cache expensive operations.
Step 10: Database Replication
Reading data is usually easier than writing data.
So we separate them.
Primary database:
Handles writes.
Replica databases:
Handle reads.
Application
|
--------------
| |
Primary Replicas
Write Read
Now thousands of users can read information without overwhelming the main database.
This pattern appears everywhere.
Step 11: Background Jobs
Another mistake is making users wait for everything.
Imagine creating a post.
The API does:
Save post.
Send notifications.
Generate recommendations.
Resize images.
Update analytics.
Everything happens before responding.
Slow.
Instead:
User Action
↓
Save Data
↓
Queue Job
↓
Process Later
Background workers handle expensive tasks.
The user gets a fast response.
The system stays responsive.
Step 12: Message Queues
Queues become extremely valuable at scale.
A queue absorbs sudden traffic spikes.
Imagine one million notifications need to be sent.
Instead of processing everything immediately:
Application
↓
Message Queue
↓
Workers
↓
Notifications
The system processes work gradually.
Queues create stability.
Step 13: Rate Limiting
At 10 million requests, not every request is necessarily legitimate.
Some users refresh aggressively.
Some clients have bugs.
Some traffic is malicious.
Rate limiting protects the system.
Examples:
100 requests per minute per user.
1000 requests per minute per API key.
Rate limits are not about restricting users.
They're about protecting everyone.
Step 14: Monitoring
At this scale, guessing becomes dangerous.
You need visibility.
Measure:
- Response time
- Error rates
- Database performance
- CPU usage
- Memory usage
- Request volume
- Queue length
A system you cannot observe is a system you cannot improve.
Step 15: Logging
Logs become your system's memory.
A request fails.
Why?
Logs should tell the story.
User authenticated
Request received
Database query started
Database timeout
Response returned
Good logs turn debugging from detective work into engineering.
Step 16: Designing APIs For Scale
The API itself matters.
Poor API design creates future problems.
Avoid unnecessary data.
Instead of:
GET /users
returning everything:
GET /users?page=1&limit=20
Pagination protects your system.
Good APIs consider future growth.
Step 17: Design Patterns At Scale
Large systems rely on simple design principles.
Repository Pattern:
Separates data access.
Strategy Pattern:
Allows different behaviors.
Observer Pattern:
Supports events.
Factory Pattern:
Controls object creation.
Patterns don't magically make systems scalable.
But they help organize complexity.
Step 18: The Algorithm Behind Scaling
Scaling itself follows an algorithm.
Observe system
↓
Find bottleneck
↓
Improve bottleneck
↓
Measure results
↓
Repeat
There is no magical final architecture.
Every successful system evolves.
Step 19: What About Microservices?
At this point, many engineers immediately say:
"Let's create microservices."
Sometimes that is the right answer.
Sometimes it creates unnecessary problems.
Microservices introduce:
Network communication.
Deployment complexity.
Distributed debugging.
More infrastructure.
The question isn't:
"Can we use microservices?"
The question is:
"Do we have a problem that requires them?"
Architecture should follow reality.
Not trends.
The Bigger Lesson
A system receiving 10 million requests isn't fundamentally different from a system receiving 10,000.
The same principles appear.
Good database design.
Clear APIs.
Efficient algorithms.
Appropriate data structures.
Caching.
Queues.
Monitoring.
Thoughtful architecture.
The difference is that mistakes become more expensive.
At small scale, a bad decision affects hundreds of users.
At large scale, it affects millions.
Why This Matters To Me
One of the reasons I love system design is because it teaches a different way of thinking.
Programming asks:
"How do I make this work?"
System design asks:
"How do I make this continue working when everything changes?"
More users.
More data.
More traffic.
More uncertainty.
Great software isn't built by predicting every possible future.
It's built by creating systems that can adapt.
Final Thoughts
When your API suddenly receives 10 million requests, the solution isn't one technology.
It isn't Kubernetes.
It isn't microservices.
It isn't buying bigger servers.
The solution is understanding your system.
Finding where pressure appears.
Reducing unnecessary work.
Separating responsibilities.
Building for failure.
Measuring everything.
Improving continuously.
The best engineers aren't the ones who build the most complicated architectures.
They're the ones who know exactly how much complexity a problem deserves.
Because at the end of the day, scaling software is not really about handling millions of requests.
It's about building systems that remain trustworthy when millions of people depend on them.
Top comments (0)