DEV Community

Cover image for 15,000 Requests Per Second: The 15-Minute API Survival Test
Elvis Manzi Rurangirwa
Elvis Manzi Rurangirwa

Posted on

15,000 Requests Per Second: The 15-Minute API Survival Test

#systemdesign #monitoring #architecture #devdiscuss

Your API is receiving 15,000 requests every second.

The servers are choking.

Latency is climbing.

Users are refreshing pages that never finish loading.

Your error rate is turning red.

And someone just posted in Slack:

“Is production down?”

You have 15 minutes.

What do you do?

The obvious answer is:

“Add more servers.”

It might also be the fastest way to make the outage worse.

This is how I would approach the incident.

1. First Rule: Don't Start Coding

When production is burning, my first instinct should not be opening IntelliJ and changing application code.

The first question is:

What is actually saturated?

15,000 requests per second tells me how much traffic we're receiving.

It doesn't tell me what's killing the system.

I want to see:

  • CPU utilization
  • Memory utilization
  • Request latency
  • p95/p99 latency
  • HTTP 5xx rate
  • Database CPU
  • Database connections
  • Connection-pool utilization
  • Cache hit rate
  • Network throughput
  • External API latency
  • Queue depth
  • Traffic by endpoint
  • Traffic by client/IP
  • Recent deployments

The objective is simple:

Find the bottleneck before trying to fix it.

Minute 0–2: Stop Making Things Worse

Before optimizing anything, I want to reduce the pressure on the system.

Check the deployment timeline

Did something change five minutes ago?

Maybe we deployed:

v1.8.1
Enter fullscreen mode Exit fullscreen mode

And immediately afterward:

Latency ↑
CPU ↑
DB queries ↑
5xx ↑
Enter fullscreen mode Exit fullscreen mode

That's a strong signal.

If the evidence points toward the deployment, rollback.

A rollback that takes 30 seconds is infinitely better than spending 10 minutes trying to understand a regression while thousands of users are experiencing it.

This is incident response, not a coding competition.

Minute 2–5: Put a Bouncer in Front of the API

Imagine a nightclub with a capacity of 500 people.

If 10,000 people show up simultaneously, opening the doors wider isn't going to solve the problem.

You need a bouncer.

For an API, that bouncer is rate limiting.

                    Internet
                       │
                       ▼
              ┌─────────────────┐
              │ Load Balancer   │
              │ + Rate Limiter  │
              └────────┬────────┘
                       │
                       ▼
                  API Servers
Enter fullscreen mode Exit fullscreen mode

If one client is generating thousands of requests per second, I don't want those requests consuming resources needed by everyone else.

I'd enforce limits such as:

Normal client:
100 requests/sec

Authenticated client:
500 requests/sec

Expensive endpoint:
20 requests/sec
Enter fullscreen mode Exit fullscreen mode

Excess traffic gets:

HTTP/1.1 429 Too Many Requests
Retry-After: 5
Enter fullscreen mode Exit fullscreen mode

The goal isn't to make the API inaccessible.

It's to make sure one consumer cannot take the entire system down.

But What If the Traffic Is Legitimate?

This is where things become interesting.

Suppose there isn't an attack.

Our customers genuinely generated 15,000 requests per second.

Rate limiting alone won't solve the problem.

Now I need to understand what kind of traffic we're dealing with.

For example:

GET /products       9,000 RPS
GET /profile        3,000 RPS
POST /orders        1,500 RPS
GET /analytics        500 RPS
Enter fullscreen mode Exit fullscreen mode

Immediately, I know something important.

Most traffic is reads.

That changes my strategy.

Minute 5–8: Scale the Application

If the application servers are CPU-bound and the application is stateless, horizontal scaling is probably appropriate.

Instead of:

              Load Balancer
                    │
                  API-1
Enter fullscreen mode Exit fullscreen mode

we move toward:

                 Load Balancer
                /      |      \
             API-1   API-2   API-3
               │       │       │
             API-4   API-5   API-6
Enter fullscreen mode Exit fullscreen mode

If we're running Kubernetes, the deployment might scale from:

3 replicas
Enter fullscreen mode Exit fullscreen mode

to:

20 replicas
Enter fullscreen mode Exit fullscreen mode

But here's the trap.

More API servers do not automatically mean more capacity.

Suppose every API request performs a database query.

We start with:

3 API servers
      ↓
Database
Enter fullscreen mode Exit fullscreen mode

The database is already at:

CPU: 98%
Connections: 100%
Enter fullscreen mode Exit fullscreen mode

Adding 17 more API servers could turn:

15,000 RPS
Enter fullscreen mode Exit fullscreen mode

into an even more aggressive database attack.

We haven't fixed the problem.

We've amplified it.

Minute 8: Follow the Bottleneck

This is one of the most important principles in distributed systems:

Scale the bottleneck, not the component you happen to be looking at.

Suppose the dashboard now says:

API CPU:              42%
API memory:           55%

Database CPU:         99%
Database connections: exhausted

Redis CPU:            15%
Enter fullscreen mode Exit fullscreen mode

The API isn't our problem.

The database is.

So now I ask:

Why is the database receiving so much work?

The Hidden Killer: The N+1 Query

Imagine this endpoint:

GET /orders
Enter fullscreen mode Exit fullscreen mode

The application retrieves 100 orders:

SELECT * FROM orders LIMIT 100;
Enter fullscreen mode Exit fullscreen mode

Then for every order:

SELECT * FROM users WHERE id = ?;
Enter fullscreen mode Exit fullscreen mode

That's potentially:

1 query
+
100 queries

= 101 database queries
Enter fullscreen mode Exit fullscreen mode

At:

1,000 requests/sec
Enter fullscreen mode Exit fullscreen mode

we could create:

101,000 DB queries/sec
Enter fullscreen mode Exit fullscreen mode

Suddenly the mystery disappears.

The API isn't necessarily slow.

The application is generating an unreasonable amount of database work.

Minute 9: Introduce Caching

If thousands of users are requesting the same information, hitting the database every single time is wasteful.

Consider:

GET /products/123
Enter fullscreen mode Exit fullscreen mode

If the product doesn't change frequently, why execute:

SELECT * FROM products WHERE id = 123;
Enter fullscreen mode Exit fullscreen mode

thousands of times per second?

Put a cache in front of it.

                 Request
                    │
                    ▼
                 Redis
                /     \
             HIT       MISS
              │          │
              ▼          ▼
           Response      DB
                           │
                           ▼
                         Redis
                           │
                           ▼
                        Response
Enter fullscreen mode Exit fullscreen mode

Now imagine:

10,000 requests/sec
Enter fullscreen mode Exit fullscreen mode

with an:

85% cache hit rate
Enter fullscreen mode Exit fullscreen mode

Instead of sending approximately 10,000 reads/sec to the database, we may reduce the database workload dramatically.

That can be the difference between:

Database dying
Enter fullscreen mode Exit fullscreen mode

and:

Database comfortably serving traffic
Enter fullscreen mode Exit fullscreen mode

But Caching Has a Dark Side

Caching isn't free.

Now we have to think about:

  • Cache invalidation
  • Stale data
  • TTLs
  • Cache stampedes
  • Memory limits
  • Serialization
  • Hot keys

Imagine a cache entry expires:

product:123
Enter fullscreen mode Exit fullscreen mode

at exactly the same time.

Suddenly thousands of requests miss the cache simultaneously.

They all hit the database.

That's a cache stampede.

The cache that was supposed to protect our database just helped attack it.

Possible mitigations include:

  • Request coalescing
  • Jittered TTLs
  • Background refresh
  • Locks
  • Stale-while-revalidate strategies

Scaling isn't just adding technology.

It's understanding the failure modes of that technology.

Minute 10–12: Get Expensive Work Out of the Request

Now suppose our endpoint does this:

HTTP Request
     │
     ├── Validate request
     ├── Query database
     ├── Generate PDF
     ├── Send email
     ├── Call payment provider
     ├── Generate analytics event
     └── Return response
Enter fullscreen mode Exit fullscreen mode

That's a terrible design under extreme load.

Why should the user wait for an email to be sent before receiving the HTTP response?

Why should generating a report block an API thread?

Why should analytics prevent a successful response?

Instead:

                  API
                   │
                   ▼
                 Queue
             /     |      \
            /      |       \
       Worker   Worker   Worker
          │        │        │
        Email    PDF     Analytics
Enter fullscreen mode Exit fullscreen mode

The API handles the work that must happen now.

The queue handles work that can happen later.

This reduces request duration and protects application threads.

Minute 12–14: Protect the Database

There's another dangerous component:

connection pools.

Suppose every API instance can open:

100 DB connections
Enter fullscreen mode Exit fullscreen mode

and we have:

20 API instances
Enter fullscreen mode Exit fullscreen mode

That's potentially:

2,000 database connections
Enter fullscreen mode Exit fullscreen mode

But the database may only be designed to handle a fraction of that.

So scaling the application can actually destroy the database.

I'd review:

  • Maximum pool size
  • Connection timeout
  • Query timeout
  • Idle timeout
  • Maximum request concurrency
  • Database connection limits

I'd rather have requests fail quickly than have the entire system become unresponsive.

Minute 14–15: Verify the Recovery

At this point, I don't celebrate because the dashboard looks slightly better.

I verify.

I'd watch:

                 Before       After

RPS              15,000       15,000
p99 latency         8s          400ms
5xx rate            18%           0.4%
DB CPU               99%           65%
DB connections      100%           60%
Cache hit rate        0%           87%
Queue depth          N/A           Stable
Enter fullscreen mode Exit fullscreen mode

And most importantly:

Can users complete the critical workflows?

For example:

Login → Browse → Add to cart → Checkout
Enter fullscreen mode Exit fullscreen mode

If those work reliably, we've stabilized the incident.

The Incident Is Not Over

This is where many engineers stop.

The dashboard is green.

Slack is quiet.

Everyone goes back to work.

But we haven't answered the most important question:

Why did this happen?

After stabilization, I'd perform a proper incident investigation.

Maybe the root cause was:

Traffic spike
      ↓
No effective rate limiting
      ↓
API concurrency increased
      ↓
N+1 queries amplified DB load
      ↓
DB connection pool exhausted
      ↓
API latency increased
      ↓
Clients retried requests
      ↓
Traffic increased even further
      ↓
System collapsed
Enter fullscreen mode Exit fullscreen mode

That last part is particularly nasty.

Retries can create a positive feedback loop.

The system gets slow.

Clients retry.

Traffic increases.

The system gets even slower.

Clients retry again.

Eventually:

The clients become part of the outage.

The Permanent Fix

Once production is stable, I'd implement several layers of protection.

1. Load testing

Don't wait for production to discover the limit.

Test:

1k RPS
5k RPS
10k RPS
15k RPS
20k RPS
30k RPS
Enter fullscreen mode Exit fullscreen mode

Find the actual breaking point.

2. Autoscaling

Automatically scale based on meaningful signals such as:

  • CPU
  • memory
  • request concurrency
  • latency
  • queue depth

Not just CPU.

3. Rate limiting

Protect the API from abusive or unexpectedly aggressive clients.

4. Caching

Cache high-volume, read-heavy workloads.

5. Database optimization

Fix:

  • Missing indexes
  • N+1 queries
  • Expensive joins
  • Unnecessary queries
  • Poor pagination

6. Asynchronous processing

Move expensive non-critical operations into queues.

7. Circuit breakers

If an external service becomes unhealthy:

API → Payment Service
Enter fullscreen mode Exit fullscreen mode

don't allow every request to wait until timeout.

A circuit breaker can fail fast:

API → Circuit Breaker → Payment Service
                          X
                       DOWN
Enter fullscreen mode Exit fullscreen mode

This prevents one failing dependency from taking down your entire system.

8. Observability

You can't operate what you can't see.

I'd want:

Metrics
Logs
Distributed traces
Alerts
Dashboards
Enter fullscreen mode Exit fullscreen mode

And alerts around things users actually experience:

p95 latency
p99 latency
error rate
saturation
availability
Enter fullscreen mode Exit fullscreen mode

The Real Lesson

The interesting part of the problem isn't the number 15,000.

It could be:

1,500 RPS
15,000 RPS
150,000 RPS
Enter fullscreen mode Exit fullscreen mode

The engineering process remains similar.

Don't guess.

Measure.

Find the bottleneck.

Protect the system.

Reduce unnecessary work.

Scale the correct layer.

Verify recovery.

Then fix the root cause.

A production system isn't resilient because it has 100 servers.

It's resilient because when one component starts failing, the architecture prevents that failure from becoming everyone else's problem.

My 15-Minute Mental Model

When production is burning, I keep this sequence in my head:

             ┌──────────────┐
             │   INCIDENT   │
             └──────┬───────┘
                    │
                    ▼
              STOP THE BLEEDING
                    │
                    ▼
             FIND THE BOTTLENECK
                    │
          ┌─────────┼─────────┐
          ▼         ▼         ▼
        CPU         DB      Dependency
          │         │         │
          ▼         ▼         ▼
       Scale      Cache    Circuit
       Horiz.     /Query   Breaker
          │         │         │
          └─────────┼─────────┘
                    ▼
             REDUCE WORK
                    │
                    ▼
              VERIFY RECOVERY
                    │
                    ▼
             ROOT-CAUSE ANALYSIS
                    │
                    ▼
             PREVENT RECURRENCE
Enter fullscreen mode Exit fullscreen mode

That's the difference between making an API work and engineering a system that survives failure.

And when someone asks me:

“Your API is receiving 15,000 requests per second and has 15 minutes before everything falls over. What do you do?”

My answer isn't:

“I add more servers.”

It's:

“I find out what is dying, protect it, reduce the work hitting it, scale the actual bottleneck, verify recovery, and then make sure we never have the same incident twice.”

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.