DEV Community

Cover image for What Actually Happens When Your API Gets 10,000 Requests in 1 Minute
Akshay Kurve
Akshay Kurve

Posted on

What Actually Happens When Your API Gets 10,000 Requests in 1 Minute

#backend #architecture #webdev #performance

It sounds like a good problem to have. More users. More traffic. More growth.

But if your system was not built for it, 10,000 requests in a minute will not feel exciting. It will feel like everything is breaking at once, and you will not immediately understand why.

Let's walk through what actually happens inside your backend when traffic spikes, why systems fail under load, and what you can do to prepare before the spike arrives.

Table of Contents

Why This Matters More in 2026

API reliability is not an abstract concern. It is directly tied to revenue, user trust, and business survival.

The numbers have gotten harder to ignore. According to Gartner, the average IT outage costs roughly $5,600 per minute, or about $300,000 per hour. And that benchmark is conservative for many companies. More recent 2024-2025 studies show that number trending upward, with some large enterprises reporting $10,000 to $14,000 or more per minute when core platforms fail.

Meanwhile, API reliability itself is getting worse, not better. Global API downtime surged 60% between Q1 2024 and Q1 2025. Average weekly API downtime rose from 34 minutes to 55 minutes, costing businesses millions in lost revenue. The reason is not that engineers have gotten worse at their jobs. The rising volume of AI-driven API calls and reliance on third-party SaaS companies are key factors placing a strain on uptime and performance.

So when we talk about what happens when 10,000 requests hit your API in a minute, this is not a thought experiment. It is Tuesday.

Breaking Down the Number

10,000 requests per minute means roughly 166 requests per second.

That does not sound massive until you think about what each of those requests actually does:

  • Hits your server (or load balancer)
  • Runs business logic
  • Queries your database (possibly multiple times)
  • Potentially calls one or more external APIs
  • Serializes and sends a response

Now multiply that chain by 166 every second. Each step adds latency, consumes resources, and opens a window for failure. That is where things get interesting.

Step 1: Requests Hit Your Server

Every incoming request lands on your server. If you are running a single server, which is more common than anyone wants to admit, all those requests are competing for the same CPU, memory, and network bandwidth.

If the server cannot keep up, requests start queuing. Response times climb. Users feel the slowdown almost immediately.

This is usually the first visible symptom: latency spikes. Your API is not down yet, but it is getting sluggish. Dashboards might still look green. Users are already frustrated.

Step 2: Your Application Starts Slowing Down

Your backend code now has to handle dozens of concurrent requests simultaneously. If your application uses blocking operations, has inefficient loops, or does heavy computation for each request, you will see the effects quickly.

In a Node.js application, the event loop starts lagging. In a threaded application (Java, .NET, Go), threads get saturated. Response times that were 50ms at low traffic climb to 500ms, then 2 seconds, then 5 seconds.

The important thing to understand is that even small inefficiencies get amplified under load. A function that takes 10ms too long is invisible at 10 requests per second. At 166 requests per second, it is consuming almost 2 full seconds of CPU time every second. Suddenly it matters a great deal.

Step 3: The Database Becomes the Bottleneck

This is where most systems actually break.

Nearly every request touches the database. At high traffic, too many queries run concurrently. Connections get exhausted. Queries that were fast under normal conditions start slowing down because they are waiting for locks, competing for I/O, or running full table scans because nobody added the right index.

The usual suspects:

  • No indexes on frequently queried columns. At low traffic, the query planner gets away with a sequential scan. At high traffic, it does not.
  • N+1 queries. Your ORM fetches a list of items, then makes one database call per item. At 166 requests per second, you are now making thousands of unnecessary queries.
  • Long-running queries blocking others. A reporting query or unoptimized join holds a lock, and everything behind it waits.

Once the database slows down, everything upstream follows. Your API is only as fast as its slowest dependency, and the database is almost always the first wall you hit.

Step 4: Connection Pool Exhaustion

Most applications use a connection pool to talk to the database. Instead of opening a new connection for every query, you maintain a pool of, say, 20 persistent connections and requests share them.

At 166 requests per second with a pool of 20, the math gets ugly fast. Only 20 queries can execute at any given moment. Everyone else has to wait. If each query takes even 50ms, that pool can handle about 400 queries per second in theory. If your queries take longer, or if each request makes multiple queries, the pool saturates.

Once the pool is full, new requests queue up waiting for a free connection. If that wait exceeds the configured timeout, requests start failing with connection timeout errors. This is the kind of failure that is hard to diagnose if you have never seen it before, because the error message does not say "your database is slow." It says "connection timed out," and you start looking in the wrong places.

At scale, connection pressure becomes a serious concern. For example, 5 application servers with 20 threads each means 100 persistent database connections competing for resources. Tools like PgBouncer exist specifically for this reason, sitting between your application and the database to pool connections more efficiently than any individual application can.

Step 5: Timeouts and Failures Begin

As delays compound through the system, clients start timing out. Your API begins returning 500 errors. And then something worse happens: retries.

Retries are dangerous under load, because they increase traffic to a system that is already overwhelmed. Your 10,000 requests per minute can quickly become 15,000 or 20,000 as clients, load balancers, and upstream services automatically retry failed requests.

This is sometimes called a "retry storm," and it is one of the most common causes of a bad situation becoming a terrible one. The system is struggling to handle the original load, and now it is receiving even more traffic than before the failures started.

Step 6: External Services Make It Worse

If your API depends on third-party services (payment gateways, email providers, identity services, AI APIs), those dependencies add latency and introduce failure modes you do not control.

This is an increasingly real problem. An API reliability analysis of over 215 services between October 2025 and February 2026 found that AI APIs show the highest incident frequency, with providers like OpenAI and Anthropic experiencing recurring short-duration outages. Cloud infrastructure incidents are less frequent but have larger blast radii and longer resolution times, sometimes cascading across hundreds of downstream services. In one notable case, a single AWS DynamoDB incident in October 2025 cascaded into 141 affected services.

When an external dependency slows down, your request is stuck waiting for it. That waiting request is holding a thread, a database connection, memory, and possibly a connection pool slot. If the external service fails entirely, your system might retry, block, or crash, depending on how you have written your error handling.

The key insight is that your overall uptime is the product of your upstream SLAs, not their average. If you depend on three services each at 99.9% uptime, your composite availability is not 99.9%. It is lower.

Step 7: Memory and CPU Spike

As load increases, every request that is in flight consumes memory: the request object, parsed body, database results, response being serialized. CPU usage spikes as the server processes more work concurrently.

If memory limits are hit, garbage collection pauses become longer and more frequent. In extreme cases, the operating system starts killing processes (OOM killer), or the process simply crashes.

This is often the point where someone gets paged. Not because of a graceful error, but because the server stopped responding entirely.

Step 8: Cascading Failure

This is the phase that turns a performance problem into an outage.

The pattern is predictable:

  1. The database slows down
  2. Requests pile up in the application layer
  3. Timeouts increase
  4. Retries amplify traffic
  5. External services start timing out too
  6. Memory and CPU hit their limits
  7. The system overloads and starts failing across the board

Everything fails together. A cascading failure occurs when a problem in one service triggers failures in dependent services, creating a chain reaction throughout your distributed system. Unlike a simple bug or a single slow endpoint, cascading failures are self-reinforcing. Each failure makes the next one more likely.

Modern research on cascading failures in microservices confirms that the most common triggers include timeout misconfigurations between services, retry storms without proper backoff strategies, resource exhaustion on shared infrastructure, and cache failures that suddenly overwhelm backend databases. These issues often compound when services lack proper circuit breakers or bulkhead isolation.

Step 9: Users Feel It Immediately

From the user's perspective, none of the internal details matter. What they experience is:

  • The app is slow
  • Actions fail randomly
  • Nothing feels reliable

This is where trust erodes. And recovering trust is significantly harder than recovering a server. In a competitive market, downtime and high latency are not just technical issues. They erode user trust, damage your reputation, and can directly lead to customer churn.

Why This Happens

Most systems are built for functionality first. They work perfectly at 10 users. They work fine at 100 users. They start cracking at 1,000. And they break at 10,000.

That is not because anyone did anything wrong. It is because scalability is not the same as correctness. Code that produces the right output at low traffic can produce timeouts, errors, and outages at high traffic without a single logic bug being involved.

The problem is almost never one big mistake. It is a collection of small decisions that were fine at low scale: a missing index, a synchronous call that should be async, a connection pool that is slightly too small, no circuit breaker on an external dependency. None of these are obvious problems until load exposes them all at once.

How to Handle This Properly

Now the important part. Here is how you design for this, starting with the changes that give you the most leverage.

1. Distribute Load Across Multiple Servers

A single server is a single point of failure. Put a load balancer in front of multiple application instances so that traffic is distributed across CPUs and memory pools.

In cloud environments, this pairs with auto scaling. Auto scaling automatically adds or removes compute resources according to conditions you define. It is the primary mechanism for handling traffic spikes without requiring someone to be on standby. You pay for capacity when you need it, and release it when you do not.

The right load balancer increases your application's capacity and reliability by sharing the workload evenly across the pool of servers. An ineffective one can mask failures and leave you unaware that a server has gone down.

2. Cache Aggressively

Not every request needs to hit the database. If the same data is requested hundreds of times per minute, serve it from memory.

Caching is a powerful strategy to reduce server load and improve response times by storing data in memory. This reduces repeated database calls, resulting in faster data retrieval and reduced server overload, which is critical during high-demand periods.

Where to cache:

  • Application-level cache (Redis, Memcached): For frequently accessed database results, session data, and computed values.
  • CDN (CloudFront, Cloudflare): For static content, images, and cacheable API responses.
  • HTTP caching headers: So clients and proxies can cache responses without hitting your server at all. GitHub's API, for example, uses Cache-Control and ETag headers to reduce server load.

A well-placed cache can reduce your database load by 80% or more, which often means the difference between surviving a traffic spike and going down.

3. Optimize Database Queries First

Of everything on this list, database optimization usually gives you the highest return on effort. Before you add more servers or more infrastructure, make sure your queries are not doing unnecessary work.

  • Add indexes on columns used in WHERE clauses, JOINs, and ORDER BY.
  • Eliminate N+1 queries by using eager loading or batch fetching.
  • Use pagination for list endpoints. Never return unbounded result sets.
  • Cache frequent query results so the database is not answering the same question thousands of times per minute.

Without scalable patterns like eager loading, pagination, or indexing, things can get slow quite quickly. This is true regardless of what language, framework, or database you use.

4. Right Size Your Connection Pool

The instinct when you hit connection pool exhaustion is to increase the pool size. That helps, up to a point. But too many connections can overload the database itself.

Every open connection consumes memory on the database server. At scale, hundreds of connections competing for the same resources can make things worse, not better. The right approach is to find the balance: enough connections to handle normal load with headroom, but not so many that the database buckles.

For applications that need to scale beyond what a direct connection pool can support, use a connection pooler like PgBouncer (for PostgreSQL) that multiplexes many application connections over a smaller number of database connections.

5. Rate Limit to Protect the System

Rate limiting controls the number of API requests a user or client can make in a given time frame, preventing any single consumer from overwhelming the system. Throttling, a related mechanism, gracefully manages excessive traffic by slowing down or queuing responses when demand exceeds capacity, rather than rejecting requests outright.

This is not just about preventing abuse. Rate limiting protects your system from sudden spikes, whether they come from a misbehaving client, a bot, a retry storm, or a legitimate traffic surge that exceeds your current capacity.

Practical implementation:

  • Set per-user or per-IP limits (e.g., 100 requests/minute per user)
  • Return 429 Too Many Requests with a Retry-After header so clients know when to try again
  • Apply limits at the API gateway layer, before requests reach your application servers

For any scalable platform, from fintech to SaaS, these controls are non-negotiable for sustainable operation.

6. Offload Heavy Work to Queues

Not everything needs to happen inside the request-response cycle. If a request triggers email sending, image processing, report generation, PDF creation, or any other time-consuming operation, move that work to a background queue.

The request returns immediately with a confirmation, and a background worker picks up the job asynchronously. This keeps your API responsive even when it is handling expensive operations.

Common queue systems in 2026:

Tool Best For
BullMQ Node.js applications, Redis-backed
Sidekiq Ruby on Rails applications
Celery Python applications
RabbitMQ General-purpose message broker, multi-language
Apache Kafka High-throughput event streaming, log aggregation

The pattern is straightforward: accept the request, queue the work, process it asynchronously, and notify the client when it is done (via webhook, polling, or WebSocket).

7. Handle Failures Gracefully

When a dependency fails, your system should bend, not break. Design your API to temporarily disable non-critical features or serve cached data to keep core functionality working.

Three patterns matter most here:

Circuit Breakers

A circuit breaker monitors calls to a downstream service. When failures cross a threshold, it "trips" and stops sending requests for a cooldown period, giving the failing service time to recover. Once the service stabilizes, the circuit breaker allows limited traffic to test its health before fully restoring operations. This pattern is crucial for preventing cascading failures.

Netflix popularized this approach. If their recommendation service goes down, it does not prevent users from streaming videos. Instead, Netflix degrades gracefully by displaying generic recommendations.

Bulkheads

The bulkhead pattern isolates resources so that failures in one component do not starve others. By allocating separate thread pools or connection pools to different downstream services, you prevent a slow or failing service from consuming all available resources and bringing down the entire application.

Retries with Backoff

When retrying failed requests, use exponential backoff with jitter. This prevents retry storms by spreading retry attempts over time rather than hammering a recovering service all at once.

// Example: exponential backoff with jitter
const delay = Math.min(baseDelay * 2 ** attempt + Math.random() * 1000, maxDelay);
Enter fullscreen mode Exit fullscreen mode

8. Monitor Everything That Matters

You cannot fix what you cannot see. And you definitely cannot prepare for the next traffic spike if you do not understand what happened during the last one.

The observability landscape has consolidated significantly. OpenTelemetry has become the de facto standard for collecting metrics, logs, traces, and profiles from your applications. It is the second most active project in the Cloud Native Computing Foundation, just behind Kubernetes, and is on track to become a Graduated CNCF project in 2026.

What to monitor:

  • Request rate and latency (overall and per-endpoint, at p50, p95, and p99)
  • Error rates (4xx and 5xx, broken down by type)
  • Database query performance (slow queries, connection pool utilization)
  • CPU, memory, and network saturation
  • Dependency health (latency and error rates for every external service you call)
  • Queue depth and processing lag (for background jobs)

The 2026 monitoring stack:

Layer Tools
Instrumentation OpenTelemetry SDKs (metrics, logs, traces)
Metrics Prometheus, Grafana Mimir
Logs Grafana Loki, Elasticsearch
Traces Grafana Tempo, Jaeger
Visualization Grafana dashboards
Alerting Grafana Alerting, PagerDuty, Opsgenie
APM (commercial) Datadog, New Relic, Dynatrace

Grafana Labs recently announced advancements spanning full-stack observability, database query analysis, and service-centric alerting, including an RCA Workbench that integrates with their AI assistant to help teams turn 30-minute war rooms into 3-minute diagnoses. The direction is clear: unified observability is becoming the default operating model. Nearly three-quarters of executives reported that they had either adopted unified observability or were actively transitioning toward it.

Start with the basics: the four golden signals (latency, traffic, errors, saturation). Then add business metrics per endpoint. The goal is to detect problems before users report them.

The Real Insight

When your API gets 10,000 requests per minute, it is never one thing breaking. It is a dozen small weaknesses all surfacing at the same time.

A missing database index that was fine at low traffic. A connection pool that was adequate for 50 concurrent users but not 500. An external API call with no timeout configured. A retry policy with no backoff. No circuit breaker on a dependency that has been reliable for months.

Each one of these is invisible under normal conditions. Traffic does not create new problems. It reveals existing ones.

My Thought

Handling high traffic is not about guessing, and it is not about over-engineering everything on day one. It is about understanding how systems behave under load, removing bottlenecks one at a time, and designing for failure rather than pretending it will not happen.

Your API does not break because of traffic. It breaks because it was not designed for it. The good news is that every failure pattern described in this article has a well-understood solution. The engineering is not mysterious. It just requires thinking about it before the spike arrives.

Build for the traffic you expect. Design for the traffic you do not.

Top comments (0)