Rate Limiting and Admission Control Solve Different Failure Modes

Rate limiting is one of the first protections we add to an API.

A client gets 100 requests per minute. A tenant gets 1,000. Maybe expensive endpoints get tighter limits. When someone exceeds the allowance, the request gets rejected.

That solves an important problem.

But it doesn't answer another question:

When capacity becomes scarce, which work should actually be allowed to consume it?

That's where admission control becomes useful.

The distinction matters increasingly for LLM workloads, where two requests can have radically different costs even though they both count as one request.

Rate limiting controls arrival rate

At a high level, a rate limiter answers something like:

How much traffic may this caller send during a period of time?

A token bucket might allow 100 requests per minute with some burst capacity.

That can protect against:

• abusive clients
• accidental retry storms
• noisy tenants
• sudden traffic spikes
• exceeding contractual quotas
• excessive API consumption

HTTP even has a status code associated with this case. RFC 6585 defines 429 Too Many Requests as indicating that a user has sent too many requests in a given amount of time.

Rate limiting is extremely useful.

But rate is only one dimension of resource pressure.

A request count isn't a resource model

Imagine an LLM service with a limit of 100 requests per minute.

Now consider two workloads.

Interactive requests:

input:        1,000 tokens
max output:     300 tokens
latency:      user is waiting

Background requests:

input:       30,000 tokens
max output:   4,000 tokens
latency:      nobody is waiting

Both requests count as:

1 request

But they aren't equivalent from the perspective of the underlying system.

They may occupy provider concurrency for different lengths of time. They may consume very different token budgets. They may have completely different latency requirements.

A rate limiter cannot see that if its unit of accounting is simply requests per second or requests per minute.

The traffic can therefore remain completely within its configured rate limit while still exhausting a scarce resource.

Concurrency limits help, but introduce another problem

Suppose we add a concurrency limit:

maximum in-flight requests = 20

Now the application can't overwhelm the downstream service with unlimited parallel work.

That's an improvement.

But imagine 20 background requests acquire all 20 slots.

One second later, an interactive request arrives from a user waiting for an answer.

The rate limiter says:

allowed

The concurrency limiter says:

no capacity

The system is protected, but it hasn't necessarily protected the work that matters most.

We have moved from a rate problem to an allocation problem.

Admission control asks a different question

In this article, I'm using admission control in the narrower capacity-aware sense.

Terminology isn't universal. Some systems use "admission control" broadly enough to include rate limiting itself. The useful distinction here is between controlling how quickly traffic may arrive and deciding whether a particular request should consume currently scarce execution capacity.

Under that definition, admission control asks:

Given the capacity available right now, should this specific piece of work be allowed to start?

That decision can incorporate more information than a conventional rate limiter:

current concurrency
current resource utilization
estimated request cost
request priority
tenant
workload class
reserved capacity
queue depth
deadlines

Instead of merely counting arrivals, we're deciding how scarce capacity should be allocated.

Consider interactive and batch traffic

Suppose a service has 32 execution slots.

Two workload classes share them:

interactive
batch

Without additional controls, batch processing may consume all 32 slots.

A concurrency limiter still prevents the system from exceeding 32 requests, but interactive traffic now waits behind work that nobody is waiting for.

One alternative is to statically divide capacity:

interactive: 28 slots
batch:        4 slots

That protects interactive traffic, but it can waste capacity.

If only 10 interactive requests are running, 18 interactive slots sit idle while batch work waits.

A more flexible admission policy could instead say:

Interactive traffic has protected capacity.

Batch traffic may borrow unused capacity.

When interactive demand increases, new batch admissions stop
until protected capacity is restored.

Now the system can simultaneously pursue two goals:

1. keep expensive infrastructure utilized when capacity is available
2. protect latency-sensitive work when contention appears

A request-per-minute limit alone cannot express that policy.

LLM workloads make the distinction more obvious

This problem exists in ordinary distributed systems, but LLM APIs make it particularly visible.

Request cost varies dramatically.

A request containing a short chat message isn't equivalent to a request asking a model to process a large document with a large maximum output budget.

So an LLM admission controller might track both concurrency and an approximate in-flight token budget.

For example:

request A:
input estimate      = 1,200
max output          =   400
reserved budget     = 1,600

request B:
input estimate      = 24,000
max output          = 3,000
reserved budget     = 27,000

Now, the admission decision can consider resource pressure rather than only the request count.

The reservation doesn't even need to perfectly predict final token usage to be useful.

It can reserve conservatively at admission and reconcile the reservation once actual usage is known.

That turns admission into a resource-allocation problem rather than simply a traffic-counting problem.

Rate limiting can also miss overload that has already started

There is another important difference.

A rate limit generally represents a policy about incoming traffic:

tenant A may send 50 requests/second

But the safe rate of a distributed system isn't necessarily constant.

Maybe a downstream provider has slowed down.

Requests that normally complete in 500 ms now take 8 seconds.

Even if the arrival rate hasn't changed, concurrency begins accumulating:

arrival rate stays constant
        ↓
requests take longer
        ↓
in-flight work grows
        ↓
queues grow
        ↓
latency rises
        ↓
timeouts trigger retries
        ↓
even more work arrives

Google's SRE guidance explicitly warns that simple rate limiting may not account for overall service health and therefore may not stop a failure that has already begun. It recommends rejecting work as systems approach overload and shedding load before resource exhaustion produces cascading failures.

This is a different failure mode from a client merely sending too many requests.

They belong together

The lesson isn't:

Replace rate limiting with admission control.

It's:

Use each mechanism for the failure mode it is good at controlling.

A production path might look roughly like this:

request
   │
   ▼
authentication
   │
   ▼
rate limit / quota
   │
   ▼
admission control
   │
   ▼
downstream service

Conceptually:

if (!rateLimiter.allow(tenant)) {
  return tooManyRequests();
}

const reservation = admissionController.tryAcquire({
  workloadClass: request.workloadClass,
  estimatedCost: estimateCost(request),
});

if (!reservation) {
  return overloaded();
}

try {
  return await callDownstream(request);
} finally {
  reservation.release();
}

The rate limiter protects the service from traffic policy violations.

The admission controller protects scarce execution capacity.

Those aren't identical jobs.

Different failure modes, different questions

I find it useful to frame the difference this way.

Rate limiting asks:

How much traffic may this caller send?

Concurrency limiting asks:

How much work may execute simultaneously?

Admission control asks:

Which work should consume scarce capacity right now?

Load shedding asks:

Which work should we stop accepting because the system is approaching overload?

These mechanisms overlap, and real systems frequently combine them. The boundaries aren't perfectly clean.

But the questions they answer are different enough that treating all of them as "rate limiting" can hide important design decisions.

Why this matters for agents

Agentic systems make the allocation problem even more interesting.

A single user action can create multiple downstream model calls.

Background agents may execute continuously.

Retries can multiply requests.

Tool calls may produce additional model calls.

Long-context operations can consume much more capacity than short interactive requests.

So eventually the question stops being:

How many requests per minute should we allow?

and becomes:

When demand exceeds available capacity,
which work gets to continue?

That is a scheduling and resource-allocation question.

Rate limiting alone doesn't answer it.

The bigger reliability lesson

Overload doesn't always look like a crash.

Sometimes every component remains technically healthy while the wrong work consumes the available capacity.

Queues grow.

Interactive requests wait behind batch jobs.

Retries increase pressure.

Latency explodes.

Eventually, users experience a failure even though the system is still processing requests exactly as designed.

Reliable systems, therefore, need more than a maximum request rate.

They need a policy for scarcity.

That is the problem admission control is trying to solve.

---

I've been exploring this problem while building async-bulkhead-llm and MoFlux, particularly around token-aware admission and protecting interactive traffic while allowing lower-priority workloads to use otherwise idle capacity.

The deeper I get into the problem, the more useful this distinction becomes:

Rate limiting controls how much traffic arrives. Admission control decides which work deserves scarce capacity when it does.