DEV Community: jaycodes

Rate Limiting and Admission Control Solve Different Failure Modes

jaycodes — Mon, 31 Aug 2026 20:52:43 +0000

Rate limiting is one of the first protections we add to an API.

A client gets 100 requests per minute. A tenant gets 1,000. Maybe expensive endpoints get tighter limits. When someone exceeds the allowance, the request gets rejected.

That solves an important problem.

But it doesn't answer another question:

When capacity becomes scarce, which work should actually be allowed to consume it?

That's where admission control becomes useful.

The distinction matters increasingly for LLM workloads, where two requests can have radically different costs even though they both count as one request.

Rate limiting controls arrival rate

At a high level, a rate limiter answers something like:

How much traffic may this caller send during a period of time?

A token bucket might allow 100 requests per minute with some burst capacity.

That can protect against:

abusive clients
accidental retry storms
noisy tenants
sudden traffic spikes
exceeding contractual quotas
excessive API consumption

HTTP even has a status code associated with this case. RFC 6585 defines 429 Too Many Requests as indicating that a user has sent too many requests in a given amount of time.

Rate limiting is extremely useful.

But rate is only one dimension of resource pressure.

A request count isn't a resource model

Imagine an LLM service with a limit of 100 requests per minute.

Now consider two workloads.

Interactive requests:

input:        1,000 tokens
max output:     300 tokens
latency:      user is waiting

Background requests:

input:       30,000 tokens
max output:   4,000 tokens
latency:      nobody is waiting

Both requests count as:

1 request

But they aren't equivalent from the perspective of the underlying system.

They may occupy provider concurrency for different lengths of time. They may consume very different token budgets. They may have completely different latency requirements.

A rate limiter cannot see that if its unit of accounting is simply requests per second or requests per minute.

The traffic can therefore remain completely within its configured rate limit while still exhausting a scarce resource.

Concurrency limits help, but introduce another problem

Suppose we add a concurrency limit:

maximum in-flight requests = 20

Now the application can't overwhelm the downstream service with unlimited parallel work.

That's an improvement.

But imagine 20 background requests acquire all 20 slots.

One second later, an interactive request arrives from a user waiting for an answer.

The rate limiter says:

allowed

The concurrency limiter says:

no capacity

The system is protected, but it hasn't necessarily protected the work that matters most.

We have moved from a rate problem to an allocation problem.

Admission control asks a different question

In this article, I'm using admission control in the narrower capacity-aware sense.

Terminology isn't universal. Some systems use "admission control" broadly enough to include rate limiting itself. The useful distinction here is between controlling how quickly traffic may arrive and deciding whether a particular request should consume currently scarce execution capacity.

Under that definition, admission control asks:

Given the capacity available right now, should this specific piece of work be allowed to start?

That decision can incorporate more information than a conventional rate limiter:

current concurrency
current resource utilization
estimated request cost
request priority
tenant
workload class
reserved capacity
queue depth
deadlines

Instead of merely counting arrivals, we're deciding how scarce capacity should be allocated.

Consider interactive and batch traffic

Suppose a service has 32 execution slots.

Two workload classes share them:

interactive
batch

Without additional controls, batch processing may consume all 32 slots.

A concurrency limiter still prevents the system from exceeding 32 requests, but interactive traffic now waits behind work that nobody is waiting for.

One alternative is to statically divide capacity:

interactive: 28 slots
batch:        4 slots

That protects interactive traffic, but it can waste capacity.

If only 10 interactive requests are running, 18 interactive slots sit idle while batch work waits.

A more flexible admission policy could instead say:

Interactive traffic has protected capacity.

Batch traffic may borrow unused capacity.

When interactive demand increases, new batch admissions stop
until protected capacity is restored.

Now the system can simultaneously pursue two goals:

keep expensive infrastructure utilized when capacity is available
protect latency-sensitive work when contention appears

A request-per-minute limit alone cannot express that policy.

LLM workloads make the distinction more obvious

This problem exists in ordinary distributed systems, but LLM APIs make it particularly visible.

Request cost varies dramatically.

A request containing a short chat message isn't equivalent to a request asking a model to process a large document with a large maximum output budget.

So an LLM admission controller might track both concurrency and an approximate in-flight token budget.

For example:

request A:
input estimate      = 1,200
max output          =   400
reserved budget     = 1,600

request B:
input estimate      = 24,000
max output          = 3,000
reserved budget     = 27,000

Now, the admission decision can consider resource pressure rather than only the request count.

The reservation doesn't even need to perfectly predict final token usage to be useful.

It can reserve conservatively at admission and reconcile the reservation once actual usage is known.

That turns admission into a resource-allocation problem rather than simply a traffic-counting problem.

Rate limiting can also miss overload that has already started

There is another important difference.

A rate limit generally represents a policy about incoming traffic:

tenant A may send 50 requests/second

But the safe rate of a distributed system isn't necessarily constant.

Maybe a downstream provider has slowed down.

Requests that normally complete in 500 ms now take 8 seconds.

Even if the arrival rate hasn't changed, concurrency begins accumulating:

arrival rate stays constant
        ↓
requests take longer
        ↓
in-flight work grows
        ↓
queues grow
        ↓
latency rises
        ↓
timeouts trigger retries
        ↓
even more work arrives

Google's SRE guidance explicitly warns that simple rate limiting may not account for overall service health and therefore may not stop a failure that has already begun. It recommends rejecting work as systems approach overload and shedding load before resource exhaustion produces cascading failures.

This is a different failure mode from a client merely sending too many requests.

They belong together

The lesson isn't:

Replace rate limiting with admission control.

It's:

Use each mechanism for the failure mode it is good at controlling.

A production path might look roughly like this:

request
   │
   ▼
authentication
   │
   ▼
rate limit / quota
   │
   ▼
admission control
   │
   ▼
downstream service

Conceptually:

if (!rateLimiter.allow(tenant)) {
  return tooManyRequests();
}

const reservation = admissionController.tryAcquire({
  workloadClass: request.workloadClass,
  estimatedCost: estimateCost(request),
});

if (!reservation) {
  return overloaded();
}

try {
  return await callDownstream(request);
} finally {
  reservation.release();
}

The rate limiter protects the service from traffic policy violations.

The admission controller protects scarce execution capacity.

Those aren't identical jobs.

Different failure modes, different questions

I find it useful to frame the difference this way.

Rate limiting asks:

How much traffic may this caller send?

Concurrency limiting asks:

How much work may execute simultaneously?

Admission control asks:

Which work should consume scarce capacity right now?

Load shedding asks:

Which work should we stop accepting because the system is approaching overload?

These mechanisms overlap, and real systems frequently combine them. The boundaries aren't perfectly clean.

But the questions they answer are different enough that treating all of them as "rate limiting" can hide important design decisions.

Why this matters for agents

Agentic systems make the allocation problem even more interesting.

A single user action can create multiple downstream model calls.

Background agents may execute continuously.

Retries can multiply requests.

Tool calls may produce additional model calls.

Long-context operations can consume much more capacity than short interactive requests.

So eventually the question stops being:

How many requests per minute should we allow?

and becomes:

When demand exceeds available capacity,
which work gets to continue?

That is a scheduling and resource-allocation question.

Rate limiting alone doesn't answer it.

The bigger reliability lesson

Overload doesn't always look like a crash.

Sometimes every component remains technically healthy while the wrong work consumes the available capacity.

Queues grow.

Interactive requests wait behind batch jobs.

Retries increase pressure.

Latency explodes.

Eventually, users experience a failure even though the system is still processing requests exactly as designed.

Reliable systems, therefore, need more than a maximum request rate.

They need a policy for scarcity.

That is the problem admission control is trying to solve.

I've been exploring this problem while building async-bulkhead-llm and MoFlux, particularly around token-aware admission and protecting interactive traffic while allowing lower-priority workloads to use otherwise idle capacity.

The deeper I get into the problem, the more useful this distinction becomes:

Rate limiting controls how much traffic arrives. Admission control decides which work deserves scarce capacity when it does.

When Better Latency Metrics Hide Worse Reliability

jaycodes — Sat, 29 Aug 2026 17:36:23 +0000

A system completes 93% of interactive requests with a p95 latency of 10 seconds.

Another completes 97% with a p95 latency of 12 seconds.

Which one performed better?

The obvious answer seems to be the first system. Twelve seconds is worse than ten.

But there is a problem hidden in that comparison:

The latency percentile is calculated only over requests that succeeded.

The second system completed requests that the first system did not.

Once systems admit, reject, retry, or shed different subsets of traffic, successful-request latency stops being an apples-to-apples comparison by itself.

This matters particularly under overload.

And it matters a lot for LLM infrastructure.

The denominator changed

Latency percentiles are extremely useful.

Google's SRE guidance recommends percentiles precisely because averages can obscure the long tail of request latency. A system with a reasonable average can still have a terrible experience for its slowest requests.

But every percentile has a population.

Consider 100 requests sent to two systems.

System A completes 90:

Requests offered:     100
Requests completed:    90
Requests failed:       10
Successful p95:       4.0s

System B completes 99:

Requests offered:     100
Requests completed:    99
Requests failed:        1
Successful p95:       5.0s

Looking only at latency produces a simple conclusion:

System A is faster.

Looking only at successful completion produces another:

System B is more reliable.

Neither metric is wrong.

They measure different properties.

The mistake is turning either one into a complete description of the system.

Overload makes this especially dangerous

Under ordinary load, two systems may successfully process almost exactly the same requests.

Their latency distributions are therefore reasonably comparable.

Under overload, admission policy starts changing the population.

A system can:

reject requests immediately;
queue them;
retry them;
prioritize one workload class;
shed expensive work;
reserve capacity for selected traffic;
allow requests through until an upstream dependency rejects them.

Those decisions determine which requests eventually appear in the successful-request latency histogram.

That means the latency measurement becomes conditional:

Latency, given that the request succeeded.

That is not the same metric as:

What happened to a request offered to the system?

The distinction becomes important as failure rates diverge.

A real example

I ran into this while evaluating admission-control strategies for mixed LLM workloads.

The benchmark compared four approaches under the same deterministic workload:

No application-level control
A static cap
Redis-coordinated admission
Local admission backed by distributed capacity coordination

The workload intentionally exceeded the provider's 32-request capacity envelope and mixed latency-sensitive interactive requests with background batch work.

At the zero-coordinator-latency rung, measured across eight matched seeds, two results initially seemed contradictory.

Redis produced better successful-request latency:

Metric	Redis	Local coordinated admission
Interactive p50	4.22s	5.25s
Interactive p95	10.30s	11.54s
TTFT p50	0.79s	1.04s
TTFT p95	2.92s	5.46s

If I stopped there, Redis would clearly look better.

But the completion results looked different:

Metric	Redis	Local coordinated admission
Interactive success	92.7%	97.4%
Interactive goodput	5.50 req/s	5.73 req/s
Retry amplification	1.478×	1.339×
Local rejects	306.5	264

The local strategy completed substantially more interactive work while performing fewer retries and shedding fewer attempts.

Redis completed fewer interactive logical requests, but the requests that did complete reached their first token faster.

Both observations are true.

So which system actually won?

That question is too broad.

Redis won one dimension of the experiment:

responsiveness among successfully completed requests.

The local admission strategy won another:

successful completion of protected interactive work.

It is tempting to collapse those into a single ranking.

Doing so throws away useful information.

If your primary objective is minimizing latency for requests that survive overload, the Redis result may be preferable.

If your primary objective is preserving interactive completion when capacity becomes scarce, the other policy may be preferable.

And if batch throughput matters most, the answer changes again: Redis also completed considerably more batch work in this particular benchmark.

The benchmark therefore does not establish a universal winner.

It exposes a tradeoff.

That is exactly what a useful benchmark should do.

The survivor effect

There is a deeper measurement issue here.

Imagine a workload with ten requests.

Their hypothetical service times, if all could run successfully, might be:

1s
1s
1s
2s
2s
2s
3s
5s
8s
15s

Now imagine System A admits all ten.

System B admits the first seven and rejects the last three.

System B's successful-request latency distribution looks excellent.

But it did not make the 5-, 8-, and 15-second requests faster.

It removed them from the population being measured.

This is an intentionally simplified example, but it demonstrates the problem.

A successful-request latency percentile tells you about the survivors.

It does not tell you what happened to the requests missing from the sample.

This resembles a familiar statistical problem: conditioning on an outcome can change the composition of the population you are comparing.

In infrastructure benchmarks, admission and load shedding can create exactly that situation.

Be careful with the stronger interpretation

There is a tempting explanation for the benchmark above:

The local strategy had worse p95 latency because it successfully carried harder requests that Redis rejected.

That is plausible.

It is not fully established by the aggregate numbers.

The benchmark did include requests with substantially different sizes—interactive request sizes varied by roughly 25× to 32×—and token pressure was a real admission constraint. The two systems also made different rejection decisions.

But proving the stronger statement would require matched request-level counterfactual evidence.

For each request rejected by one policy and completed by another, we would want to know what its latency would have been had the rejecting system admitted it.

We cannot observe both outcomes in the same run.

So the defensible conclusion is narrower:

When completion rates differ, successful-request latency percentiles describe different populations and therefore should not be interpreted independently of completion.

That claim does not require speculation.

The measurements themselves establish it.

This is why availability and latency belong together

SRE practice already points in this direction.

Google treats both availability and latency as distinct service-level indicators. Availability measures the proportion of requests that succeed; latency measures how quickly requests are served.

Neither replaces the other.

Suppose a release changes your dashboard from:

Success: 99.5%
p95:     800ms

to:

Success: 94.0%
p95:     500ms

Celebrating the 300ms latency improvement would be premature.

Maybe the service became dramatically more efficient.

Or maybe it stopped serving a meaningful portion of its workload.

You have to investigate both.

The same principle applies when comparing architectures.

Retries distort the picture further

Retries introduce another layer.

Suppose a logical user request is rejected twice and succeeds on its third attempt.

At the attempt level, you may observe:

Attempt 1: rejected
Attempt 2: rejected
Attempt 3: TTFT 900ms

If the dashboard reports only the final successful provider attempt, the user appears to have received a response with a 900ms time to first token.

They didn't.

They experienced rejection, backoff, another rejection, more backoff, and then 900ms to first token.

A benchmark needs to distinguish at least three concepts:

provider-attempt latency;
successful logical-request latency;
end-to-end user-observed latency.

Otherwise retry-heavy systems can appear healthier than they are.

This matters operationally as well as statistically. Google warns that retries can amplify overload and contribute to cascading failure, which is why exponential backoff and jitter are standard overload protections.

In the LLM benchmark, this was one reason I tracked retry amplification alongside latency.

Redis produced lower successful-request TTFT, but required 1.478 attempts per logical interactive request at the median.

The local strategy recorded 1.339.

Neither number means much without the other metrics around it.

Rejections need context too

Even rejection count can mislead.

One admission controller may reject an attempt locally before expensive work begins.

Another may admit that attempt, consume provider capacity, and eventually receive a provider-side 429.

A third may queue the request until the caller times out.

All three can result in an unsuccessful request.

Operationally, however, they are very different outcomes.

Early rejection is often deliberate overload protection.

Google's SRE guidance recommends graceful load shedding when a service exceeds sustainable capacity rather than allowing overload to turn into extreme latency or cascading failure.

So "fewer rejects" is not automatically better either.

What ultimately matters is what happens to the logical workload:

Was the important request eventually completed?

How long did the caller wait?

How many attempts did it require?

How much capacity was consumed along the way?

What other work was displaced?

Use a scorecard, not a hero metric

For overload experiments, I now prefer to think in terms of a scorecard.

For latency-sensitive traffic, I want at least:

Offered logical requests
        |
        +-- completed
        |
        +-- exhausted / failed
        |
        +-- attempts
              |
              +-- admitted
              +-- locally rejected
              +-- upstream rejected

And then I measure:

Completion

logical-request success rate;
successful goodput;
exhaustion rate.

User experience

end-to-end p50;
end-to-end p95;
TTFT p50;
TTFT p95.

Admission behavior

local rejects;
queue wait;
token-budget rejects;
concurrency-bound rejects.

Amplification

attempts per logical request;
retries per request;
upstream 429s.

Capacity

peak occupancy;
in-flight concurrency;
in-flight token exposure.

And for mixed workloads, every important result should be segmented by workload class.

An aggregate 95% success rate is not reassuring if interactive success is 80% and low-priority batch success is nearly 100%.

Always print the denominator beside the percentile

A simple reporting habit prevents a surprising amount of confusion.

Instead of:

TTFT p95: 2.9s

report:

Interactive success: 92.7%
Successful-request TTFT p95: 2.9s

Even better:

Offered:                   270 logical requests
Completed:                 250
Interactive success:       92.7%
Successful-request p95:    10.3s
Successful-request TTFT:    2.9s
Attempts/logical request:   1.478

Now the reader can see the population behind the latency statistic.

The word successful-request is worth keeping.

It is a reminder that failures are absent from the percentile.

Matched workloads matter

The measurement problem becomes worse when benchmark workloads themselves differ.

If one system receives smaller requests, fewer long prompts, or a different arrival sequence, latency comparisons tell you very little.

This is why controlled overload experiments should use matched workloads.

In the benchmark described above, each strategy was run against matched seeds, and results were reported across eight seeds rather than selecting one representative-looking execution. The broader coordinator-distance experiment ran eight matched seeds at four distance rungs, producing 32 recorded proof combinations.

Repeating runs matters because overload is timing-sensitive.

Queue ordering changes.

Retries shift arrival patterns.

Capacity becomes available at slightly different moments.

A single run can produce a compelling graph that turns out not to represent the system reliably.

Don't optimize the dashboard

There is a dangerous feedback loop in performance engineering:

Choose an important metric.
Optimize the system.
Watch the metric improve.
Assume the system improved.

Usually that works.

Sometimes the optimization changes what gets counted.

Admission control is a particularly clear example.

A stricter policy can improve the latency distribution simply by excluding more work.

A more permissive policy can make latency look worse while completing requests that would otherwise disappear from the successful population.

Neither outcome is inherently desirable.

The business requirement determines which tradeoff is acceptable.

If a user clicks "Generate" in an interactive application, I might willingly accept an additional second of tail latency to materially improve the probability that the request completes.

For a latency-sensitive trading system, that trade might be unacceptable.

For an asynchronous document-processing pipeline, throughput might dominate both.

Metrics don't decide this.

They tell us what trade we made.

A better question

When comparing overloaded systems, don't ask only:

Which one has the lower p95?

Ask:

Lower p95 over which requests?

Then ask:

What happened to the rest?

That small change in framing makes overload benchmarks much harder to misread.

Latency percentiles remain indispensable. Tail latency matters, and averages absolutely can hide serious performance problems.

But latency is only one part of the outcome.

Once two systems make different admission, retry, or load-shedding decisions, their successful-request distributions may no longer contain equivalent populations.

At that point, a prettier latency number can coexist with a worse reliability result.

The percentile isn't lying.

We're just asking it to tell us something it doesn't measure.

When batch traffic competes with interactive LLM requests

jaycodes — Thu, 27 Aug 2026 22:12:00 +0000

Not all LLM traffic is equally urgent.

A user waiting for a response cares about the next few seconds, whereas a batch task summarizing thousands of documents usually doesn't.

If both types of work share the same capacity, your infrastructure may treat them the same.

Here's how that can become a problem.

Imagine your application handles two kinds of workloads: interactive (user waiting for a response) and batch (background summarization, classification, extraction, or enrichment jobs). Both call the same LLM provider.

When batch processing ramps up, suddenly your interactive requests have to compete with batch traffic that nobody is waiting for.

Try limiting concurrency

One obvious solution is to limit the LLM concurrency. However, a concurrency limit knows how many requests are running. It has no idea which requests matter most. So if 20 batch requests acquire all 20 slots, latency-sensitive interactive traffic has nowhere to go.

Queueing doesn't necessarily solve it

What about a queue for excess requests? In that case, the interactive request waits behind the batch traffic. Technically, the system is behaving correctly, but the user experience is terrible.

This is one way overload can be deceptive. No crashes, but requests spend more time waiting for capacity. TTFT rises, tail latency worsens, callers time out or retry, retries create more work.

Resource contention can turn into an overload feedback loop.

LLM workloads worsen the problem

But it's not just concurrency. Let's consider two requests:

Interactive:
1,000 input tokens
500 maximum output tokens

Batch:
30,000 input tokens
4,000 maximum output tokens

Each consumes one concurrency slot, but they aren't anywhere near the same amount of work.

This is why it's useful to consider both concurrent requests and in-flight token commitments in making admission decisions. Before starting a request, estimate its potential token footprint and reserve that amount. It doesn't have to be exact; a reasonable approximation can still prevent a small number of very large requests from consuming a disproportionate amount of capacity.

Priority alone isn't enough

You could also prioritize interactive requests, which helps when multiple requests are waiting. However, priority cannot recover capacity already tied up in running requests. If batch work occupies every slot, a high-priority request would still be stuck waiting.

It's important not only to ask which request should run next, but also whether a request should be allowed to consume capacity now. That's why admission control matters.

One approach is to reserve part of the available capacity for different workload classes. If you have 20 available concurrency slots, try something like the following:

Interactive protected capacity: 8
Batch protected capacity: 2
Shared capacity: 10

These numbers can vary depending on workload requirements. The point is that protected capacity can prevent either workload from starving the other. Interactive does not always win.

The goal is to protect latency-sensitive work, ensure background processes run, and share the remaining capacity efficiently. This is workload isolation.

Fail fast can be better than fail slowly

When capacity isn't available, there are often two choices: wait or reject. Waiting sounds friendlier, but if requests accumulate faster than the system can complete them, queueing converts overload into latency.

Sometimes the more reliable behavior is to reject work immediately.

The broader lesson is that overload management encompasses more than limiting how much work enters the system. It's also about deciding which work is allowed to proceed when demand exceeds system capacity.

Sources and further reading

async-bulkhead-llm README — fail-fast admission control, concurrency limits, token budgeting, reconciliation, telemetry, and overload protection.
async-bulkhead-llm: Admission classes — protected floors, ceilings, shared capacity, borrowing, and class telemetry.
llm-d: Batch Serving — discussion of co-locating offline/batch inference with real-time workloads and protecting interactive traffic from batch contention.

Concurrency Limits Aren’t Enough for LLM APIs

jaycodes — Tue, 25 Aug 2026 03:27:59 +0000

A concurrency limit treats every request as roughly equal.

For LLM workloads, that assumption breaks down.

A request with a 500-token budget and one with a 30,000-token budget might both occupy a single concurrency slot, but they can represent very different amounts of provider capacity.

That’s the problem I built async-bulkhead-llm to address.

Instead of limiting only concurrent requests, it can bound two things at once:

concurrent requests
+
estimated in-flight tokens

A request is admitted only when both budgets have enough capacity.

Conceptually:

const bulkhead = new AsyncBulkhead({
  maxConcurrent: 8,
  maxInFlightTokens: 40_000,
});

That gives you a bulkhead around an LLM provider that accounts for workload size rather than simply request count.

This becomes especially useful when interactive and large batch requests share the same upstream model. A conventional concurrency limiter can still allow a few very large requests to consume most of the useful capacity.

Token-aware admission gives you another control surface.

async-bulkhead-llm is an open-source TypeScript/npm library I’ve been developing specifically around this problem.

The broader lesson has been simple: for LLM systems, concurrency is only one dimension of load.

If you're building production LLM infrastructure, I'd be interested in how you're handling admission control and overload protection.