Rate Limits Are a Queueing Problem: Capacity Math for Agent Fleets
Standup ends. Five people go back to their desks and invoke five agents within about ninety seconds, because standup is what schedules agent work now, not anybody's planning.
By 10:30 the first person is staring at a spinner. By noon, the requests that should have finished by lunch are queued behind each other, somebody is complaining in the channel about "the limit hitting early today," and the provider is getting blamed for a traffic jam the team built themselves.
The provider is innocent. The math will convict the wrong defendant every time until you run it once.
Two Numbers and One Law
A queue has two rates. The arrival rate, how many units of work show up per unit of time. The service rate, how many units the server finishes per unit of time. Little's Law connects them to the backlog: the average number of items in the system equals the arrival rate times the average time each item spends in it.
One implication does all the work here: if arrivals exceed service over any sustained window, the queue grows without bound, and your latency stops being a property of your system and starts being a property of your backlog.
Your provider plan is the server. Its published limits, the tokens per minute, the requests per minute, the five-hour window, are the service rate. Your agents are the arrival rate. Nobody in the standup scene above ran these two numbers, which is why the spinner felt like bad luck instead of arithmetic.
The Herd, Not the Limit
Here is the part that took me too long to see. Agent arrivals are not random. They are scheduled by human rituals.
Standup ends, five agents fire. After lunch, the batch jobs wake up. Sprint planning ends, the spikes start. The arrival rate spikes on a calendar you can read, and the plan's service rate is flat all day. The gap between a spiky lambda and a flat mu is the entire complaint.
This is why the upgrade instinct is usually wrong. A bigger plan raises the service rate. The herd is an arrival-rate problem. When the spike is five deep against a plan that serves three, the fix that costs nothing is to stop releasing all five at 9:02.
tip: Before upgrading the plan, compute both sides. If arrivals only exceed service during predictable bursts, you have a scheduling problem, and scheduling is free. If arrivals exceed service all day, you have a capacity problem, and now the upgrade is the right conversation.
Three Levers, One of Them Costs Money
Batch. Hold arrivals in your own queue and release them at a rate you choose. A twenty-line drip feeder turns the 9:02 herd into a steady stream. Same work, same plan, no spinner at 10:30. This is the free lever and it goes first.
Checkpoint. Make interruptions survivable. A run that dies at the wall should resume from where it stopped, not restart from zero, because restarted work re-buys what you already paid for. This one is sneaky: a wall causes retries, retries inflate arrivals, and the limit feeds itself. Checkpointing breaks the loop.
Degrade. Ladder down mid-run. Small model for the routine steps, big model for the decisions, defer the optional work until the window resets. This lever needs a map of which steps are routine, which you should have anyway.
The fourth lever is the upgrade, and it is fine, but it belongs after the free three, not before them.
The Arithmetic in One Worked Example
Symbolic, so it survives the next pricing change.
A plan that sustains three concurrent heavy tasks. A team that fires five at standup plus three more drifting in over the hour. Eight arrivals in the 9-to-10 window against a server that finishes three. Five waiting, residence time climbing all morning, and the "limit" feeling personal by 11.
Apply batching: hold all arrivals, release three at a time, everything else waits in your queue where it costs nothing. Same eight tasks, same plan. Finish time moves from "never, effectively" to "early afternoon," and the only thing built was a drip feeder.
That is the entire argument for running the math instead of reading the dashboard. The queue was never going to negotiate.
The Honest Limitation
Agent arrivals are bursty, not Poisson, and the one-line Little's Law assumes averages. Real fleets have correlated bursts, retry storms, and the pathology where every agent's context grows through the day, so the afternoon tasks cost more tokens than the morning ones did.
The math gives planning bounds, not guarantees. It tells you which regime you are in: stable, borderline, or saturated. It does not tell you your exact 3 PM latency. Use it to pick levers, then watch the logs, because the logs are the measurement and the math is just the map.
Run It This Week
One week of usage logs. Total tokens over working hours gives your arrival rate. Your plan's published sustained rate is the service rate. Compare, and note the windows where the herd gathers. Then pick levers in order: batch, checkpoint, degrade, upgrade.
So here is the question, and it is answerable with a calculator and ten minutes: how many concurrent agents can your current plan sustainably run before somebody's five-hour wall hits at 2 PM, and do you know that number, or does the provider know it better than you do?
If the provider knows your capacity better than you do, you have been reading the wrong dashboard.
Top comments (2)
Framing the standup burst as the arrival rate is the useful shift, because the daily average can sit well under the limit while a 90 second spike blows past it. Have you compared staggering agent starts with a small random delay against a shared queue with a concurrency cap? I'd guess the cap is easier to reason about, since the five hour window then becomes a budget the team can plan against.
The hidden multiplier on the retry storm is that provider rate limits track rolling tokens rather than raw requests. When five agents get 429'd at the same second with twenty thousand tokens in flight, standard exponential backoff with full jitter still tends to synchronize. They all wake up right as the sixty-second token bucket refills, dump one hundred thousand tokens into the API at once, and knock each other back into rate limiting.
In my own multi-agent setups, what finally flattened the curve was decoupling tool execution time from LLM turns through a shared local leaky bucket. When one agent spends forty seconds waiting on a test suite or git clone, its slot on the token regulator opens up for another worker. Tracking token grants at the transport layer before the HTTP call ever fires keeps local concurrency high without ever triggering the provider backoff loop.