Your LLM Capacity Plan Is Lying to You: Inside vLLM Overload Traces

During an overload experiment on vLLM, the engine's own prompt token counter missed a third to half of every prompt token it computed. Requests that were later aborted had already burned 79 to 85 percent of their prefill work before anyone cancelled them.

That is not a bug report. That is your capacity plan sitting on a meter that undercounts by half during exactly the incident you plan for.

We covered the client side of retries in an earlier note, where idempotency and reconciliation decide whether a retry is a second attempt or a second charge. This is the other end of the wire: what the engine does when every client retries at once.

THE EXPERIMENT THAT DID NOT GO AS PLANNED

Two researchers ran overload experiments on vLLM with three open models and recorded scheduler traces of one of them. A scheduler trace is the record that ties each client attempt to the execution it created inside the engine. That instrument matters more than the headline. Without it you see requests arrive and finish. With it you see the work a dead client left behind.

The assumption under test was the fix most teams ship first because it looks free: propagate the cancellation and the wasted compute goes away.

It did not. With retries on, cancellation alone failed to restore service in any run. Only 12 to 38 percent of the calls that arrived after the overload were delivered.

Cancellation is free, and you get what you pay for.

WHY CANCELLATION DOES NOT UNBURN TOKENS

The traces show why. Retries that vLLM ran typically started prefill less than half a second before they timed out. Prefill is the prompt processing that happens before the first token comes out, and for a long agent context it is most of the bill. It was nearly finished when the deadline fired.

So the client gave up, the engine kept computing, and the cancel signal arrived after the money was spent.

Then the accounting failure. The engine's prompt token counter missed a third to half of all the prompt tokens it computed during the incident. Your autoscaler, your budget alerts, your per tenant cost report and your capacity plan all read from that counter. They all read low at the worst possible moment, and they all read low in the same direction.

THE OVERLOAD TRIAGE RULE

Three remedies get sold for retry storms, and they fail in different ways. Pick by which failure you can afford.

1. Cancellation stops new waste. It does not recover work already spent.
2. A retry budget stops amplification. It does so by refusing service to some callers.
3. Request coalescing stops duplicate work. It delivered every call from the overload onward, some of them late, unless cancellation had already aborted the running execution.

The order matters, and that is the part most teams get backwards. Coalescing wins only while the original execution is still alive, so cancellation running ahead of it is what turns a late answer into no answer. Coalesce first, budget second, cancel last.

That order is the whole interceptor. The shape below is illustrative, not the paper's code, but every line maps to one of the three remedies:

# gateway interceptor, illustrative shape only inflight = {}

def intercept(req):
    key = (req.prompt_hash, req.model, req.deadline_bucket)
    if key in inflight:
        return inflight[key].subscribe()   # coalesce: one execution, many waiters
    if not retry_budget.take(req.client):
        return shed(req)                   # budget: refuse on purpose, not by luck
    ex = engine.submit(req)                # prefill starts burning tokens here
    inflight[key] = ex
    return ex.result(timeout=req.timeout)  # no cancel call, and that is deliberate

Leaving out the cancel call is the design decision, not an oversight. Kill an execution that other callers are attached to, and you have converted a late answer into no answer for all of them.

Pick your remedy by which failure you can afford: waste, refusal, or latency.

WHY AGENTS MAKE THIS THE COMMON CASE

DeepSeek disclosed that its Harness agent app was consuming roughly 92 trillion tokens per day at 1.54 million daily users, which works out to about 60 million tokens per user per day. Chat traffic never produced a number like that. An agent with a retry loop multiplies request volume and prompt length at the same time, because every retry carries the same long context into prefill.

Load shapes are also diverging. None of the top ten open source models on OpenRouter share an architecture, and average prompt lengths differ by region, 97k tokens against 70k. When the shape of the load keeps moving, the overload point moves with it, and the timeout you tuned last quarter is now a retry generator.

The overload path is not the edge case you deprovision capacity for. When your users are agents holding timeouts, it is the product path.

WHAT TO DO TODAY

1. Stop trusting the engine's prompt token counter during an incident. Log scheduler level executions, not just completed requests.
2. Measure prefill completed before timeout. If retries start prefill under a second from their deadline, your timeout is your retry policy.
3. Build coalescing at the gateway, keyed on the request, so duplicate calls share one execution instead of buying one each.
4. Cap client retries with a budget and make the budget visible to the server, so shedding is a decision rather than a coincidence.
5. Run an overload drill on purpose and record delivered call percentage, not average latency. Average latency hides undelivered calls completely.

That last one is the measurement that would have caught this. Every team watching this incident had a latency graph. The graph that told the truth was the one counting calls that never got an answer.

THE UNCOMFORTABLE QUESTION

Your dashboards showed green through the last incident because the counter feeding them undercounts by half under load. How much of your capacity plan is built on a meter you have never verified during an overload?