Continuous Batching: Why Your GPU Waits for the Slowest Request

When a language model serves more than one user, the requests never finish together. One person asks a yes-or-no question and gets three tokens back. The next asks for a function and gets two thousand. If the server runs them as one fixed batch, the GPU keeps working on the short request after it is done, producing nothing, until the long one finishes.

vLLM and TGI avoid this with continuous batching, and both have it on by default. Below I work out what those empty steps cost with a small example you can check by hand, and then show the vLLM metrics that tell you whether continuous batching is doing its job on your server.

Eight batch slots, sixteen decode steps. A static batch ends at 52% busy. Continuous batching gives each free slot to a waiting request and stays at 100%.

What is continuous batching?

A GPU serving a model works in steps. Each step moves every request it holds forward by one token, and all of those requests share one read of the model's weights. That shared read is what makes serving many users at once affordable, and the group of requests sharing it is the batch.

Static batching keeps the group fixed. You collect some requests and run them until every one has finished, and only then start the next group. A request that finishes early cannot leave, so the GPU keeps stepping it, producing nothing, until the slowest request in the batch is done.

Continuous batching rebuilds the batch at every step. After each token, the server drops the requests that just finished, frees their memory, and admits waiting requests in their place. The idea comes from the Orca paper (OSDI 2022). As a loop, it looks roughly like this:

# A sketch of the scheduling loop, not vLLM's actual code.
while True:
    batch = [r for r in batch if not r.finished]       # drop finished requests, free their memory
    while queue and len(batch) < max_num_seqs and cache_has_room():
        batch.append(queue.pop(0))                     # admit waiting requests into free slots
    run_one_decode_step(batch)                         # one new token for every request in the batch

It combines two techniques. Ragged batching packs requests of different lengths into one sequence and uses the attention mask to keep them apart, so nothing needs padding. Dynamic scheduling is the drop-and-admit loop above. If you want the full mechanism, HuggingFace's continuous batching from first principles derives both from the tensor shapes.

Padded, the three prompts take 21 slots and 6 are padding. Packed, they fit in 15, and the mask lets each token see only the earlier tokens of its own request.

How much does a static batch waste?

Take the toy from the animation. Eight slots over sixteen steps give you 128 slot-steps of GPU time, and that is the capacity you pay for in that window. The eight answers come out at these lengths:

3   5   8   11   4   14   6   16   tokens

They add up to 67. So the GPU does 67 slot-steps of useful work and spends the other 61 stepping requests that have already finished, because the batch cannot move on until the 16-token answer is done. Only 52% of the window does work. With continuous batching a waiting request takes each free slot on the next step, so as long as the queue is not empty, all 128 slot-steps do work.

By utilization I mean batch slots doing useful work, which is different from what your GPU monitor shows. The monitor reports the card as busy through those idle steps, because the card reads the full weights at every step whether one request is live or eight. The waste shows up in tokens per second instead.

How much a static batch wastes depends on how much your answer lengths vary. Keep eight requests and sixteen steps, but make the lengths 1, 1, 2, 2, 3, 3, 4 and 16. They add up to 32, so the static batch does useful work in only a quarter of its slot-steps, and seven short requests sit through twelve to fifteen empty steps each while the long one finishes. You can check both numbers yourself:

def static_utilization(lengths):
    """Share of a static batch's slot-steps that do useful work."""
    steps = max(lengths)                  # the batch runs until its longest request ends
    return sum(lengths) / (len(lengths) * steps)

print(static_utilization([3, 5, 8, 11, 4, 14, 6, 16]))   # 0.5234375  (67 of 128)
print(static_utilization([1, 1, 2, 2, 3, 3, 4, 16]))     # 0.25       (32 of 128)

Try your own traffic. Pull the output token counts from a few minutes of logs, group them into batches the size you serve, and see how far below 1.0 the number lands.

Where does the 23x come from?

You will see continuous batching quoted with big multipliers like 23x. Most of that size comes from the baseline it was compared with.

Anyscale ran OPT-13B on a single A100. Continuous batching alone gave up to 8x the throughput of naive static batching, and up to 23x once vLLM's PagedAttention was added, the memory system that fits more requests on the card. vLLM's launch post reports, with one completion per request, 14x to 24x the throughput of HuggingFace Transformers and only 2.2x to 2.5x that of Text Generation Inference (TGI). The 3.3x to 3.5x over TGI in the post's headline comes from a separate run with three completions per request.

The four claims drawn to scale. Only the green bar compares two engines that both already do continuous batching.

TGI already does continuous batching, so whatever vLLM gains over TGI comes from PagedAttention rather than from the scheduler. The multiples above 20 appear only when the baseline does no continuous batching at all. So continuous batching on its own gives a big gain over a naive loop and very little over an engine that already does it, and most of the rest of the 23x comes from better memory management.

These numbers come from other people's models and hardware, so I would not use them to predict your own gain. What does apply to your server is the pattern from the toy example: the more your output lengths vary, the more continuous batching saves you.

Does it help latency too?

Yes, which is unusual, because batching normally trades latency for throughput. Anyscale's post says so in its title. Under static batching, a new request cannot start until the whole current batch has finished, so it may wait behind two thousand tokens of someone else's answer. Under continuous batching it starts as soon as any slot frees up, usually a few steps later.

What should you check on your own server?

You are probably running continuous batching already, so the job is to check that it works. vLLM exposes the numbers you need on the same port as the API:

curl -s localhost:8000/metrics \
  | grep -E '^vllm:(num_requests_running|num_requests_waiting|num_preemptions_total)'

1. Make sure you are using it at all. vLLM and TGI do it by default. A home-grown generation loop, or code that collects requests and waits for the whole group to finish, does not. If that is you, switching to an engine that does it will raise throughput more than any kernel or quantization change.
2. Compare the running batch with the cap. --max-num-seqs is only the most requests vLLM may run at once. The batch you get under load is vllm:num_requests_running. Under real traffic it should sit near the cap. If it sits far below the cap while requests wait, a higher cap will not help, because something else limits the batch, almost always the KV cache memory that decides how many requests fit on the card.
3. Watch preemptions. When the cache fills up, the scheduler pulls a running request out of the batch to free its memory and puts it back in the queue to finish later. Nothing crashes, but vllm:num_preemptions_total goes up. If it climbs at a load you used to handle, you are short of memory. If it stays flat while latency still gets worse, the time is going to queueing and to slower steps.
4. Watch throughput when output lengths spread out. If tokens per second drop exactly when answers get uneven, something in your stack still keeps finished requests in the batch.
5. Change one setting at a time. --max-num-seqs interacts with KV cache capacity and with the per-step token budget, and if you move two at once you cannot tell which one did what.

My recommendation

Run the curl command above during your busiest hour, before you change any setting. That one reading tells you where to look. If the running batch stays near the cap, continuous batching is doing its job and your next limit is somewhere else. If it stays well below the cap while requests are waiting, spend your time on KV cache memory rather than on the scheduler.

The sums in this post, and the rest of the serving arithmetic I write about, are in a small MIT-licensed Python package, llm-inference-arithmetic, with browser calculators if you would rather not install anything.