If you serve a language model to more than one person, batching decides most of your bill. It is also one of the first things an interviewer asks about once a conversation turns to model serving, and the follow-up is always about size: how big a difference it really makes.
This article answers that with numbers. Each measurement here was taken in a lesson of our Hands-On Inference Engineering course, with vLLM serving on an RTX 4090 we rented on RunPod, and I copied the figures from those runs rather than recalling them.
The short version. Writing an answer reads all of the model's weights for every token, so the GPU has arithmetic to spare, and a batch spends it on other people's tokens. A static batch wastes some of it, because short answers leave their places empty until the longest one ends. Continuous batching refills each place at the very next step. It keeps adding people until the seats, the step budget or the KV cache runs out, and then people wait. On our RTX 4090, the same 48 answers took 24.33 seconds in fixed batches of twelve and 16.83 seconds with continuous batching.

A simple batch waits for its longest answer; continuous batching fills each free place at once. The answer lengths in this drawing are made up to show the idea; the measured runs come next.
Part 1: What batching is, and why it works
Continuous batching is a better way to batch, so start with why batching helps at all. It comes down to how a GPU writes an answer.
What is batching in LLM inference?
Batching means working out the next token for several people in the same step on the GPU. A language model writes one token per pass through its weights, and every pass reads all of those weights from memory. With a batch, that one read writes a token for everyone in it.
Writing tokens is limited by memory speed, not by arithmetic, so the GPU has spare arithmetic to give. That is why extra people in a batch cost each other so little at first.
What we measured: with Qwen3-8B on our RTX 4090, one person got 57 tokens a second and 32 people at once got 1,161 in total, 20.2 times as much, while each person's own speed fell only from 58.1 to 39.9 tokens a second.
Why does batching help writing so much more than reading?
A request has two halves. Prefill reads the whole question in one pass, so the GPU's arithmetic is already busy. Decode writes the answer one token per pass, and each pass pays for a full read of the weights to produce a single token, so the arithmetic sits mostly idle. Batching fills that idle arithmetic with other people's tokens.
What we measured: on the same card, Qwen3-0.6B read a 256-token prompt at 35,796 tokens a second and wrote at 642.0 tokens a second, about 56 times slower per token.
Why does batching make each answer cheaper?
A GPU is paid for by the hour. The cost of a token is the price of the hour divided by the tokens written in that hour, so writing more tokens in the same hour makes each one cheaper. The person pays with a little speed: each answer takes longer than it would alone.
What we measured: Qwen3-8B on our RTX 4090 wrote 161 tokens a second for one person and 2,179 for 32 at once, about 13.5 times as many tokens from the same GPU hour, while each person's answer took about 2.4 times longer.
Part 2: Static batching against continuous batching
This is the heart of it, and the part interviewers push on most: where a simple batch loses time, and what continuous batching does differently.
What is static batching, and what is wrong with it?
Static batching sends a group of requests through together, and the group finishes only when its slowest member does. Answers have very different lengths, so the places freed by short answers sit empty while the longest one keeps writing, and the people waiting in the queue cannot start until the whole group is done.
It is like a restaurant that seats twelve parties together and seats nobody new until all twelve tables have finished.
What we measured: in our run of 48 answers in fixed waves of 12, 47% of the space in the batch sat empty, and the run took 24.33 seconds.
What is continuous batching?
Continuous batching decides the batch afresh at every step. Between any two steps, finished requests leave and waiting requests join, as long as their KV cache fits, so the batch stays full and nobody waits for a stranger's long answer. The restaurant seats a new party the moment any table empties.
You will also hear it called iteration-level scheduling, the name it was given in the Orca paper of 2022, because the decision is made at every iteration of the model rather than once per batch.
What we measured: the same 48 answers, 12 at a time, took 16.83 seconds with continuous batching against 24.33 seconds in waves: 929 steps instead of 1,342. The answer lengths alone predicted a ratio of 0.69, and the measured ratio was 0.69.

The same 48 answers in waves of twelve, and with continuous batching. The lengths alone predicted the ratio of 0.69, and the real run matched it.
Does vLLM use continuous batching?
Yes, and you cannot switch it off. The scheduler lives in vLLM's engine core process, and before every step it picks the conversations to work on and how many tokens each one gets. The settings you can change only cap the size of a step.
What we measured: serving Qwen3-0.6B on the RTX 4090, a single request ran at 359 tokens a second. With 64 requests sharing the card the server produced 12,087 tokens a second, roughly 189 for each person, and the last of the 64 answers arrived 1.06 seconds after they were sent.
Can a new question be read in the same step as other people's answers are written?
Yes. A step has a budget of tokens. Every conversation that is writing takes one token of it, and whatever is left can go to reading new questions. Chunked prefill cuts a long question into pieces that fit the leftover budget and reads them over several steps, so the people already being answered keep getting words.
What we measured: at the default budget, vLLM read a question of 12,563 tokens in 7 pieces, and none of the eight answers running beside it stalled for longer than 300 ms. When we raised the budget to 16,384, the whole question went through in a single step and those eight answers froze for as long as 1,499 ms.
Part 3: The limits: seats, step budget and memory
Continuous batching keeps adding people until something runs out. Three things can run out, and each has its own setting in vLLM.

What a bigger batch gives the server and costs each person. At 64 people the KV cache was full and the total rose only 16% more.
What does max-num-seqs do?
--max-num-seqs sets the most conversations that can be running at once, like seats in a room. Requests beyond it wait in a queue even if memory is free. The default for vLLM 0.33's API server is 256, and 1,024 on H100-class cards of 70 GB or more.
Lowering it makes each running person faster and everyone else wait. Use it to protect a speed promise, not as a default.
What we measured: setting it to 8 cut the time per token from 26.08 to 18.65 ms, but raised the mean wait for the first token from 0.99 to 17.35 seconds and cut throughput 5.1 times.
What does max-num-batched-tokens do?
--max-num-batched-tokens is the token budget of one step: the most tokens the model works on at once. The default is 2,048 on GPUs like the RTX 4090 and 8,192 for the API server on H100-class cards. A bigger budget reads long questions in fewer steps, and makes everyone else's words stop for longer while it does.
How many requests can one GPU batch at once?
Roughly as many as the KV cache can hold, and fewer if you owe users a certain speed. Take the cache size vLLM prints when it starts and divide it by the length of a typical conversation. Go past that number and the total hardly grows while people begin to queue.
What we measured: for Qwen3-8B on an RTX 4090, 33,216 tokens of cache divided by 768-token conversations is about 43. In our sweep 32 people ran with no pauses, and 64 raised the total by only 16% while vLLM paused requests 59 times and the p99 wait for the first word reached 8.69 seconds.
What is preemption, and why does it happen?
People are let in while their notes fit, but every answer keeps getting longer as it is written. Sooner or later a running conversation asks for one more page of the KV cache and there is none. vLLM then stops the conversation that joined most recently, gives its pages to the others, and puts it back at the head of the queue. When room comes back, it reads that conversation's text again to rebuild the notes and carries on. Nothing is lost; the paused answers simply finish later.
What we measured: thirty-two answers of 1,500 tokens needed 3,104 pages of a cache that had 2,375. vLLM preempted 8 times, and every answer was complete after 40.64 seconds.

When long answers need more pages than the cache has, the newest wait.
How does PagedAttention help continuous batching?
Continuous batching needs memory that can be handed out and given back at any step. PagedAttention stores each conversation's notes in small pages of 16 tokens, taken from a shared pool as it grows, so a newcomer can start the moment a few pages are free and a finished conversation gives its pages straight back.
Part 4: Measuring it honestly
Batching numbers are easy to flatter by accident. Two habits keep them honest.
Why must a batching load test use fresh prompts?
Prefix caching is on by default in vLLM, and it remembers the work done on questions it has seen before. Send the same questions twice and the second run is timing the cache instead of the batching. Write new questions for every run, and confirm that vllm:prefix_cache_hits_total still reads 0 when the run ends.
Which numbers should you report for a batching test?
The total tokens a second for the server, each person's own speed, and the wait for the first word at the median and at the 99th percentile. The total tells you the cost; each person's speed tells you what users feel; the slow percentile shows the unlucky users that an average hides. Also count preemptions, because they are the first sign the batch is too big.
What we measured: at 64 people on our RTX 4090, the median first word took 1.61 seconds while the p99 took 8.69 seconds.
What to take away
A GPU writing an answer is waiting on memory, not on arithmetic, so a batch is nearly free until something runs out. Static batching throws part of that away by making everyone wait for the longest answer. Continuous batching refills a place the moment it empties, and on our card that alone took the same work from 24.33 seconds to 16.83. After that, the job is knowing which of the three limits you will hit first: the seats, the step budget or the KV cache.
The full page, with a link to the lesson each number comes from, is on our site: Continuous batching explained, with static batching measured beside it.
Go deeper: Hands-On Inference Engineering
I teach the runs behind this article in Hands-On Inference Engineering on System Design Academy. You rent the same RTX 4090, push the 48 answers through in fixed waves and then continuously, step the load from 1 person up to 64, and watch the cache fill until vLLM starts pausing requests. It suits backend, ML platform and AI engineering people who plan to host their own models and want to explain every number on their dashboard: see the Hands-On Inference Engineering course.
Top comments (0)