Chunked Prefill: Why max_num_batched_tokens=512 Kills Throughput

A team ships a 70B model behind vLLM. Time-to-first-token looks fine at low load, then goes to multiple seconds whenever someone pastes a 30k-token document, because that one prefill blocks every decode step behind it. They enable chunked prefill, reason that "smaller chunks mean smoother latency," and set --max-num-batched-tokens 512. TTFT jitter improves. Throughput drops by more than half, and p99 TTFT on long prompts gets worse than before.

Nothing is broken. The scheduler is doing exactly what it was told. The problem is that max_num_batched_tokens is not a smoothness dial — it is the total token budget per forward pass, shared by prefill and decode, and it sits directly on top of the roofline crossover for your GPU.

TL;DR

• Chunked prefill splits a long prompt across several forward passes so decode requests can ride along in the same batch, converting head-of-line blocking into a steady tax.
• max_num_batched_tokens is a per-step budget shared by prefill chunks and decode tokens. Every running decode consumes one token of it. With 256 concurrent decodes and a 512 budget, only 256 tokens of prefill move per step.
• On an H100, the compute/memory crossover for a dense BF16 model is roughly 300 tokens in theory and closer to 1–2k in practice. Chunks below that re-read the entire weight matrix from HBM to do almost no math — you pay a full forward pass for a sliver of work.
• Set the budget at or above the measured knee of your prefill-throughput curve (commonly 2048–8192), then bound TTFT with concurrency limits and partial-prefill caps, not by shrinking chunks.
• Chunked prefill and prefix caching interact: budget against uncached tokens, and keep chunk boundaries aligned to the KV block size or you will fragment cache hits.

What is chunked prefill actually doing to the batch?

Chunked prefill lets the scheduler put a slice of a long prompt into the same forward pass as other requests' decode steps. Without it, a scheduler has two bad options: run prefill alone (decodes stall for the whole prompt) or delay prefill (TTFT stalls). Chunking makes prefill preemptible at chunk granularity.

Mechanically, a step looks like this. The scheduler picks a token budget B. Each running request in decode contributes exactly 1 token. Whatever is left goes to prefill chunks:

budget            = max_num_batched_tokens
decode_tokens     = num_running_seqs          # 1 token per decoding sequence
prefill_available = budget - decode_tokens

That single line is where most misconfigurations die. Decode is not free of the budget — it is charged first. Under load, decode fills the batch and prefill gets the scraps.

Concretely, with max_num_batched_tokens=512 and 400 sequences in decode, a 32,768-token prompt needs 32768 / (512 - 400) ≈ 293 scheduler steps before it emits its first token. Each of those steps is a complete forward pass through the model. Raise the budget to 8192 and the same prompt clears in about 5 steps.

Why does a small max_num_batched_tokens destroy throughput?

Because a forward pass reads the entire weight set from HBM regardless of how many tokens are in it. Below a few hundred tokens, you are memory-bandwidth bound, and the marginal cost of adding tokens is nearly zero — so tiny chunks throw that free capacity away.

The crossover is easy to derive. For a dense model with P parameters in BF16, a forward pass over T tokens costs roughly 2·P·T FLOPs and reads about 2·P bytes of weights:

# H100 SXM, BF16 dense
PEAK_FLOPS = 989e12      # ~989 TFLOPS BF16, no sparsity
HBM_BW     = 3.35e12     # ~3.35 TB/s HBM3

# compute_time(T) = 2*P*T / PEAK_FLOPS
# memory_time     = 2*P    / HBM_BW        (weights, independent of T)
# compute-bound when 2*P*T/PEAK_FLOPS > 2*P/HBM_BW

T_crossover = PEAK_FLOPS / HBM_BW          # ≈ 295 tokens

The P cancels. The break-even token count is a property of the hardware ratio, not the model size — roughly 300 tokens on H100, similar order on A100 and MI300X. Below it, you are burning a bandwidth-limited forward pass to do almost nothing.

In practice the knee is higher than 295. You never hit peak FLOPs, attention over long prefixes adds quadratic work that does not amortize weights, and per-step Python/scheduler overhead is fixed. Measured prefill-throughput curves usually keep climbing until 1–2k tokens and flatten somewhere in the 2k–8k range. That is why a 512-token budget lands squarely in the worst region: too small to saturate the GPU, small enough to multiply the number of passes.

There is a second, subtler cost. Chunk k of a prompt must attend to all k-1 previous chunks' KV. Splitting a prompt into N chunks does not change total attention FLOPs much, but it does re-read the accumulated KV cache on every chunk — turning one streaming pass over KV into O(N) passes. For a 100k-token prompt at a 512 budget, that re-reading dominates.

How do I choose max_num_batched_tokens?

Measure the knee, don't guess it. Run prefill-only benchmarks at fixed chunk sizes and find where tokens/second stops improving; set your budget at or just past that point.

# Sweep the budget with decode traffic held out.
for B in 256 512 1024 2048 4096 8192 16384; do
  vllm serve meta-llama/Llama-3.3-70B-Instruct \
    --tensor-parallel-size 4 \
    --max-num-batched-tokens $B \
    --max-num-seqs 256 \
    --no-enable-prefix-caching &          # isolate prefill cost first
  sleep 90
  vllm bench serve \
    --model meta-llama/Llama-3.3-70B-Instruct \
    --dataset-name random \
    --random-input-len 8192 --random-output-len 1 \
    --num-prompts 200 --request-rate inf \
    --percentile-metrics ttft,tpot,itl
  pkill -f "vllm serve"; sleep 20
done

Then re-run the winner with --random-output-len 512 and real concurrency to see what it does to inter-token latency. You are looking for two curves that move in opposite directions:

• Throughput / TTFT under long prompts improves with a larger budget, then flattens.
• ITL p99 for streaming users degrades with a larger budget, because any decode step sharing a batch with a big prefill chunk takes as long as that chunk.

That second effect is the real trade-off and it is often mis-attributed. A pure decode step on a large model is a short, bandwidth-bound operation. Fold a 4096-token prefill chunk into it and that step becomes compute-bound and several times longer. Every streaming user sees a hitch. Shrinking the chunk shrinks the hitch — and that is the only thing shrinking the chunk buys you.

So the right shape of a config is: budget large enough to be efficient, with separate knobs to bound the tail.

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-num-batched-tokens 8192 \      # sized from the measured knee
  --max-num-seqs 256 \
  --max-num-partial-prefills 1 \       # only one long prompt in flight
  --long-prefill-token-threshold 4096  # "long" = above this many tokens

--max-num-partial-prefills is the anti-starvation knob. Without a cap, ten 100k-token prompts arriving together get round-robined across chunks, and all ten finish their prefill late — the classic fair-scheduling failure where everybody loses. Capping concurrent long prefills to 1 (or 2) makes them complete serially and gets first tokens out sooner for most of them.

If your ITL SLO is genuinely tight and your TTFT SLO is genuinely tight, no single budget satisfies both, because they are the same knob pulled in opposite directions. That is the argument for disaggregated prefill/decode: run prefill on one pool of GPUs, decode on another, ship KV between them. You stop trading and start paying in hardware and interconnect instead.

Why does chunked prefill interact badly with prefix caching?

Because the budget must be charged against uncached tokens, and chunk boundaries must line up with KV block boundaries. Get either wrong and you either misschedule or silently lose cache hits.

With prefix caching on, a 20k-token prompt whose first 18k tokens are already cached needs only 2k tokens of actual computation. A scheduler that budgets on prompt length would split it into chunks that are mostly no-ops; one that budgets on uncached length packs it into a single efficient step. Modern vLLM does the latter — but your own admission control, queueing, and capacity planning frequently do not. If you size concurrency from raw prompt tokens, you will under-admit by a wide margin on cache-heavy workloads like multi-turn agents.

The alignment issue is quieter. KV cache blocks are fixed-size (16 tokens is a common default). A chunk that ends mid-block leaves a partial block that cannot be shared as a prefix by another request. On workloads where many requests share a long system prompt and tool schema, misaligned boundaries can turn a clean shared prefix into a set of near-miss prefixes. Keep max_num_batched_tokens a multiple of the block size — 8192 and 4096 are safe, 500 is not.

One more: increasing the token budget increases activation memory per step, which comes out of the same GPU memory pool as the KV cache. Going from 2048 to 16384 can shrink num_gpu_blocks enough to trigger preemption under concurrency. Check the reported KV cache blocks after any budget change; a throughput regression right after "we made chunks bigger" is usually this.

What if I'm calling Claude or GPT instead of self-hosting?

You don't set these knobs, but you can see them in your latency traces, and the same mechanism explains the shape. TTFT tracks uncached input tokens, not total input tokens — which is why a well-structured, cache-friendly prompt (stable system prompt and tool definitions first, volatile content last) improves TTFT far more than shaving a few hundred tokens off the middle. Inter-token latency jitter that correlates with nothing you sent is other tenants' prefill chunks landing in your batch. Measure TTFT and ITL as separate distributions; a single "latency" number blends two mechanisms that respond to completely different fixes.

Summary

max_num_batched_tokens=512 kills throughput because chunked prefill charges every running decode sequence against the same per-step token budget, and 512 tokens sits below the compute/memory crossover of modern accelerators — roughly 300 tokens in theory on an H100, 1–2k in practice. Small chunks force many full forward passes, each re-reading all model weights and the accumulated KV cache to do a sliver of useful math, while under concurrency the decode tokens eat most of the budget and long prompts crawl through hundreds of scheduler steps before their first token. Size the budget from a measured prefill-throughput knee (usually 2048–8192, and a multiple of the KV block size), bound tail latency with --max-num-partial-prefills and concurrency limits rather than by shrinking chunks, budget against uncached tokens when prefix caching is on, and recheck your KV block count after every budget change.