Chunked Prefill: Why One Long Prompt Stalls Every Decode

Your p99 inter-token latency chart has spikes. Not a slow ramp — near-vertical walls, 40ms steady state jumping to 900ms for one step, then back to normal. Nobody's GPU is out of memory. Throughput looks fine. Every dashboard says the cluster is healthy.

Then you correlate the spikes with request arrivals and find it: one user pasted a 60k-token document. While your server ran the prefill for that single request, every other user's stream froze. That's the failure mode chunked prefill exists to fix, and the reason max_num_batched_tokens is the most consequential knob in your vLLM config that you probably never touched.

TL;DR

• Prefill and decode compete for the same GPU step. An unchunked 60k-token prefill occupies the model for hundreds of milliseconds to seconds, during which every in-flight decode produces zero tokens. Users see it as a frozen stream, not as slow TTFT.
• Chunked prefill splits a long prompt into fixed-size slices and packs each slice into a batch alongside the 1-token decodes, so per-step latency stays bounded no matter how long the incoming prompt is.
• The extra attention FLOPs are tiny: the overhead ratio is chunk_size / prompt_length. A 32k prompt in 2k chunks costs ~6% more attention work, and zero extra linear-layer work.
• The real floor on chunk size is arithmetic intensity. A GEMM over M tokens has intensity ≈ M FLOP/byte; below roughly 300–500 tokens on an H100 you fall off the roofline and re-stream model weights for nothing.
• Tune it as a latency dial: bigger max_num_batched_tokens → better TTFT and prefill throughput, worse ITL tail. Smaller → smooth streams, lower ceiling. Speculative decoding silently eats this budget.

Why does prefill block decode in the first place?

Because a continuous-batching scheduler runs one forward pass at a time, and classically it had to choose: this step is a prefill step, or this step is a decode step.

The two phases have opposite shapes. Prefill processes N prompt tokens in parallel — it's a big, compute-bound GEMM that saturates the tensor cores. Decode processes exactly one token per sequence — it's memory-bandwidth-bound, dominated by streaming weights and KV cache out of HBM, with the tensor cores mostly idle.

A naive scheduler that admits a 60k-token prefill hands the whole GPU to that one request for the duration. With 64 users mid-stream, all 64 get a stall equal to the full prefill time. Your average TPOT barely moves — one bad step out of hundreds. Your p99 ITL is destroyed. And ITL tail is what users actually perceive: a stream that hitches for a second reads as "broken," while a uniformly 20% slower stream reads as normal.

This is the generation stall that SARATHI (Agrawal et al.) named and that DeepSpeed-FastGen shipped as "dynamic SplitFuse." Both landed on the same answer.

What does chunked prefill actually do?

Chunked prefill caps the number of tokens in any single forward pass and fills that budget decode-first.

Each scheduling step, the engine:

1. Admits every running decode — one token per sequence, so 64 concurrent streams cost 64 tokens of budget.
2. Fills the remaining budget with a slice of one or more pending prefills.
3. Runs a single mixed forward pass over the concatenated token stream.

The attention kernel handles the mix because it's already a varlen kernel: each sequence in the batch carries its own query length and its own KV range. A decode sequence is just a sequence with query length 1. A prefill chunk is a sequence with query length 2048 whose keys include all previously cached chunks of the same prompt. Nothing special is needed beyond correct cu_seqlens bookkeeping.

Two things follow. First, per-step latency is bounded by the token budget, not by the largest prompt anyone sends. Second — and this is the part people miss — the decodes get faster per unit of work. Decode alone is bandwidth-bound; you pay a full sweep of model weights from HBM to produce 64 tokens. Piggyback 2000 prefill tokens onto that same weight sweep and the weight-read cost is amortized across 2064 tokens instead of 64. Chunked prefill is usually a throughput win, not just a latency trade.

How much extra compute does chunking cost?

Chunking makes the same prompt re-read its own KV cache once per chunk. Do the arithmetic instead of guessing.

For chunk index i (0-based), the queries number C and the keys number (i+1)·C. Attention FLOPs for that chunk scale as C · (i+1)C · d. Sum over k = N/C chunks:

chunked:    d·C²·(k(k+1)/2)  =  d·(N² + N·C)/2
unchunked:  d·N²/2
overhead:   C/N

A 32k prompt in 2k chunks: 6.25% more attention FLOPs. A 128k prompt in 2k chunks: 1.6%. And that overhead applies only to attention — the QKV/MLP projections process each token exactly once either way, and at moderate context lengths those linear layers dominate the FLOP budget. Net overhead lands in the low single digits.

So why not chunk into 128-token slices and get perfectly smooth latency? Because of the other side of the ledger.

What sets the floor on chunk size?

Arithmetic intensity. Each chunk is a separate forward pass that re-streams the entire weight set from HBM.

For a GEMM against a weight matrix with P parameters in bf16, processing M tokens: FLOPs ≈ 2·M·P, bytes moved ≈ 2·P. Intensity ≈ M FLOP/byte. The token count is the arithmetic intensity.

An H100 does roughly 1000 TFLOPS of dense bf16 against roughly 3.3 TB/s of HBM — a machine balance around 300 FLOP/byte. Below ~300 tokens per pass you are memory-bound: you paid to move all the weights and used a fraction of the compute they could have fed. Chunk a 60k prompt into 128-token slices and you've turned one compute-bound pass into 470 memory-bound ones. Prefill throughput craters.

That's why the useful range is roughly 512 to 8192 tokens, and why the answer is never "chunk as small as possible."

How do I tune max_num_batched_tokens?

Treat it as a latency/throughput dial, and set it from your target ITL rather than from a blog post's default.

# vLLM V1 — chunked prefill is on by default; the budget is the real knob.
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-num-batched-tokens 4096 \
  --max-num-seqs 128 \
  --enable-prefix-caching \
  --max-num-partial-prefills 2 \
  --max-long-partial-prefills 1 \
  --long-prefill-token-threshold 8192

The last three flags matter more than they look. Without them, a single 200k-token prefill monopolizes the prefill share of the budget for its entire duration, and a 300-token prompt that arrives behind it waits out all 100 chunks. --long-prefill-token-threshold classifies anything above it as "long," --max-long-partial-prefills limits how many long ones run concurrently, and --max-num-partial-prefills bounds total concurrent partial prefills. That's head-of-line blocking control for prefill — a short-prompt fast lane.

Sizing heuristic: pick your ITL SLO (say 50ms), measure how many tokens per forward pass your deployment processes in that time at your TP size, and set the budget just under it. Then raise --max-num-seqs until GPU utilization plateaus. Defaults have moved across vLLM versions, so read them off your build rather than trusting memory.

Other engines, same idea, different spelling: SGLang uses --chunked-prefill-size, TensorRT-LLM uses enable_chunked_context with max_num_tokens.

Why did my ITL get worse after enabling speculative decoding?

Because draft tokens are charged against the same token budget, and nobody tells you.

With speculative decoding at 4 draft tokens, each running sequence contributes 5 tokens per step (4 speculative + 1 bonus), not 1. At 64 concurrent streams that's 320 tokens of budget consumed by decode instead of 64. With --max-num-batched-tokens 512, prefill drops from 448 tokens per step to 192 — prefill progress falls by more than half, TTFT under load climbs, and queued long prompts drain far more slowly.

If you turn on speculative decoding, raise the token budget in proportion to (1 + num_speculative_tokens) or you've silently reconfigured your scheduler.

Prefix caching interacts too, but in your favor: cached blocks skip prefill entirely, so a 30k-token system prompt with a warm cache may contribute only its uncached tail to the budget. Cache hit rate is effectively a prefill-load multiplier.

How do I measure this?

Don't average. Look at the ITL distribution and its worst step, under a workload that mixes prompt lengths — that mix is the whole point.

import asyncio, time, statistics
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="x")

async def stream_once(prompt: str, tag: str):
    itls, t_prev, ttft = [], None, None
    t0 = time.perf_counter()
    s = await client.chat.completions.create(
        model="meta-llama/Llama-3.3-70B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=256, stream=True,
    )
    async for chunk in s:
        if not chunk.choices or not chunk.choices[0].delta.content:
            continue
        now = time.perf_counter()
        if t_prev is None:
            ttft = now - t0
        else:
            itls.append((now - t_prev) * 1000)
        t_prev = now
    return tag, ttft, itls

async def main():
    short = "Summarize the tradeoffs of continuous batching."
    long_ = "Analyze this document.\n" + ("lorem ipsum dolor sit amet. " * 20_000)
    # 32 short streams running when one huge prefill lands mid-flight
    tasks = [stream_once(short, "short") for _ in range(32)]
    async def delayed():
        await asyncio.sleep(1.0)
        return await stream_once(long_, "long")
    results = await asyncio.gather(*tasks, delayed())

    victim = [itl for tag, _, itls in results if tag == "short" for itl in itls]
    victim.sort()
    p = lambda q: victim[int(len(victim) * q)]
    print(f"short-stream ITL  p50={p(.5):.1f}ms  p99={p(.99):.1f}ms  max={victim[-1]:.1f}ms")

asyncio.run(main())

The diagnostic signature is a max that is 10–30x p50 while p50 looks great. Sweep --max-num-batched-tokens across 512 / 2048 / 8192 and watch that max collapse toward p50 while aggregate throughput drops. That curve is your actual trade-off, measured on your model and your hardware.

When should I disaggregate prefill and decode instead?

When one token budget can't satisfy both SLOs at once — typically at high volume with genuinely bimodal prompt lengths.

Chunked prefill interleaves two workloads on the same GPU, so they still share memory bandwidth, SM occupancy, and one batch shape. Prefill/decode disaggregation (DistServe, Splitwise, Mooncake, vLLM's KV-transfer connectors, NVIDIA Dynamo) runs them on separate GPU pools and ships the KV cache over NVLink or RDMA. Each pool gets its own parallelism strategy and its own tuning: prefill pool optimized for compute throughput, decode pool for batch size and bandwidth.

The costs are real. You now move gigabytes of KV per request across the fabric, you need both pools sized correctly or one idles, and the added hop shows up in TTFT. Below a few hundred requests per second, chunked prefill on colocated workers usually wins on simplicity and utilization.

The short answer

Chunked prefill stops one long prompt from stalling every decode by capping how many tokens any single forward pass may process, then packing each prompt slice into the same batch as the in-flight 1-token decodes. The extra attention FLOPs are bounded by chunk_size / prompt_length — single digits at realistic sizes — and the piggybacked prefill tokens amortize the weight-read cost of the decode step, so it often raises throughput too. The floor on chunk size is arithmetic intensity: a pass over M tokens has intensity ≈ M FLOP/byte, so slices below a few hundred tokens turn compute-bound work into memory-bound work. Set max_num_batched_tokens from your ITL SLO, add partial-prefill limits so short prompts don't queue behind a 200k-token document, scale the budget when you enable speculative decoding, and reach for prefill/decode disaggregation only once a single token budget genuinely can't serve both SLOs.