Originally published on AI Tech Connect.
What you need to know Prefill — the phase that reads your prompt and produces the first token — is a set of large matrix multiplications over every prompt token at once. It saturates a GPU's floating-point units. Decode — the phase that emits every token after the first — processes exactly one token per step per sequence, and spends most of its time re-reading the key-value cache out of high-bandwidth memory. It saturates memory bandwidth and leaves the floating-point units largely idle. These are not similar workloads wearing different hats. They are different workloads. When both run on the same GPU pool, a scheduler has to interleave them, and every interleaving choice hurts somebody. Let a long prefill run to completion and every user currently streaming tokens sees their stream…
Top comments (0)