vLLM prefix caching on vs off: 7.5x the throughput with a 4,000-token system prompt on one L40S

Most chat traffic sends the same system prompt in front of every request. vLLM can keep the KV cache for that shared prefix and skip recomputing it (automatic prefix caching, on by default since the V1 engine). I wanted to know what that is worth in numbers, so I rented one L40S on Runcrate today (2026-10-27), ran the same chat-shaped load with prefix caching on and off, and terminated the machine as soon as the results were on disk.

Short version: with a 4,000-token shared system prompt and 64 users at once, turning prefix caching off cut output throughput from 1,902 to 254 tokens per second and pushed p99 time to first token from 0.38 s to 23.9 s.

Setup

• GPU: one L40S (48 GB), Kansas City, US, $1.14 per hour on demand at launch time.
• Model: Qwen/Qwen2.5-7B-Instruct, bf16, no quantization.
• Server: vLLM 0.11.0 (torch 2.8.0+cu128, transformers 4.57.6), --max-model-len 8192 --gpu-memory-utilization 0.90. vLLM reported 14.25 GiB of weights and 24.26 GiB of KV cache (454,272 tokens).
• Two modes: default (prefix caching on), and --no-enable-prefix-caching (off). Same server flags otherwise, fresh server per mode.
• Load: vllm bench serve with the random dataset. Every request is one shared prefix (1,000 or 4,000 tokens, the "system prompt") followed by 200 unique tokens, and exactly 128 output tokens (--ignore-eos). Concurrency 1, 16 and 64 (24, 160 and 384 requests). The client ran on the same machine, so no network is in the numbers.
• Cache hit rate comes from vLLM's own /metrics counters (vllm:prefix_cache_hits / vllm:prefix_cache_queries), read before and after every run.

The commands

Launch (the instance_type_id comes from GET /api/v1/instances/types):

curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"prefix-l40s","instance_type_id":"<id from /instances/types>",
       "ssh_key_id":"<your key id>","max_runtime_minutes":60}'

On the machine:

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install vllm==0.11.0 'transformers<4.58' --torch-backend=cu128

M=Qwen/Qwen2.5-7B-Instruct
for MODE in on off; do
  EXTRA=""; [ $MODE = off ] && EXTRA="--no-enable-prefix-caching"
  vllm serve $M --dtype bfloat16 $EXTRA --max-model-len 8192 \
    --gpu-memory-utilization 0.90 --port 8000 &
  until curl -sf localhost:8000/v1/models >/dev/null; do sleep 1; done
  for P in 1000 4000; do
    for C in 1 16 64; do
      case $C in 1) N=24;; 16) N=160;; 64) N=384;; esac
      vllm bench serve --backend vllm --model $M --dataset-name random \
        --random-prefix-len $P --random-input-len 200 --random-output-len 128 --ignore-eos \
        --num-prompts $N --max-concurrency $C --seed $((P + C)) \
        --save-result --result-dir results --result-filename bench_${MODE}_p${P}_c$C.json
      curl -s localhost:8000/metrics | grep -E '^vllm:prefix_cache_(queries|hits)'
    done
  done
  kill %1; wait
done

Then DELETE /api/v1/instances/<id> and a check that the instance list is empty.

Results

Prefix	Users	Cache	Requests/s	Output tok/s	Median TTFT	p99 TTFT	Median time per token	Hit rate
1,000	1	on	0.37	48	40 ms	44 ms	20.8 ms	85%
1,000	1	off	0.36	47	103 ms	106 ms	20.8 ms
1,000	16	on	5.27	675	163 ms	208 ms	22.5 ms	88%
1,000	16	off	3.72	476	314 ms	1,416 ms	31.1 ms
1,000	64	on	14.61	1,870	303 ms	773 ms	31.8 ms	88%
1,000	64	off	6.63	849	507 ms	5,940 ms	71.5 ms
4,000	1	on	0.37	47	57 ms	63 ms	21.1 ms	96%
4,000	1	off	0.33	42	377 ms	385 ms	21.1 ms
4,000	16	on	5.21	667	98 ms	147 ms	23.4 ms	99%
4,000	16	off	1.90	243	850 ms	4,987 ms	59.1 ms
4,000	64	on	14.86	1,902	172 ms	384 ms	32.1 ms	99.6%
4,000	64	off	1.99	254	1,936 ms	23,935 ms	240 ms

What that means for the bill, at $1.14 per hour, with the GPU kept busy at that concurrency:

Prefix	Users	$ per 1,000 requests, cache on	cache off	Ratio
1,000	16	0.060	0.085	1.4x
1,000	64	0.022	0.048	2.2x
4,000	16	0.061	0.167	2.7x
4,000	64	0.021	0.160	7.5x

What surprised me

1. With the cache on, the length of the system prompt almost stopped mattering. At 64 users, the 4,000-token prefix ran at 14.86 requests/s and the 1,000-token prefix at 14.61. The median time to first token was even lower with the longer prefix (172 ms against 303 ms), because a larger share of each prompt was already cached (99.6% hit rate against 88%). Without the cache, going from 1,000 to 4,000 tokens of prefix cut throughput by 3.3x.
2. The cost of "off" shows up in decode speed too, not only in TTFT. Median time per output token at 64 users went from 32 ms to 240 ms with a 4,000-token prefix. Every new request has to prefill 4,200 tokens, and those prefills share the GPU with everyone else's decoding, so every user in the batch slows down, not only the one who just arrived.
3. The tail is where it hurts. p99 time to first token at 64 users and a 4,000-token prefix was 23.9 s with the cache off. The median was 1.9 s. Some users wait for a long queue of prefills ahead of them.
4. For a single user, the cache saves latency but not money. At concurrency 1 the GPU is bound by decoding one stream, so requests per second barely moved (0.37 against 0.33). The 4,000-token prefix still cost 377 ms of prefill per request without the cache and 57 ms with it.
5. The first request pays. The 85% hit rate at 1,000 tokens and concurrency 1 is close to the share of each prompt that is shared (1,000 of 1,200 tokens). The cache never helps the unique part of a prompt.

What to take from it

• Prefix caching is already on in current vLLM. These "off" numbers are also what you get with the cache on if your prompts do not actually share a prefix: a timestamp, user name or request id at the top of the system prompt changes the first tokens and every request misses. Put the parts that never change first and the parts that change last.
• If your traffic is many users behind one long system prompt, size the GPU from the cache-on numbers, but check the hit rate in /metrics in production to see whether you really get them.

Caveats

• One run per setting, on one machine, on one day. Treat differences under about 10% as noise.
• Synthetic prompts from vLLM's random dataset: one prefix shared by every request. Real traffic mixes several system prompts, multi-turn history and different lengths; hit rates will be lower.
• The cache could hold the whole prefix easily here (454k tokens of KV cache). With many distinct long prefixes and a full cache, evictions would lower the hit rate.
• Price is the on-demand price at launch time and moves daily.

Reproduce it

1. Create an API key and add an SSH key (POST /api/v1/ssh-keys).
2. Pick an L40S from GET /api/v1/instances/types and launch with the curl above, max_runtime_minutes set as a safety net.
3. Run the block above, copy results/ back, delete the instance.

The whole run, including install, model download and both server starts, was billed $0.38.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

Comments

[CodeArchitect_J]

The p99 time-to-first-token going from 0.38s to 23.9s is the number that should scare people more than the throughput drop. Anyone who "optimizes" by putting per-user data at the start of the system prompt is silently turning this off. Keep the shared part first and the variable part last.

[Runcrate]

Agreed, the tail is where it shows. In this run, with the 4,000-token prefix and 64 users, the median TTFT went from 0.17 s to 1.9 s with the cache off, but p99 went from 0.38 s to 23.9 s.

One caveat: we didn't test per-user data at the top of the prompt directly. The cache-off run is the closest thing we measured, not the same case (for example, a timestamp on the first line followed by the same 4,000 tokens). Happy to run that and post the numbers if it's useful.un, with the 4