AI Tech News

Posted on Nov 3

vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap

#ai #machinelearning #llm #vllm

TL;DR

What shipped: vLLM 0.31.0 (2026-10-05) added vllm preload, a per-GPU daemon that keeps post-quantized weights resident so a restarting engine maps them over CUDA IPC instead of reloading from disk, plus experimental CRIU-based initialized-engine snapshots. The design goal, in the original PR's words, is that "engine restarts are dominated by weight loading from disk."
What it does not do: the daemon is node-local and must itself load from disk once, so it speeds up restarts on a GPU that already holds the weights. It does not speed up scaling out to a fresh node.
What a faster start is worth (illustrative simulation): with a 1.5x traffic burst and a 10 s queue-wait SLO, cutting replica startup from 300 s to 45 s raises the safe baseline utilization from 69% to 83%. For 100 GPUs of baseline load that is 145 GPUs versus 120. At 2x bursts it is 196 versus 159.
A correctness gap to know about: as of 2026-11-03, issue #59647 is still open. The weight-cache fingerprint hashes safetensors headers, not weight values, so a daemon holding checkpoint A can serve an engine started on checkpoint B without any warning. The proposed fix samples only 1 KiB per tensor, which on Qwen3-8B is 0.002% of the file.

What did vLLM actually ship for fast restarts?

The feature landed in two pieces. The first is the weight cache daemon, merged as PR #54921 on 2026-09-04 and exposed as vllm preload in v0.31.0. One daemon process per GPU loads its tensor-parallel shard, runs quantization post-processing, and exports every parameter as a CUDA IPC handle. An engine started with --load-format ipc_cache builds the model on the meta device and asks the daemon for the tensors. The second piece is vllm snapshot create/restore, which uses CRIU to freeze a fully initialized engine and bring it back before the HTTP server binds.

# one daemon per GPU, loads from disk once
vllm preload --model meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 4

# engine restarts map the resident weights instead of reading disk
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --tensor-parallel-size 4 --load-format ipc_cache

The two mechanisms keep very different amounts of state, and their constraints differ accordingly. Everything in this table comes from the vLLM docs pages for the two features as of today.

	Weight preload	Initialized snapshot
What persists	Post-quantized weights in GPU memory (and FlashInfer autotune results on disk)	The whole engine process tree and CUDA state
Scope	Same node, same user, Unix socket	Same host, GPU, driver, kernel, vLLM version, engine args
Parallelism	Tensor, expert and data parallel, including multi-node (one launcher per node)	TP1, one NVIDIA GPU only
Validated	PR test results include TP2 and MTP draft models	Only dense float16 TP1; acceptance run was Qwen3-0.6B on an H200
Cost while idle	The daemon pins the weights in GPU memory	Disk: artifact size can approach process plus GPU memory
Fallback	Disk load if daemon missing or fingerprint mismatched	None: restore fails closed before CRIU runs

One number from outside vLLM gives a sense of scale for the weight-load part alone. The README of a separate PyPI plugin built on the same idea, vllm-ipc-cache 0.2.0, reports a 27B model at tensor parallel 4 on a MetaX N300-A dropping from 35.80 s to 0.47 s. It is one vendor-reported run on non-NVIDIA hardware and covers the weight-attach step only, not an end-to-end restart.

Which startup events does a faster restart actually help?

Preload covers only some of the reasons you start an engine.

Rolling upgrades and config changes. The daemon outlives the engine, so you restart the engine and keep the weights. This is the cleanest win.
Engine crashes where the GPU is healthy. The daemon is a separate process, so a crashed engine can remap the same weights. If the fault is at the GPU level, the daemon probably goes down with it and you are back to a disk load. I did not test that case. The docs say that if any daemon rank exits, you should restart the whole preload group before attaching engines.
Scale-out to a new node. Not helped. CUDA IPC handles are node-local, the docs state that each node serves only its own GPUs, and the daemon itself loads from disk. The one scale-out feature is warmup-cache seeding: a new daemon can fetch rank-matched FlashInfer autotune tables from a source replica over an authenticated, unencrypted TCP channel.
Time-sharing a GPU between models. This is what snapshots are for: "repeatedly activating the same model and engine configuration on the same machine."

The autotune detail matters more than it looks. The docs say that once weights come from the daemon, the FlashInfer autotune pass "dominates what is left of the startup time" on a large MoE model, which is why vllm preload now runs it at preload time. v0.31.0 also parallelized Triton warmup compilation (#58582). So the honest accounting of a restart is weights, then autotune, then warmup and CUDA graph work, and each has its own fix.

Here is the arithmetic that makes rolling upgrades the main prize. A rolling restart of 16 replicas, one at a time, runs the fleet at 15/16 capacity (93.75%) for 16 times the per-replica restart time.

At 300 s per replica that is 80 minutes of reduced capacity; at 45 s it is 12 minutes; at 10 s it is 2.7 minutes. If you ship engine upgrades every couple of weeks, that is the number to compare against the cost of keeping a daemon pinned.

How much utilization does a faster start buy you?

Startup time matters to capacity planning through one mechanism: during a burst, demand exceeds capacity for roughly the detection lag plus the startup time, and the backlog built in that window has to drain inside your queue-wait SLO. If the burst multiplies load by B, baseline utilization is u, the window is W seconds and the SLO is S seconds of queue wait, the backlog stays inside the SLO when:

(B * u - 1) * W <= S      so      u_max ~= (1 + S / W) / B,   W ~= startup + detection lag

I checked this with a fluid-queue simulation (illustrative, not a measurement of any real fleet): 16 replicas, a 30-minute run, demand ramping by a factor B over 30 s, a scaler that averages demand over 30 s, checks every 10 s and restores the baseline utilization, and a violation counted for every second the queue wait exceeds 10 s. The core loop:

for t in range(T_END):
    lam = demand(t, lam0, burst)
    n += sum(c for r, c in pending if r <= t)            # replicas that finished starting
    pending = [(r, c) for r, c in pending if r > t]
    if t % 10 == 0 and t >= 30:
        want = ceil(mean(hist[-30:]) / (u0 * MU))        # restore baseline utilization
        if want > ordered:
            pending.append((t + startup, want - ordered)); ordered = want
    q = max(0.0, q + lam - n * MU)                       # fluid backlog
    violations += (q / (n * MU)) > 10.0

For each startup time I searched for the highest baseline utilization with zero violation seconds:

Replica startup	Max safe baseline utilization, burst x1.5	GPUs for 100 GPUs of load	Max safe utilization, burst x2.0	GPUs for 100 GPUs of load
300 s	0.69	145	0.51	196
120 s	0.73	137	0.54	185
45 s	0.83	120	0.63	159
10 s	0.99 (search cap)	101	0.85	118

The closed form with W set to startup plus 45 s predicts 0.686 for the 300 s row against 0.69 in the simulation. It is conservative for fast starts, because the ramp is finite and the scaler orders in steps; at 45 s it predicts 0.74 against 0.83. The 10 s row is the idealized limit, since it ignores KV cache warmup, load balancer registration and anything else that happens after the process is up.

The same data as violation seconds shows how sharply the cliff moves. At 80% baseline utilization and a 1.5x burst, a 300 s startup spends 413 s over the SLO with a peak queue wait of 57 s. At 120 s it is 113 s over, and at 45 s it is zero violations with a peak wait of 7.6 s.

Two readings follow. First, going from 300 s to 45 s buys about 17% to 19% fewer GPUs in these scenarios, which is real money but not a revolution. Second, those gains apply to scale-out, and preload does not accelerate scale-out to fresh nodes. Where this model really applies is a parked warm replica on the same host (a daemon on an idle GPU, or a snapshot on disk), and that parked GPU costs about the same as the headroom replica you were trying to avoid. For most fleets the dependable return is the rolling-restart arithmetic above, not the autoscaling story.

Can the weight cache serve the wrong checkpoint?

It can today, in a specific case, and the project knows. The fingerprint that decides whether a daemon's weights match the engine covers architecture, TP and DP layout, dtype, quantization, RoPE parameters, KV cache dtype and vLLM version. For the checkpoint itself it hashes each safetensors shard's header: tensor names, dtypes, shapes and byte offsets. A header contains no weight values. Issue #59647, filed 2026-10-01 and still open on 2026-11-03, reports that a base model and its fine-tune therefore get the same key. If a daemon preloaded from A is still running and an engine starts on B with ipc_cache, the daemon serves A's tensors, the engine never reads B from disk, and nothing is logged. A write-up on dev.to by a separate author reproduced it on 0.30.0 by zeroing one norm tensor while leaving the header intact. The docs promise a safe fallback on mismatch, which is why this surprises people.

The open fix, PR #59648, adds file size plus the first 1 KiB of every tensor to the hash. I re-implemented both schemes (the logic is small) and ran them on eight 1024x1024 float16 tensors, then changed the values in different ways:

Change to the checkpoint	Header-only hash detects it	Sampled 1 KiB hash detects it
Noise added to every tensor (a dense fine-tune)	no	yes
One whole tensor shifted by a constant	no	yes
First row of one tensor zeroed	no	yes
Row 500 of one tensor zeroed	no	no
One element at row 900 changed	no	no
Byte-identical copy in another directory	same key (wanted)	same key (wanted)

The proposal catches realistic fine-tunes, because training changes nearly every value, including each tensor's first kilobyte. It cannot catch a surgical edit that leaves the head of each tensor alone. The reason is coverage: on the real headers I fetched, Qwen3-0.6B has 311 tensors and 1.50 GB, so 1 KiB per tensor is 0.018% of the file; Qwen3-8B is 0.0022%; Qwen3-32B has 707 tensors over 65.5 GB and the sample is 0.00095%.

That is a fine tradeoff for a startup-path check, since reading everything would defeat the purpose. The practical conclusion is not to rely on the fingerprint for identity. Tie the daemon's lifetime to a digest you computed once, at promotion time, with the full file bytes. SHA-256 ran at about 1.8 GB/s on my PC (median of five runs on an in-memory buffer), so a 65.5 GB checkpoint is roughly 36 s of CPU time if the disk can feed it, paid once in CI and not on every restart.

# at promotion time, once per checkpoint
DIGEST=$(sha256sum /models/qwen3-8b/*.safetensors | sha256sum | cut -c1-16)
SOCK=/run/user/$UID/vllm-wc/$DIGEST
mkdir -p -m 700 "$SOCK"

vllm preload --model /models/qwen3-8b --weight-cache-socket-dir "$SOCK"
vllm serve /models/qwen3-8b --load-format ipc_cache \
    --model-loader-extra-config "{\"socket_dir\": \"$SOCK\"}"

An engine pointed at a different checkpoint computes a different digest, looks in a different directory, finds no socket, and falls back to disk, which is the documented behavior for an unavailable daemon. I used only flags from the preload docs and did not run this against a GPU engine for this post. Add a temperature-0 canary request per checkpoint after every start; the snapshot feature does exactly that internally, comparing a recorded token and log probability before it returns.

A pre-flight checklist for adopting fast start

Count your restart events by type for a month: upgrades, crashes, scale-outs. Preload only pays on the first two.
Measure your own restart phases separately: weight load, autotune, warmup and CUDA graphs. Do not assume weights are the whole story.
Key the daemon to a full-file digest, and run a canary after each start.
Keep the disk fallback enabled in production and alert on it. Budget the GPU memory the daemon pins, and do not combine the loader with sleep mode.

FAQ

What does vllm preload do?
It starts one daemon per GPU that loads and post-processes the weights once and keeps them in GPU memory. Engines started with --load-format ipc_cache map those weights over CUDA IPC instead of reading the checkpoint again.

Does preload speed up autoscaling to a new GPU node?
No. The handles are node-local and the daemon must load from disk the first time. It helps restarts on a node that already holds the weights. Only the FlashInfer autotune tables can be seeded to a new daemon from another replica.

Can the weight cache load the wrong model?
Under the current fingerprint, yes, if two checkpoints share names, shapes, dtypes and offsets and a daemon for the other one is still running (issue #59647, open on 2026-11-03). A digest-keyed socket directory and a canary request avoid it.

How much GPU capacity does faster startup save?
In my illustrative simulation with 1.5x bursts and a 10 s queue-wait SLO, 45 s startup needs 120 GPUs where 300 s needs 145, for 100 GPUs of baseline load. Your own burst size, detection lag and SLO will move that.

DEV Community