Can vLLM Run GGUF? Yes — on GPU Only

Originally published at mrsaynothing.dev.

Last week a reader asked why their GGUF file 404s inside vLLM on a CPU box while the same file runs fine in Ollama. Short answer, before anything else: yes, vLLM runs GGUF — through an official plugin, on GPU only. The plugin is vllm-gguf-plugin, the syntax is repo:quant_type, and the moment you try it on a CPU you are outside the supported hardware table. Everything below comes from the vLLM documentation and the vllm-project/vllm repo record (92,000+ stars since February 2023) — the docs, not my benchmark rig.

How do you serve a GGUF model with vLLM?

Two steps: install the plugin, then point vLLM at the model. GGUF support no longer ships inside core vLLM — the docs note it "has migrated to OOT vllm-gguf-plugin", so a plain pip install vllm is not enough:

uv pip install vllm-gguf-plugin

# Directly from Hugging Face, repo_id:quant_type format:
vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M \\
  --tokenizer Qwen/Qwen3-0.6B

# Or a local file you already downloaded:
vllm serve ./Qwen3-0.6B-Q4_K_M.gguf \\
  --tokenizer Qwen/Qwen3-0.6B

The --tokenizer flag is not decoration. The official docs recommend the base model's tokenizer because GGUF tokenizer conversion is "time-consuming and unstable, especially for some models with large vocabularies". Skip it and you trade a one-line flag for a long, flaky startup.

Two GPUs, one model: add --tensor-parallel-size 2 to shard the same GGUF across both cards — tensor parallelism works with GGUF the same as any other format.

Why does vLLM refuse GGUF on CPU?

This is the part that surprises people. GGUF's reputation is CPU-first — it is the format llama.cpp built its name on, running models on laptops and Raspberry Pis. Inside vLLM the situation inverts. The official quantization hardware compatibility table marks GGUF:

Hardware	GGUF in vLLM
NVIDIA Volta / Turing / Ampere / Ada / Hopper	supported
AMD GPU	supported
Intel GPU	not supported
x86 CPU	not supported
Arm CPU	not supported

Source: vLLM quantization docs. The reason is architectural: vLLM's GGUF path dequantizes blocks into GPU kernels built for batched serving. There is no CPU kernel behind it, because vLLM is a serving engine, not a laptop toy. If your machine has no GPU, no flag will help — use llama.cpp.

What breaks: the honest limits list

The same docs page carries a warning worth quoting verbatim: "GGUF support in vLLM is highly experimental and under-optimized." Concretely:

• Quant coverage is narrower than llama.cpp. The K-quants people actually download (Q4_K_M and friends) work; exotic schemes may not. llama.cpp remains the reference implementation for the format — we mapped the whole quant family in our GGUF quantization guide.
• Architecture support trails. New model families land in llama.cpp first; the plugin follows later.
• No mmap-style lazy loading. llama.cpp memory-maps the file; vLLM loads it like any other checkpoint.
• It is a memory-footprint feature first. The docs frame GGUF as a way to shrink VRAM use, not a throughput play.

None of this is hidden. It is all in the first paragraph of the official GGUF page — which is more than most experimental features get.

vLLM or llama.cpp for GGUF: which wins?

Different tools that happen to read the same file:

	vLLM + GGUF	llama.cpp
CPU inference	no	yes, first-class
Concurrent users	continuous batching, built for it	limited
Quant coverage	subset, experimental	the reference
Setup	vLLM + plugin	one binary
Best for	one GPU, many users	one user, any hardware

If you serve a model to a team from a single GPU, vLLM + GGUF lets you reuse the same Q4_K_M files the rest of the local-LLM world shares. For the full engine comparison, see llama.cpp vs Ollama and how to run GGUF models locally.

The one-line rule: same file, opposite instincts — llama.cpp treats GGUF as the format, vLLM treats it as an option.

---

More field notes at mrsaynothing.dev · code at GitHub · say hi: [e-mail]

This ships daily at mrsaynothing.dev — the full archive, every piece in 21 languages, zero missed days. New posts land in the newsletter the moment they ship: join it here. Code at GitHub.