Quantize or Offload? A 16GB GPU Local LLM Strategy

🎯 The Setup

I've got a single RTX 4080 (16GB VRAM), 64GB system RAM, and a stubborn desire to run 13B-ish open models without renting a cloud GPU. The two usual answers are:

1. Quantize the model so the whole thing fits in VRAM at lower precision.
2. Offload layers to CPU/RAM and keep weights in full or near-full precision, paying a PCIe tax for it.

Every "how to run LLMs locally" thread eventually turns into a religious war between these two camps, usually without numbers. So I ran both strategies against the same models on the same hardware and actually measured throughput and memory behavior instead of vibing.

🔧 The Problem

16GB sits in an annoying spot. It's enough to comfortably run a 7B model in fp16 (~14GB) with a small KV cache, but a 13B model in fp16 needs ~26GB — nowhere close. You have three real options:

• 4-bit or 8-bit quantize the 13B model so it fits entirely on the GPU
• Keep it in fp16/bf16 and offload the layers that don't fit to system RAM
• Just use a smaller model (boring, but sometimes correct — more on that later)

I tested with Llama-3-8B and Qwen2.5-14B as representative "fits" and "doesn't fit" cases, using llama.cpp for quantized GGUF runs and transformers + accelerate for offloaded fp16 runs.

🧪 Test Methodology

Same prompt set (512 token input, 256 token generation, batch size 1), same GPU, measured with nvidia-smi --query-gpu=memory.used --loop-ms=500 running in the background and time around generation.

Quantized runs:

bash

Convert + quantize with llama.cpp

python convert_hf_to_gguf.py ./Qwen2.5-14B-Instruct --outfile qwen14b-f16.gguf

./llama-quantize qwen14b-f16.gguf qwen14b-q4_k_m.gguf Q4_K_M
./llama-quantize qwen14b-f16.gguf qwen14b-q8_0.gguf Q8_0

./llama-bench -m qwen14b-q4_k_m.gguf -p 512 -n 256 -ngl 99
./llama-bench -m qwen14b-q8_0.gguf -p 512 -n 256 -ngl 99

Offloaded runs:

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch, time

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-14B-Instruct",
torch_dtype=torch.float16,
device_map="auto", # accelerate decides GPU vs CPU per layer
max_memory={0: "14GiB", "cpu": "48GiB"},
)

prompt = tok(open("prompt.txt").read(), return_tensors="pt").to("cuda")

start = time.time()
out = model.generate(**prompt, max_new_tokens=256)
elapsed = time.time() - start
print(f"{256 / elapsed:.2f} tok/s")

📊 Quantization Results

Model	Format	VRAM used	tok/s (gen)	Notes
Llama-3-8B	fp16 (no quant needed)	15.1 GB	38.2	baseline, fits natively
Qwen2.5-14B	Q8_0	15.8 GB	21.4	borderline fit, tight KV cache room
Qwen2.5-14B	Q4_K_M	9.3 GB	34.7	plenty of headroom for longer context

The Q4_K_M run is the standout here: it's not just that it fits, it's that the entire model and KV cache live on the GPU with room to spare, so there's zero PCIe round-tripping per token. Quality loss versus fp16 on MMLU-style spot checks was small (roughly 1-2 points), and basically imperceptible on conversational tasks.

Q8_0 technically fits but only barely — push context past ~3K tokens and you start getting CUDA OOM or llama.cpp silently falling back to partial CPU layers, which defeats the purpose.

📊 Offloading Results

Model	Split	VRAM used	tok/s (gen)	Notes
Qwen2.5-14B fp16	~60% GPU / 40% CPU	14.9 GB	4.1	heavy PCIe traffic every token
Qwen2.5-14B fp16	~80% GPU / 20% CPU	14.9 GB	7.6	fewer offloaded layers, still bottlenecked

The nvidia-smi memory trace for the offloaded run is telling: GPU memory usage is flat and near the max_memory cap the whole time, while GPU utilization oscillates between ~90% and ~15% in a sawtooth pattern — that's the model waiting on CPU-computed layers to come back over PCIe before it can continue. Full precision is preserved, but you're paying roughly a 5-8x throughput penalty to get it.

For comparison, that's the difference between a usable chat loop and something you start and go make coffee for.

⚖️ Trade-offs

This isn't close, but it's worth being honest about why:

• Quantization cost is paid once, offline. You quantize the weights, you're done — inference has no extra runtime overhead beyond slightly cheaper matmuls.
• Offloading cost is paid every single forward pass. PCIe 4.0 x16 gives you ~32 GB/s in theory, far less in practice with the overhead of partial-layer compute on CPU, which is why throughput craters.
• Quantization does touch quality. Q4_K_M is a genuinely lossy transform. For tasks sensitive to exact numeric reasoning or long-tail factual recall, run your own eval before trusting benchmarks from a different model family.
• Offloading wins only when quantization isn't an option at all — e.g., you need the full fp16 model for a specific reproducibility requirement, or you're running something where even 4-bit doesn't fit and you have no smaller checkpoint available.
• KV cache headroom matters more than people expect. A model that "fits" at 512 tokens of context can OOM at 4K. Always benchmark at the context length you'll actually use, not the default in whatever script you copied.

🏆 Recommendation

For a 16GB consumer card, default to 4-bit quantization (Q4_K_M or equivalent AWQ/GPTQ) for anything in the 7B-14B range. It's faster, it leaves context headroom, and the quality hit is small enough to not matter for most real workloads. Reach for CPU offloading only as a fallback when you need a model that won't fit even at 4-bit and no smaller variant exists — and go in expecting single-digit tokens/sec.

The one scenario where I'd flip this: if you're doing quantitative work where output precision genuinely matters (numeric reasoning benchmarks, exact reproduction of a published fp16 result), the throughput loss from offloading is the lesser evil.

What's your actual tok/s been with similar hardware — and has anyone found an 8-bit sweet spot that didn't just turn into a worse version of 4-bit? Curious whether these numbers hold up on 3090s/4090s with more VRAM headroom to play with.