How Much VRAM You Actually Need for a Local LLM (2026)

Every week on the local-LLM subreddits there's the same thread: "I have an 8GB card. What can I actually run?" And the answers are all guesses. I thought I'd write the version I wish I'd had before I bought mine, because I got this wrong the first time and it cost me a long night of frustration.

Here's the thing about how much VRAM you need for a local LLM: it's not about the model name. It's about two numbers you can compute on a napkin.

The whole trick is quantization. A model is just a pile of weights, and those weights are usually stored as 16-bit floats. At that precision, a 7B model needs around 14GB of VRAM. But you almost never run full precision at home. Quantized models compress those weights into 4 or 8 bits each, which is where everything gets practical.

The arithmetic people actually use:

VRAM in GB ≈ (billions of parameters) × (bytes per weight) + overhead

Q4_K_M: 0.5 bytes/param  →  7B needs ~4-5 GB
Q8_0:   1.0 bytes/param  →  7B needs ~8 GB
F16:    2.0 bytes/param  →  7B needs ~14 GB

Don't glaze over the overhead part. A model that's exactly the size of your VRAM won't load, because the context window eats memory too. Add a couple of gigabytes for that. This is why people say "it fits" and then it OOMs.

So what's realistic per card?

• 8GB — Q4 model up to about 7B-8B. That's Llama 3.1 8B, Qwen 2.5 7B, Mistral 7B. Fine for chat and writing. This is the floor where local LLMs stop feeling like a hobby.
• 12GB — comfortable home for a 13B, and the sweet spot most hobbyists end up at. You stop fighting OOM errors and start actually editing documents.
• 16GB — most 7B/13B quantized models fit easily, some 24-34B at heavy quant work. Genuinely pleasant.
• 24GB+ — this is where the big 30B+ models and coding beasts live. This is enthusiast territory.

A 70B model needs something like 40GB at Q4. No single consumer card does that well. Stop planning around it unless you're serious about dual-GPU builds.

One opinion so you don't chase the wrong thing: Q4_K_M is the default for almost everyone. The quality gap between Q4 and Q8 is roughly 1-3% on benchmarks — imperceptible for chat and writing. The gap that actually matters is between Q4 and Q3, which is a real cliff. So unless you're doing math-heavy reasoning or code where precision counts, start at Q4_K_M and don't upgrade until you've hit a specific problem.

The last worth-nothing rule: if a model doesn't fit, my first move isn't a smaller model. It's a smaller quantization. Dropping a 14GB F16 down to Q4_K_M keeps the same brain at a quarter of the memory. That single habit solved more "it won't run" moments than any GPU purchase.

I've been building this exact setup — LM Studio, Open WebUI, and a stack of quantized models in Docker — for a while now, and I packed the whole working thing into a pay-what-you-want bundle so you don't have to reassemble it from scraps. It's the home lab folder I'd have wanted on day one: Self-Hosted AI Home Lab.

FAQ

Do I really need a GPU at all for a local LLM?

No. You can run a 7B at Q4_K_M on CPU — a modern 8-core with fast RAM gets you 2-8 tokens a second. Usable for light stuff, too slow for serious work. A GPU makes it feel like a product.

Is Q8 noticeably better than Q4?

For everyday chat and writing, no — the gap is around 1-3% on benchmarks. For coding and multi-step math it's worth the extra VRAM. When in doubt, start at Q4_K_M.

What's the minimum VRAM for a local LLM in 2026?

8GB is the practical floor — it runs 7B-8B models at Q4 comfortably. 12GB is the comfortable middle where things stop being a struggle. Anything under 8GB means you're stuck with tiny models.