Thurmon Demich

Posted on Aug 5 • Originally published at bestgpuforllm.com

Local LLM VRAM 2026: The 12GB Trap Most Buyers Hit

#vram #llm #inference #quantization

Cross-posted from Best GPU for LLM — visit the original for our VRAM calculator, GPU comparison table, and current Amazon pricing.

Quick answer: For most local LLM users, 16GB VRAM is the sweet spot. It handles all 7B models and most 13B models quantized. If you want to run 34B+ models, you need 24GB or more.

VRAM chart available at the original article

The VRAM rule of thumb

A rough formula for estimating VRAM needs:

VRAM needed = (Parameters in billions x Bytes per parameter) + 2GB overhead

At FP16 (2 bytes per param): a 7B model needs ~16GB. But with quantization, you can cut that dramatically.

You can sanity-check any model-and-quant combo right here:

VRAM requirements by model size

7B parameter models (Llama 3 8B, Mistral 7B, Gemma 7B)

Quantization	Model Size	VRAM Needed	Minimum GPU
FP16	~14GB	16GB	RTX 4060 Ti 16GB
Q8	~7.5GB	10GB	RTX 3060 12GB
Q6_K	~5.5GB	8GB	RTX 4060 8GB
Q4_K_M	~4.5GB	6GB	RTX 3060 12GB
Q3_K_M	~3.5GB	6GB	Any 6GB GPU

Verdict: Any modern GPU with 8GB+ VRAM handles 7B models comfortably. 16GB gives you room for full precision and longer contexts. For Llama 3 8B specifically, see how much VRAM Llama 3 8B actually uses across all quantization levels.

13B parameter models (Llama 2 13B, CodeLlama 13B)

Quantization	Model Size	VRAM Needed	Minimum GPU
FP16	~26GB	28GB	RTX 5090 32GB
Q8	~13GB	16GB	RTX 4060 Ti 16GB
Q6_K	~10GB	12GB	RTX 3060 12GB
Q4_K_M	~7.5GB	10GB	RTX 3060 12GB
Q3_K_M	~6GB	8GB	RTX 4060 8GB

Verdict: 12-16GB VRAM is ideal for 13B models. The RTX 4060 Ti 16GB or RTX 4070 handles these well.

34B parameter models (CodeLlama 34B, Yi 34B)

Quantization	Model Size	VRAM Needed	Minimum GPU
FP16	~68GB	70GB+	Multi-GPU required
Q8	~34GB	36GB+	A6000 48GB
Q6_K	~26GB	28GB	RTX 5090 32GB
Q4_K_M	~20GB	22GB	RTX 4090 24GB
Q3_K_M	~15GB	18GB	RTX 4070 Ti Super 16GB (tight)

Verdict: 24GB is the minimum for comfortable 34B use. The RTX 4090 is the go-to card for this tier. For a detailed test of whether the 12GB RTX 5070 can handle 34B at all, see can the RTX 5070 run 34B?

70B parameter models (Llama 3 70B, Qwen 72B)

Quantization	Model Size	VRAM Needed	Minimum GPU
Q8	~70GB	72GB+	Multi-GPU required
Q6_K	~54GB	56GB+	Multi-GPU required
Q4_K_M	~40GB	42GB+	2x RTX 4090 or A6000
Q3_K_M	~32GB	34GB	RTX 5090 (tight)
Q2_K	~25GB	28GB	RTX 5090 32GB

Verdict: 70B models are not practical on a single consumer GPU at good quality. Plan for dual GPUs, a workstation card, or cloud. For a specific look at what a 16GB mid-range card can do, see can the RTX 4060 Ti run Llama 70B? — and for the complete VRAM breakdown at every quantization level, see how much VRAM for a 70B model. For 1M-context MoE models like MiniMax M3 specifically, KV cache dominates the VRAM math — see our best GPU for MiniMax M3 breakdown.

What is quantization?

Quantization reduces the precision of model weights to use less memory. The trade-off is quality:

Level	Bits	Quality	Use Case
FP16	16-bit	Best	If VRAM allows
Q8	8-bit	Near-perfect	Default if you have room
Q6_K	6-bit	Excellent	Best quality/size balance
Q5_K_M	5-bit	Very good	Slight degradation
Q4_K_M	4-bit	Good	Most popular for daily use
Q3_K_M	3-bit	Acceptable	Noticeable on complex reasoning
Q2_K	2-bit	Poor	Emergency only

The sweet spot is Q4_K_M to Q6_K. Below Q4, quality degrades noticeably. Above Q6, the quality gains are minimal for most use cases. For a deeper dive into how quantization choices affect real-world output, see our best quantization for local LLM guide.

Context length eats VRAM too

VRAM usage isn't just the model. The KV cache for context grows with conversation length:

Context Length	Additional VRAM (7B)	Additional VRAM (13B)
2,048 tokens	~0.5GB	~1GB
4,096 tokens	~1GB	~2GB
8,192 tokens	~2GB	~4GB
16,384 tokens	~4GB	~8GB
32,768 tokens	~8GB	~16GB

This is why a model that "fits" in VRAM can crash during a long conversation. Always leave 2-4GB of headroom beyond the base model size.

GPU recommendations by budget

VRAM Target	Best GPU	Price	Handles
8GB	RTX 4060	~$300	7B quantized
12GB	RTX 3060 12GB (used)	~$250	7B full, 13B quantized
16GB	RTX 4060 Ti 16GB	~$400	7B-13B comfortably
24GB	RTX 4090	~$1,600	Up to 34B quantized
32GB	RTX 5090	~$2,000+	34B comfortable, 70B tight
48GB	2x RTX 4090 or A6000	~$3,200+	70B at good quality

Which GPU should YOU buy?

Running 7B models only? Get an RTX 3060 12GB ($250 used) or RTX 4060 Ti 16GB ($400) for more headroom.
Running 13B-34B models? Get an RTX 4090 ($1,600). Nothing else in consumer space gives you 34B at usable quantization on a single card.
Future-proofing for 34B+ and experimental 70B? Get an RTX 5090 ($2,000). The only single consumer GPU that handles it.
Need 70B at good quality? Get 2x RTX 4090 ($3,200) or try cloud GPUs for occasional use.

Common mistakes

"I'll just use CPU offloading" — Offloading layers to RAM works, but inference speed drops 5-10x for offloaded layers. A model that's 50% on GPU and 50% offloaded is painfully slow. If your primary use case is local LLM summarization, long-document context is your binding constraint — plan VRAM for the KV cache first, not just the model weights.

"8GB is enough for everything" — It's enough for 7B quantized. That's it. If you plan to explore different models, get at least 12-16GB.

"I'll buy the cheapest card with enough VRAM" — Memory bandwidth matters too. An old card with 16GB VRAM but low bandwidth will have much slower inference than a newer card with the same VRAM.

"More VRAM is always better" — True, but only up to what you need. Don't buy a $3,500 A6000 if you only run 7B models. Wondering if a Mac Mini's unified memory counts the same way? See our can the Mac Mini run LLMs guide for how Apple's shared memory pool changes the VRAM math. For model-family specific breakdowns, see how much VRAM Qwen 3 needs if Qwen is your target, or how much VRAM Gemma 4 needs for Google's latest.

Final verdict

You want to run...	Get this GPU
7B models (Mistral, Llama 8B)	RTX 4060 Ti 16GB ($400)
13B models	RTX 4070 Ti Super 16GB ($700)
34B models	RTX 4090 24GB ($1,600)
70B models	2x RTX 4090 or cloud GPU

16GB is the new minimum for anyone serious about local LLM inference. The landscape has shifted — 8GB cards are increasingly limiting as models grow.

Buy for the model size you want to run, not the model size you're running today. VRAM is the one thing you can't upgrade later.

Frequently Asked Questions

What LLMs can I run with 8GB VRAM?

With 8GB VRAM, you can run all 7B parameter models (Llama 3 8B, Mistral 7B, Gemma 7B) at Q4_K_M to Q6_K quantization. You can also fit some 7B models at Q8 with tight headroom. 13B models will not fit at any usable quantization level. Budget 2GB for context window overhead, so plan for models under 6GB in size.

What LLMs can I run with 16GB VRAM?

16GB VRAM handles all 7B models at full FP16 precision and all 13B models at Q4_K_M to Q8 quantization. Popular models like CodeLlama 13B, Qwen 14B, and Phi-3 Medium 14B all fit comfortably. You can also run 34B models at Q3_K_M in a pinch, though quality degrades noticeably at that quantization level. 16GB is the sweet spot for most local LLM users in 2026.

How much VRAM does Llama 4 need?

Llama 4 Scout (17B active parameters, 109B total in a mixture-of-experts architecture) is still early in community testing, but early reports suggest you should expect to need roughly 12-20GB VRAM at Q4 quantization depending on context length and runtime. Llama 4 Maverick (17B active, 400B total) requires significantly more — likely in the 40-50GB range at Q4, meaning dual RTX 4090s or a workstation-class GPU. These figures may shift as quantization support matures.

Does quantization reduce VRAM usage?

Yes, dramatically. Quantization compresses model weights from 16-bit (FP16) down to 4-bit or lower, reducing VRAM usage by 2-4x. A 7B model uses about 14GB at FP16 but only 4.5GB at Q4_K_M. The quality trade-off is minimal between Q4_K_M and Q8 for most use cases — below Q4, quality degrades noticeably on complex reasoning tasks.

Related guides on Best GPU for LLM

Read the full guide on Best GPU for LLM — includes our VRAM calculator, GPU comparison table, and live pricing.

DEV Community