Cross-posted from Mr Say Nothing — the receipts and figures live there.
An "8 GB" card holds 8 binary GiB — 8.59 decimal GB — and roughly 7.3 GiB of that survives the desktop and the driver. Models are sold in the other unit, decimal GB. Sizes measured today, KV math shown, one honest fit table: at 8k context qwen2.5:7b Q4_K_M is the roomiest fit, llama3.1:8b and qwen3:8b fit, and gemma2:9b — the tables' favourite — does not fit past 4k.
tl;dr
Best ollama model for 8GB VRAM: qwen2.5:7b (Q4_K_M) — about 5.4 GiB all-in at 8k context, the most headroom, and the only one of the five that still fits at 32k. llama3.1:8b (≈6.2 GiB) and qwen3:8b (≈6.4 GiB) fit at 8k. gemma2:9b needs ≈8.6 GiB at 8k and fails — 4k context only.
Total GiB at 8k — Q4_K_M weights + 8k KV cache + 0.6 GiB overhead floor, against 7.3 usable GiB (file sizes pulled live from the Hugging Face file API, 2026-11-01):
| model | total GiB @ 8k |
|---|---|
| qwen2.5:7b | 5.40 |
| mistral:7b | 5.67 |
| llama3.1:8b | 6.18 |
| qwen3:8b | 6.41 |
| gemma2:9b | 8.59 |
usable: 7.3 GiB — gemma2:9b is the bar that crosses.
What does 8GB VRAM actually hold?
An "8 GB" card never holds 8 decimal GB of model — it holds 8 binary GiB, and the desktop takes its cut before the model arrives. Marketing bought the "8" twice and gave it two different meanings; your out-of-memory killer only honours one of them.
The card's 8 GB is 8 × 2³⁰ bytes = 8.59 decimal GB. A GGUF file's 4.92 GB is 4.92 × 10⁹ bytes. Same word, two rulers, a fixed 7.4% gap between them — I've started calling it the seven-percent ghost, because it haunts every VRAM table on the internet and almost no table confesses which ruler it picked.
The second cut nobody budgets: the card is never empty. A Linux desktop with a compositor parks 0.3–0.5 GiB there, the CUDA/ROCm context takes ~0.3 more. Headless, you get ~7.6 GiB to play with; on a desktop, plan on 7.3 GiB usable. Every fit call in this piece is judged against that line.
Why do VRAM tables disagree by 7 percent?
The tables aren't rounding differently — they're mixing rulers, and at the margin the mix decides the answer.
Take gemma2:9b. The file lists 5.76 GB (decimal). A naive table compares 5.76 against "8" and prints 72% full — fine, ship it. The honest stack at 4k context: 5.37 GiB (file) + 1.31 GiB (KV) + 0.6 GiB (overhead floor) = 7.28 GiB against 7.3 usable. Same model, same card, same context: the table's 72% and reality's 99.7% are the same row.
The division is one line — GiB = GB ÷ 1.074 — and every fit verdict below shows its division.
Best ollama model for 8GB VRAM: the verdict table
At 8k context, four of the five community defaults fit — and the one that doesn't is the 9B the listicles keep recommending. File sizes measured today from the Hugging Face file API (bartowski GGUF repos, Q4_K_M — the ollama default quant for these families; mistral:7b ships q4_0 by default, within ~2% of the measured file):
curl -s https://huggingface.co/api/models/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/tree/main \
| jq '.[] | select(.path | test("Q4_K_M"))'
verify: Q4_K_M.gguf → 4,920,739,232 bytes, pulled 2026-11-01. Same one-liner ran for all five repos.
| model (ollama tag) | file, decimal GB | ÷ 1.074 = GiB | KV @ 8k | overhead floor | total GiB @ 8k | verdict |
|---|---|---|---|---|---|---|
| qwen2.5:7b | 4.68 | 4.36 | 0.44 | 0.6 | 5.40 | fits — roomiest; holds 32k (6.71) |
| mistral:7b | 4.37 | 4.07 | 1.00 | 0.6 | 5.67 | fits, roomy |
| llama3.1:8b | 4.92 | 4.58 | 1.00 | 0.6 | 6.18 | fits; 16k hairline (7.18); 32k fails (9.18) |
| qwen3:8b | 5.03 | 4.68 | 1.13 | 0.6 | 6.41 | fits — watch the thinking budget |
| gemma2:9b | 5.76 | 5.37 | 2.63 | 0.6 | 8.59 | fails 8k — 4k only (7.28, hairline) |
The winner is boring on purpose:
ollama pull qwen2.5:7b
That gets you the most usable context per GiB, and it is the only row here that still fits when you double the context to 32k (6.71 GiB). The fail row is the receipt that justifies the piece: gemma2:9b misses 8k by 1.29 GiB even with zero overhead budgeted — 5.37 + 2.63 = 8.0 against 7.3 usable. No quant of it fixes that; Q4_K_M is already the small one.
How much VRAM does the KV cache eat?
Context is the second model you're loading, and architecture decides the price — per token, gemma2:9b charges six times what qwen2.5:7b does.
The KV cache is the model's working memory for the conversation: 2 bytes × layers × kv-heads × head-dim, at fp16, per token. Qwen2.5-7B's slim 28×4×128 attention charges 56 KiB/token. Gemma-2-9B's 42×8×256 charges 336 KiB/token. That per-token rate, not the file size, is what kills long context on a small card.
KV cache per token, fp16 (2 bytes × layers × kv-heads × head-dim, from public architecture sheets):
| model | KiB/token |
|---|---|
| qwen2.5:7b | 56 |
| mistral:7b | 128 |
| llama3.1:8b | 128 |
| qwen3:8b | 144 |
| gemma2:9b | 336 |
gemma2:9b burns 6× qwen2.5's rate. Scaled into context:
| model | KV @ 4k | KV @ 8k | KV @ 16k | KV @ 32k |
|---|---|---|---|---|
| qwen2.5:7b (56 KiB/tok) | 0.22 GiB | 0.44 | 0.88 | 1.75 |
| mistral:7b / llama3.1:8b (128) | 0.50 | 1.00 | 2.00 | 4.00 |
| qwen3:8b (144) | 0.56 | 1.13 | 2.25 | 4.50 |
| gemma2:9b (336) | 1.31 | 2.63 | 5.25 | 10.50 |
One 2026-specific footnote: qwen3 is a thinking model. Its reasoning tokens are context tokens — a short answer that "thought" for 3,000 tokens pays for those 3,000 at 144 KiB each until the context resets. On a 7.3 GiB card the thinking budget is not a vibe setting; it is the KV table above, compounding.
The arithmetic, line by line
- Usable line: 8 GiB − 0.4 (desktop, compositor + display) − 0.3 (CUDA context) ≈ 7.3 GiB; headless recovers ~0.3 more.
- Overhead floor 0.6 = CUDA context ~0.3 + compute buffer ~0.3 at 8k; the honest range is 0.4–1.2 GiB depending on backend and context — the gemma2 fail verdict holds across the entire range, which is why the floor is enough.
- KV rates: llama3.1-8B 32L×8kv×128d; qwen2.5-7B 28×4×128; qwen3-8B 36×8×128; gemma2-9B 42×8×256; mistral-7B-v0.3 32×8×128 — all ×2 bytes ×2 (K and V).
- GiB conversions: file bytes ÷ 2³⁰ (4,920,739,232 → 4.58; 4,683,074,240 → 4.36; 4,372,812,000 → 4.07; 5,027,784,224 → 4.68; 5,761,057,728 → 5.37).
- gemma2 honesty note: half its layers are sliding-window, so llama.cpp's real KV is smaller than this conservative full-attention figure — the fail verdict does not need the discount to hold.
last tested 2026-11-01.
What this table cannot tell you
Every number above is arithmetic on public sizes — and the two numbers the table still guesses are overhead and MoE, which is exactly where the next measurement lives.
The overhead floor is a range wearing a uniform; any calculator that prints one overhead number is guessing, including our own GGUF VRAM calculator piece — its wound, the one that refuses MoE math, is documented there. And the fit line moves with your desktop: boot headless, recover ~0.3 GiB, and gemma2:9b at 4k goes from hairline to merely tight.
What would settle it: the same five models through ollama on a real 8 GB card, nvidia-smi numbers printed against this table's predictions. That bench is not this piece — the card on this desk has no business pretending to be one.
Top comments (3)
Filed under rows that earn their filename: gemma2:9b misses 8k by 1.29 GiB even with zero overhead budgeted — 5.37 of weights plus 2.63 of KV against a 7.3 usable line. The one figure I half-distrust is my own gemma2 KV: half its layers are sliding-window, so llama.cpp is quietly paying less than my conservative 336 KiB/token suggests. Has anyone watched nvidia-smi on a real gemma2:9b load and seen the gap, or is the full-attention number still the honest one to budget with?
The table survives. Two things in front of it don't.
First, the sentence: "gemma2:9b misses 8k by 1.29 GiB even with zero overhead budgeted, 5.37 of weights plus 2.63 of KV against a 7.3 usable line." 8.00 minus 7.30 is 0.70. The 1.29 is 8.59 (floor included) minus 7.30. Both numbers are correct, but only one of them is zero overhead. The fail holds at 0.70, so use 0.70 and the claim stays clean.
Second, the headline number nearly cancels itself in your own example. Naive table: 5.76 GB against 8 GB, 72%. Now fix the ruler but keep the KV column missing: 5.76 against the honest 8.59 decimal GB is 67%, and 5.37 GiB against 7.3 GiB is 73.6%. So the ruler moves the answer between 67 and 73.6, and the naive 72 sits inside that band. The 27.7 points between 72% and 99.7% are the column that table never printed: 1.31 GiB of KV at 4k plus the 0.6 floor.
Which means the seven percent explains almost none of the wrong fits. A table can hold both rulers correctly and still ship the 9B, and the tables that fail loudly fail because KV is absent, not because GB and GiB got confused. The mixed ruler makes the number unquotable. The missing column makes it wrong.
The row I'd keep is still gemma2 at 8k: it fails at 0.70 even with an empty budget, and it survives every overhead in your 0.4 to 1.2 range, while llama3.1 at 16k does not. Your hairline label on that one is honest.
Riccardo
7 percent is exactly the margin that flips a fit into an OOM at load time.