DEV Community

Aditya Vishwakarma
Aditya Vishwakarma

Posted on Originally published at thezerodollardev.blogspot.com

Your 8GB GPU isn't the problem: 4 Ollama settings that stop the silent CPU fallback

You install Ollama, pull a 7B model, run it, and get 1.8 tokens/sec. Fans barely spin. nvidia-smi shows almost no VRAM used. Nothing errors out.

That's the most common local-LLM failure mode, and it isn't a broken install. Ollama decided the model didn't fit and quietly moved layers to the CPU. It rarely tells you.

Here's how to confirm it, fix it, and measure the difference.


1. Confirm the fallback

ollama ps
Enter fullscreen mode Exit fullscreen mode

The PROCESSOR column is the whole story:

NAME              ID              SIZE      PROCESSOR    UNTIL
qwen2.5-coder:7b  dae161e27b0e    5.2 GB    100% CPU     4 minutes from now
Enter fullscreen mode Exit fullscreen mode

100% CPU on a machine with a working GPU means the model didn't fit in VRAM at the context size Ollama asked for. Not "the GPU is too weak" — it means the allocation failed and Ollama degraded instead of refusing.

Get a baseline before changing anything:

ollama run qwen2.5-coder:7b --verbose "write a python function to debounce a callback"
Enter fullscreen mode Exit fullscreen mode

The final line prints eval rate, your real tokens/sec. Write it down.


2. The real culprit is usually the KV cache, not the weights

A 7B model at Q4_K_M is roughly 4.5–5 GB of weights. On an 8 GB card that leaves ~3 GB minus whatever the desktop compositor is using — call it 2.5 GB of usable headroom.

The KV cache is what eats it. Per token, per layer:

bytes = 2 (K and V) x layers x kv_heads x head_dim x bytes_per_element
Enter fullscreen mode Exit fullscreen mode

For Llama 3.1 8B (32 layers, 8 KV heads, head_dim 128) at f16 that's 128 KiB per token. Run the multiplication:

Context KV cache (f16)
4,096 ~512 MB
8,192 ~1 GB
32,768 ~4 GB
131,072 (model default) ~16 GB

The model file says it supports 128k context. Ollama tries to honor that. Your 8 GB card cannot. Layers spill to CPU, and you get the 1.8 tok/s.

This is why "it works fine on my friend's 16 GB Mac" — unified memory plus a larger budget changes the answer completely.


3. The four settings that actually fix it

Set these before starting the server, not after.

Linux / macOS

export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0
export OLLAMA_NUM_PARALLEL=1
ollama serve
Enter fullscreen mode Exit fullscreen mode

Windows (PowerShell, persistent)

setx OLLAMA_FLASH_ATTENTION 1
setx OLLAMA_KV_CACHE_TYPE q8_0
setx OLLAMA_NUM_PARALLEL 1
Enter fullscreen mode Exit fullscreen mode

Then quit Ollama from the tray and restart it — setx only affects new processes.

What each one buys you:

  1. OLLAMA_FLASH_ATTENTION=1 — uses a fused attention kernel with a smaller memory footprint. Free win on any modern GPU.
  2. OLLAMA_KV_CACHE_TYPE=q8_0 — quantizes the KV cache to 8-bit, roughly halving the table above. At 4k context that's ~256 MB instead of ~512 MB. Quality impact at q8 is negligible in practice; q4 KV cache is where you start noticing degradation.
  3. OLLAMA_NUM_PARALLEL=1 — this is the one nobody mentions. Ollama defaults to serving multiple requests concurrently, and each parallel slot gets its own KV cache. On a single-user laptop that's pure waste. Setting it to 1 can free gigabytes.
  4. num_ctx — set it explicitly, never inherit the model's declared maximum:
ollama run qwen2.5-coder:7b
>>> /set parameter num_ctx 4096
>>> /save qwen-coder-4k
Enter fullscreen mode Exit fullscreen mode

Or bake it into a Modelfile:

FROM qwen2.5-coder:7b
PARAMETER num_ctx 4096
PARAMETER num_batch 512
PARAMETER temperature 0.2
Enter fullscreen mode Exit fullscreen mode
ollama create qwen-coder-4k -f Modelfile
Enter fullscreen mode Exit fullscreen mode

4. Pick a model that fits the budget, not the benchmark

Rule of thumb for a 4-bit quant:

weights ~= params_in_billions x 0.6 GB
+ KV cache at your chosen context
+ ~0.8 GB for the display/compositor
< total VRAM
Enter fullscreen mode Exit fullscreen mode
VRAM Realistic ceiling Example
4 GB 3B Q4, 4k ctx llama3.2:3b
6 GB 7B Q4, 4k ctx qwen2.5-coder:7b at num_ctx 4096
8 GB 7–8B Q4, 8k ctx llama3.1:8b with q8_0 KV cache
12 GB 14B Q4, 8k ctx qwen2.5-coder:14b
16 GB 14B Q5 or 32B Q3 trade quality for size deliberately

A 3B model that runs at 40 tok/s on the GPU beats a 14B model that runs at 1.5 tok/s on the CPU for almost every coding loop, because you iterate more times per minute.


5. Verify the fix

ollama ps
Enter fullscreen mode Exit fullscreen mode

You want 100% GPU. If it says 83%/17% CPU, you're close — drop num_ctx by half and check again.

Then re-run the same prompt from step 1 and compare eval rate. On a mobile 8 GB GPU, going from CPU fallback to full offload typically moves 7B models from ~2 tok/s to ~25–40 tok/s. That's the difference between unusable and genuinely useful.


Why this is worth ten minutes

The alternative is $20/month forever for an autocomplete box. A local 7B coder model at 30 tok/s covers most inline-completion and small-refactor work, and it costs nothing after the hardware you already own.

If you'd rather not work through the arithmetic for your specific machine, I do a $1 version: send your CPU, GPU + VRAM, RAM, and the model you want, and you get back a Modelfile plus the exact environment flags and a KV-cache headroom calculation for your hardware.

No subscription, no account, no telemetry.

Top comments (0)