You install Ollama, pull a 7B model, run it, and get 1.8 tokens/sec. Fans barely spin. nvidia-smi shows almost no VRAM used. Nothing errors out.
That's the most common local-LLM failure mode, and it isn't a broken install. Ollama decided the model didn't fit and quietly moved layers to the CPU. It rarely tells you.
Here's how to confirm it, fix it, and measure the difference.
1. Confirm the fallback
ollama ps
The PROCESSOR column is the whole story:
NAME ID SIZE PROCESSOR UNTIL
qwen2.5-coder:7b dae161e27b0e 5.2 GB 100% CPU 4 minutes from now
100% CPU on a machine with a working GPU means the model didn't fit in VRAM at the context size Ollama asked for. Not "the GPU is too weak" — it means the allocation failed and Ollama degraded instead of refusing.
Get a baseline before changing anything:
ollama run qwen2.5-coder:7b --verbose "write a python function to debounce a callback"
The final line prints eval rate, your real tokens/sec. Write it down.
2. The real culprit is usually the KV cache, not the weights
A 7B model at Q4_K_M is roughly 4.5–5 GB of weights. On an 8 GB card that leaves ~3 GB minus whatever the desktop compositor is using — call it 2.5 GB of usable headroom.
The KV cache is what eats it. Per token, per layer:
bytes = 2 (K and V) x layers x kv_heads x head_dim x bytes_per_element
For Llama 3.1 8B (32 layers, 8 KV heads, head_dim 128) at f16 that's 128 KiB per token. Run the multiplication:
| Context | KV cache (f16) |
|---|---|
| 4,096 | ~512 MB |
| 8,192 | ~1 GB |
| 32,768 | ~4 GB |
| 131,072 (model default) | ~16 GB |
The model file says it supports 128k context. Ollama tries to honor that. Your 8 GB card cannot. Layers spill to CPU, and you get the 1.8 tok/s.
This is why "it works fine on my friend's 16 GB Mac" — unified memory plus a larger budget changes the answer completely.
3. The four settings that actually fix it
Set these before starting the server, not after.
Linux / macOS
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0
export OLLAMA_NUM_PARALLEL=1
ollama serve
Windows (PowerShell, persistent)
setx OLLAMA_FLASH_ATTENTION 1
setx OLLAMA_KV_CACHE_TYPE q8_0
setx OLLAMA_NUM_PARALLEL 1
Then quit Ollama from the tray and restart it — setx only affects new processes.
What each one buys you:
-
OLLAMA_FLASH_ATTENTION=1— uses a fused attention kernel with a smaller memory footprint. Free win on any modern GPU. -
OLLAMA_KV_CACHE_TYPE=q8_0— quantizes the KV cache to 8-bit, roughly halving the table above. At 4k context that's ~256 MB instead of ~512 MB. Quality impact at q8 is negligible in practice; q4 KV cache is where you start noticing degradation. -
OLLAMA_NUM_PARALLEL=1— this is the one nobody mentions. Ollama defaults to serving multiple requests concurrently, and each parallel slot gets its own KV cache. On a single-user laptop that's pure waste. Setting it to 1 can free gigabytes. -
num_ctx— set it explicitly, never inherit the model's declared maximum:
ollama run qwen2.5-coder:7b
>>> /set parameter num_ctx 4096
>>> /save qwen-coder-4k
Or bake it into a Modelfile:
FROM qwen2.5-coder:7b
PARAMETER num_ctx 4096
PARAMETER num_batch 512
PARAMETER temperature 0.2
ollama create qwen-coder-4k -f Modelfile
4. Pick a model that fits the budget, not the benchmark
Rule of thumb for a 4-bit quant:
weights ~= params_in_billions x 0.6 GB
+ KV cache at your chosen context
+ ~0.8 GB for the display/compositor
< total VRAM
| VRAM | Realistic ceiling | Example |
|---|---|---|
| 4 GB | 3B Q4, 4k ctx | llama3.2:3b |
| 6 GB | 7B Q4, 4k ctx |
qwen2.5-coder:7b at num_ctx 4096
|
| 8 GB | 7–8B Q4, 8k ctx |
llama3.1:8b with q8_0 KV cache |
| 12 GB | 14B Q4, 8k ctx | qwen2.5-coder:14b |
| 16 GB | 14B Q5 or 32B Q3 | trade quality for size deliberately |
A 3B model that runs at 40 tok/s on the GPU beats a 14B model that runs at 1.5 tok/s on the CPU for almost every coding loop, because you iterate more times per minute.
5. Verify the fix
ollama ps
You want 100% GPU. If it says 83%/17% CPU, you're close — drop num_ctx by half and check again.
Then re-run the same prompt from step 1 and compare eval rate. On a mobile 8 GB GPU, going from CPU fallback to full offload typically moves 7B models from ~2 tok/s to ~25–40 tok/s. That's the difference between unusable and genuinely useful.
Why this is worth ten minutes
The alternative is $20/month forever for an autocomplete box. A local 7B coder model at 30 tok/s covers most inline-completion and small-refactor work, and it costs nothing after the hardware you already own.
If you'd rather not work through the arithmetic for your specific machine, I do a $1 version: send your CPU, GPU + VRAM, RAM, and the model you want, and you get back a Modelfile plus the exact environment flags and a KV-cache headroom calculation for your hardware.
- Custom tuning for your exact laptop ($1): https://ko-fi.com/seanvishwakarma/commissions
- Pre-built Modelfiles for 8 GB / 16 GB / Apple Silicon ($1): https://ko-fi.com/s/124024c020
- Longer write-up with the full benchmark tables: https://thezerodollardev.blogspot.com/2026/11/the-zero-dollar-local-ai-guide-how-to.html
No subscription, no account, no telemetry.
Top comments (0)