DEV Community

Aditya Vishwakarma
Aditya Vishwakarma

Posted on

Your 8GB GPU Isn't the Problem: 4 Ollama Settings That Stop the Silent CPU Fallback

The 2 tokens/sec trap that tricks everyone into upgrading hardware

You pull a 7B or 8B model (Qwen 2.5 Coder, Llama 3.3, or DeepSeek-R1 Distill) on a machine with an 8GB RTX GPU or an M-series Mac. For the first two prompts, it screams along at 35 tokens per second. Then, on prompt three, it slows to a crawl: 2.1 tokens per second, CPU fan pinned at maximum, laptop burning hot.

Most developers assume: "My 8GB GPU is too small. I need a $1,500 RTX 4090 or a Mac Studio with 64GB unified memory."

It is almost never the GPU. In 9 out of 10 cases, your model was running 100% in VRAM until the conversation history passed a specific token boundary. At that exact moment, the unquantized KV-cache exceeded remaining VRAM by 150 megabytes, forcing Ollama to silently split layers and dump execution onto the system CPU.


The Memory Math: Where your VRAM actually goes

Total memory consumption in Ollama or llama.cpp is not just the model weights. It follows this exact formula:

Total VRAM Required = Model Weights + KV Cache Size + CUDA Context Overhead
Enter fullscreen mode Exit fullscreen mode

For an 8-billion parameter model quantized at Q4_K_M:

  • Model weights: ~4.7 GB
  • CUDA runtime overhead: ~450 MB
  • Available VRAM left on an 8GB card: ~2.8 GB

By default, Ollama initializes the KV cache in 16-bit unquantized float (FP16). At an 8,192 token context window, FP16 KV-cache requires 2.6 GB. Add a 12,000 token system prompt or a couple of web search results, and your cache demands 3.8 GB. 4.7 GB + 3.8 GB = 8.5 GB. VRAM overflow. Instant fallback to 2 tok/s.


The 4 Settings That Fix It Today

1. Enable Quantized KV Cache (q8_0)

By default, Ollama caches key-value pairs in FP16. Setting the cache to 8-bit cuts the KV cache footprint by 50% with zero perceptible degradation in coding or reasoning benchmarks.

export OLLAMA_KV_CACHE_TYPE=q8_0
Enter fullscreen mode Exit fullscreen mode

On an 8k context window, this single environment variable shrinks your memory footprint from 2.6 GB down to 1.3 GB, leaving ample headroom on 8GB cards.

2. Enable Flash Attention

Flash Attention calculates attention with linear memory complexity rather than quadratic memory spikes during long prompt evaluations. It keeps memory usage flat as conversations grow.

export OLLAMA_FLASH_ATTENTION=1
Enter fullscreen mode Exit fullscreen mode

3. Cap the Context Window in Your Modelfile

Many modern models declare a 32k or 128k maximum context window in their GGUF metadata. If your tool doesn't specify a context limit, Ollama may preallocate or scale context beyond your physical memory limits.

Create a custom Modelfile and fix the context window to what your VRAM can mathematically guarantee:

FROM qwen2.5-coder:7b-instruct-q4_K_M
PARAMETER num_ctx 4096
PARAMETER temperature 0.2
PARAMETER repeat_penalty 1.15
Enter fullscreen mode Exit fullscreen mode

For 95% of programming and reasoning tasks, 4,096 tokens is more than enough for a full function, error log, or code review pass.

4. Force Single-Thread Concurrency

Ollama defaults to handling multiple requests in parallel if requested by client applications. When two requests hit at once, Ollama clones the context space, instantly doubling VRAM usage.

export OLLAMA_NUM_PARALLEL=1
Enter fullscreen mode Exit fullscreen mode

This guarantees that Ollama never attempts to allocate a second context buffer while running local coding agents or background tools.


How to Start Ollama With These Flags

Linux / macOS (terminal):

OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_NUM_PARALLEL=1 ollama serve
Enter fullscreen mode Exit fullscreen mode

Windows (PowerShell):

$env:OLLAMA_FLASH_ATTENTION="1"
$env:OLLAMA_KV_CACHE_TYPE="q8_0"
$env:OLLAMA_NUM_PARALLEL="1"
ollama serve
Enter fullscreen mode Exit fullscreen mode

Need a Tailored Modelfile For Your Exact Machine? ($1)

If you don't want to calculate the layer math yourself, I provide a Custom Local AI Hardware Tuning service for $1.00 on Ko-fi. Send me your CPU, GPU model/VRAM, and RAM, and within 2 hours I will generate a custom Modelfile + exact startup configuration tailored to keep your model 100% on GPU at maximum speed.

Originally published at The Zero-Dollar Dev.

Top comments (0)