DEV Community

niuniu
niuniu

Posted on

Quick Tip: Ollama's `OLLAMA_CONTEXT_LENGTH` Env Var — Double Your Local Model's Memory for Free

Most people don't realize Ollama silently truncates your context to 2,048 tokens by default — even if the model supports 128K. Your RAG pipeline isn't failing because the model is dumb. It's failing because it only saw the first page of your document.

The Fix 

# check what you're actually running with
curl http://localhost:11434/api/show -d '{"name": "llama3.1"}' | grep -i context
# -> "context_length": 2048  (default!)

# set it properly before starting the server
OLLAMA_CONTEXT_LENGTH=8192 ollama serve
Enter fullscreen mode Exit fullscreen mode

Or per-request, which overrides the server default:

import requests

r = requests.post("http://localhost:11434/api/generate", json={
    "model": "llama3.1",
    "prompt": long_document_prompt,
    "options": {"num_ctx": 8192},   # <- this is the one that matters
    "stream": False,
})
Enter fullscreen mode Exit fullscreen mode

What It Costs You

Context isn't free — KV cache eats VRAM. Measured on my RTX 3060 12GB with llama3.1:8b (Q4_K_M):

num_ctx VRAM used tokens/sec
2048 5.6 GB 47
8192 6.9 GB 44
16384 8.8 GB 39
32768 11.9 GB (OOM risk) 31

Rule of thumb for 7-8B Q4 models: 8K is free, 16K is safe on 12GB, 32K needs 16GB+ or a smaller quant.

The Bonus Tip Nobody Mentions

If you're on a laptop with 8GB VRAM, don't shrink the context — shrink the batch: options: {"num_ctx": 8192, "num_batch": 128}. Speed drops ~15% but it stops the random OOMs that make people give up on local models entirely.

I found this while debugging why my local setup kept mangling long refactors — my AI coding assistant MonkeyCode pointed at the truncated context in the Ollama logs in about 30 seconds, after I'd spent an hour blaming my prompts.

What context length are you running locally, and on what GPU? Curious where the real-world ceiling is.

Top comments (0)