Ollama keep_alive: My Model Reloaded 214 Times in One Day

My local chat app was fast every single time I tested it, and slow every single time I actually used it.

That's the tell, and I ignored it for weeks. I'd type a question during development, get a first token in under a second, ship the change. Then I'd come back after lunch, ask one thing, and sit there for eleven seconds watching a cursor blink. Same box. Same model. Same prompt.

It wasn't the model being slow. It was Ollama loading the model off disk again, because it had quietly evicted it while I was living my life. The knob is called keep_alive, and Ollama keep_alive turns out to have three separate ways of not doing what you think it does.

So I put a timer on every request for 24 hours. 1,180 requests, 214 model load events. Here's the autopsy.

TL;DR

• Ollama unloads a model after 5 minutes idle by default. Any request arriving after that pays a full cold load from disk.
• On my box that cold load cost 11.4s to first token vs 0.9s warm — 18.1% of my requests were cold, which dragged my overall p50 to 3.1s.
• Passing keep_alive in the request body did nothing on the OpenAI-compatible /v1/chat/completions endpoint. The native /api/chat endpoint honors it.
• Setting keep_alive: -1 on two models that don't both fit in VRAM made things worse, not better: partial CPU offload dropped generation from 42 tok/s to 6 tok/s.
• The fix was boring: OLLAMA_KEEP_ALIVE=24h as a server-side env var, one resident model per GPU, embeddings moved to a separate CPU-only Ollama instance. Loads went 214/day → 9/day.

Why does Ollama reload my model on every request?

Because Ollama's default keep_alive is 5 minutes. After five minutes with no traffic, the runner exits and the weights leave VRAM. The next request re-reads gigabytes from disk, re-allocates VRAM, and only then starts generating.

This is a completely reasonable default for a laptop. It is a terrible default for anything with bursty traffic, which is every side project ever built.

My setup, so you can judge whether my numbers transfer:

• One box, 16GB VRAM, models on an NVMe drive
• llama3.1:8b for chat (~4.9GB), qwen2.5-coder:14b for a code helper (~9GB), nomic-embed-text for embeddings (~274MB)
• Three clients: a chat UI I use by hand, a cron job that summarizes my notes every 10 minutes, and a small RAG indexer

Look at that cron interval. Every 10 minutes, against a 5-minute idle timeout. That job was cold 100% of the time. It had never once hit a warm model. For weeks I assumed "local summarization just takes 12 seconds."

How do I check if Ollama is reloading my model?

Two commands, thirty seconds, and you'll know.

First, is anything resident right now?

ollama ps

If that's empty while you think your model is "running," it's not. It's on disk. UNTIL in that output is your real keep_alive, not what you put in your config.

Second, count load events in the server log over a day:

# Linux (systemd)
journalctl -u ollama --since "24 hours ago" | grep -ci "llama runner started"

# macOS
grep -ci "llama runner started" ~/.ollama/logs/server.log

Log wording drifts between versions, so grep your own log once by hand and pick the line that appears exactly once per load. Mine said 214 over a day against 1,180 requests. That ratio is the whole story: one reload for every 5.5 requests.

Then I wrapped the client to record time-to-first-token, because "it feels slow sometimes" is not a bug report:

import time, json, requests

def ttft(prompt, model="llama3.1:8b"):
    t0 = time.perf_counter()
    r = requests.post("http://localhost:11434/api/chat",
                      json={"model": model,
                            "messages": [{"role": "user", "content": prompt}],
                            "stream": True},
                      stream=True)
    for line in r.iter_lines():
        if not line:
            continue
        chunk = json.loads(line)
        if chunk.get("message", {}).get("content"):
            return time.perf_counter() - t0   # first real token

Log that number with a timestamp for a day. The histogram was not a bell curve. It was two spikes: one at 0.9s, one at 11.4s. Nothing in between. That shape means you have a binary state problem, not a slow model.

Does keep_alive work on the OpenAI-compatible endpoint?

In my testing, no. This is the part that cost me an entire evening.

My app talked to Ollama through the OpenAI SDK, pointed at /v1, because that's the path of least resistance when you want to swap providers later. So I did the obvious thing and added keep_alive to the request:

client.chat.completions.create(
    model="llama3.1:8b",
    messages=[...],
    extra_body={"keep_alive": "24h"},   # did nothing on my version
)

Load count the next day: 200-something. Unchanged. I'd "fixed" it and the graph didn't move, which is the only reason I caught it. If I hadn't been counting loads, I would have declared victory and kept eating 11-second requests.

keep_alive is an Ollama concept, not an OpenAI one, and the compatibility layer on my version drops it. The native endpoint honors it fine:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role":"user","content":"hi"}],
  "keep_alive": "24h"
}'

Don't take my word for the version behavior. Send one request, then run ollama ps and read the UNTIL column. If it says 5 minutes from now, your keep_alive was ignored, whatever the docs say about your build.

Why did keep_alive: -1 make my local LLM slower?

Because -1 means "never unload," and never-unload plus two models that don't both fit in 16GB means something has to give. What gives is layer placement.

I set keep_alive: -1 on both the 8B chat model and the 14B coder model, feeling clever. Loads dropped from 214 to about 60 a day. Latency got worse.

With both pinned, the 14B model no longer got a clean full-GPU allocation. It landed partially on the GPU with the remaining layers on CPU. Generation went from 42 tok/s to 6 tok/s. A 400-token answer went from 10 seconds to over a minute. I had traded a one-time 11-second cold start for a permanent 7x tax on every token.

The lesson I'd tattoo on the inside of my eyelids: a cold start is cheaper than a partial offload. Reloading is a fixed cost paid once. Spilling layers to CPU is a cost paid per token, forever, silently, and it never shows up in a load counter.

What actually fixed it?

Four changes, in order of how much they mattered.

1. Set keep_alive on the server, not in requests. One env var covers every client, including the ones you forgot about and the ones going through /v1:

# /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="OLLAMA_KEEP_ALIVE=24h"

sudo systemctl daemon-reload && sudo systemctl restart ollama

On macOS, launchctl setenv OLLAMA_KEEP_ALIVE 24h before starting the app. Setting it in your shell profile does nothing, because the server isn't your shell.

2. One resident model per GPU. The chat model stays loaded. The coder model is invoked maybe 15 times a day, so I let it cold start and I stopped pretending that mattered. Alternating two big models on one GPU is thrash, and no timeout setting fixes thrash.

3. Embeddings on a separate CPU-only instance. The indexer was firing hundreds of tiny embedding calls and stealing VRAM for a 274MB model that runs fine on CPU:

CUDA_VISIBLE_DEVICES="" OLLAMA_HOST=127.0.0.1:11435 ollama serve

Point the indexer at :11435, leave :11434 for the chat model. Two processes, zero contention.

4. I deleted my warmup cron. Before I understood any of this, my instinct was a job pinging the model every 4 minutes to keep it hot. It worked, sort of, and it also kept the GPU awake 24/7 for a model I use in two bursts a day. OLLAMA_KEEP_ALIVE does the same job without a second moving part to debug at 2am.

What did it cost, in numbers?

One box, one workload, 24 hours before and 24 hours after.

Metric	Before	After
Requests logged	1,180	1,206
Model load events	214	9
Cold requests	18.1%	0.8%
p50 time to first token	3.1s	1.0s
p95 time to first token	12.6s	1.9s
Cron summarizer, cold rate	100%	0%
Chat generation speed	42 tok/s	42 tok/s

That last row is the one I keep pointing at. Tokens per second never changed. The model was never slow. Every second I'd spent for weeks blaming quantization, context length, and my GPU was time spent in open().

Caveats, because I'd want them from you: this is one machine, one GPU, one traffic shape, one Ollama version. The /v1 behavior in particular is a version detail and might already differ on yours. The method transfers even if my numbers don't. Count loads, measure first-token latency, and read ollama ps instead of trusting a config file.

So what does Ollama keep_alive actually do?

Ollama keep_alive controls how long a model stays resident in memory after its last request, defaulting to 5 minutes, after which the next request pays a full cold load from disk — in my case 11.4s to first token instead of 0.9s. Set it server-side with OLLAMA_KEEP_ALIVE rather than per-request, because the OpenAI-compatible /v1 endpoint ignored the body parameter in my testing. Verify with ollama ps and by counting runner-start lines in the server log, not by reading your config. And resist keep_alive: -1 on multiple models sharing one GPU: pinning models that don't both fit forces partial CPU offload, which cost me 42 tok/s down to 6 tok/s and is far worse than the cold start you were trying to avoid.

---

Written by the developer behind Preterview, an interview prep platform.