Ollama says my model does 13,826 tokens/sec. It does 43.

That number is not a typo, and my GPU has not improved. Both figures came out of the same daemon, from the same prompt, thirty seconds apart.

If you have Ollama running, you can check this on your own machine before you read any further. Send the same prompt twice and print three fields:

import json, urllib.request

def chat(prompt, model="qwen2.5:7b"):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": False,
        "options": {"num_predict": 1, "temperature": 0},
    }).encode()
    req = urllib.request.Request("http://localhost:11434/api/chat", body,
                                 {"Content-Type": "application/json"})
    return json.load(urllib.request.urlopen(req, timeout=300))

prompt = "You are a helpful assistant. " + ("Context paragraph about nothing. " * 40) + "Say OK."

for label in ("cold", "warm"):
    d = chat(prompt)
    count, cached = d["prompt_eval_count"], d.get("prompt_eval_cached_count")
    seconds = d["prompt_eval_duration"] / 1e9
    print(f"{label:5} count={count} cached={cached} "
          f"rate={count / seconds:,.0f} tok/s")

Here is what I get. Ollama 0.34.0, qwen2.5:7b, one machine, nothing else running:

	prompt_eval_count	prompt_eval_cached_count	prompt_eval_duration	count ÷ duration
cold	318	0	139.97 ms	2,272 tok/s
warm	318	317	23.00 ms	13,826 tok/s

The second row claims the model prefilled 318 tokens at nearly fourteen thousand tokens per second. It did not. It prefilled one token and read the other 317 out of the KV cache. The honest prefill rate for that row is 43 tok/s.

Why the number comes out wrong

prompt_eval_duration times only the tokens that were actually computed.
prompt_eval_count still reports the total prompt size, cached tokens included. So the obvious calculation divides a whole prompt by the time taken to process a fraction of it, and the error is not a small constant — it is exactly the cache hit ratio:

inflation = prompt_eval_count / (prompt_eval_count - prompt_eval_cached_count)

With 317 of 318 tokens cached, that is 318x. Note what this means in practice: the better your cache is working, the more the number lies. A cold prompt reports truthfully. A perfectly warm one is off by the length of your system prompt. Any dashboard plotting prefill tokens/sec over a conversation is drawing a curve of its own cache hit rate and labelling it throughput.

The fix is one term:

uncached = count - (cached or 0)
rate = uncached / seconds if uncached > 0 else None

This is also what Ollama's own Metrics.Summary() does, so the daemon is not really disagreeing with itself — it just exposes two fields whose units stopped matching, and the older, more obvious one is the one everybody already had in their code.

Two details worth having:

• When uncached is zero, report nothing. A fully cached prompt has no prefill rate. Print an em dash, not 0, and not ∞.
• Absent is not zero. Daemons before 0.33.3 omit prompt_eval_cached_count entirely. Defaulting a missing field to 0 silently turns "I don't know" into a confident, wrong claim that nothing was cached. Leave it undefined and say so in the UI.

The same fact under three different names

Ollama exposes this through three API surfaces, and each calls it something else. All three verified on 0.34.0, same prompt, same warm cache:

endpoint	field	value
/api/chat (native)	prompt_eval_cached_count	317
/v1/chat/completions (OpenAI-compatible)	usage.prompt_tokens_details.cached_tokens	317
/v1/messages (Anthropic-compatible)	usage.cache_read_input_tokens	317

One trap on that last row. The Anthropic-compatible endpoint returned
input_tokens: 1 for a 318-token prompt. input_tokens there means total minus cache reads, not the total — so reconstructing prompt size means adding the two together. If you are summing input_tokens across turns to estimate load, warm turns will quietly contribute almost nothing.

Now that the cache is visible, it is worth measuring

The useful consequence of a readable cached_count is that you can finally see something most of us only assert: a local model reuses its KV cache only while the prompt matches from the very first token. One volatile value near the top forfeits everything after it, every single turn.

Same body text, 227 tokens, one timestamp. Turn one primes the cache; turn two sends the same layout with a fresh timestamp, which is what real traffic does:

timestamp position	tokens	reused	prefill
front	227	40	41.2 ms
back	227	211	16.3 ms

Identical words, identical model, 2.5x the prefill time, purely from where the changing value sits. The 40 tokens still reused in the front case are the chat template's own stable preamble, not your content — everything you wrote is recomputed.

So the rule is boring and worth following anyway: stable text first, volatile text last. Current date, session id, user name, retrieved chunks that change per turn — push them to the end of the prompt, below anything that stays put.

The measurement I got wrong first

I want to flag this, because I spent a day producing a confidently inverted result.

My first attempt asked: how much of layout A's cache does layout B reuse? I primed with one layout, sent the other, and read the hit count. It said the original prompt was getting 323 of 324 tokens cached and my improved rewrite was getting 3 — that the fix made things four times worse.

The question was wrong. Nobody alternates between two prompt layouts. Real traffic sends the same layout every turn with a fresh value in it, so each layout has to be measured against itself: send it twice, mutate the volatile value between the sends, and take the second send as the measurement. Measuring the first send only tells you how well a prompt matches itself, which is ~100% for every layout ever written and tells you nothing.

The thing I keep taking from this: the bug was not in the arithmetic, and no unit test would have caught it, because the code did exactly what I asked. Only the live daemon could tell me I was asking the wrong question. If you are measuring cache behaviour, prime it and mutate it — a single cold send measures nothing you care about.

Go and check your own numbers

The two scripts above are the whole point of this post; everything else is commentary on what they print. If you are running an observability dashboard, a benchmark harness, or anything that reports prefill tokens/sec for a local model, it is worth thirty seconds to find out whether it divides by prompt_eval_count or by prompt_eval_count - prompt_eval_cached_count. Mine divided by the wrong one for months.

Numbers here are from a single machine (Ollama 0.34.0, qwen2.5:7b, 2026-09-16) and yours will differ — the inflation factor in particular is just your cache hit ratio, so it scales with how long your system prompt is. The direction of the effect is not machine-specific.

I maintain LLMxRay, a local LLM observatory that now surfaces the cached counts across all three protocols and has a page for running the prime-and-mutate test against your own daemon. It is Apache-2.0, local-only, and there is nothing to sign up for. But the fields are in the API whether or not you use anything of mine, and
the one-term fix above is the part that actually matters.