DEV Community: Ivan Stankovic

Ollama says my model does 13,826 tokens/sec. It does 43.

Ivan Stankovic — Tue, 15 Sep 2026 22:14:04 +0000

That number is not a typo, and my GPU has not improved. Both figures came out of the same daemon, from the same prompt, thirty seconds apart.

If you have Ollama running, you can check this on your own machine before you read any further. Send the same prompt twice and print three fields:

import json, urllib.request

def chat(prompt, model="qwen2.5:7b"):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": False,
        "options": {"num_predict": 1, "temperature": 0},
    }).encode()
    req = urllib.request.Request("http://localhost:11434/api/chat", body,
                                 {"Content-Type": "application/json"})
    return json.load(urllib.request.urlopen(req, timeout=300))

prompt = "You are a helpful assistant. " + ("Context paragraph about nothing. " * 40) + "Say OK."

for label in ("cold", "warm"):
    d = chat(prompt)
    count, cached = d["prompt_eval_count"], d.get("prompt_eval_cached_count")
    seconds = d["prompt_eval_duration"] / 1e9
    print(f"{label:5} count={count} cached={cached} "
          f"rate={count / seconds:,.0f} tok/s")

Here is what I get. Ollama 0.34.0, qwen2.5:7b, one machine, nothing else running:

	`prompt_eval_count`	`prompt_eval_cached_count`	`prompt_eval_duration`	count ÷ duration
cold	318	0	139.97 ms	2,272 tok/s
warm	318	317	23.00 ms	13,826 tok/s

The second row claims the model prefilled 318 tokens at nearly fourteen thousand tokens per second. It did not. It prefilled one token and read the other 317 out of the KV cache. The honest prefill rate for that row is 43 tok/s.

Why the number comes out wrong

prompt_eval_duration times only the tokens that were actually computed.
prompt_eval_count still reports the total prompt size, cached tokens included. So the obvious calculation divides a whole prompt by the time taken to process a fraction of it, and the error is not a small constant — it is exactly the cache hit ratio:

inflation = prompt_eval_count / (prompt_eval_count - prompt_eval_cached_count)

With 317 of 318 tokens cached, that is 318x. Note what this means in practice: the better your cache is working, the more the number lies. A cold prompt reports truthfully. A perfectly warm one is off by the length of your system prompt. Any dashboard plotting prefill tokens/sec over a conversation is drawing a curve of its own cache hit rate and labelling it throughput.

The fix is one term:

uncached = count - (cached or 0)
rate = uncached / seconds if uncached > 0 else None

This is also what Ollama's own Metrics.Summary() does, so the daemon is not really disagreeing with itself — it just exposes two fields whose units stopped matching, and the older, more obvious one is the one everybody already had in their code.

Two details worth having:

When uncached is zero, report nothing. A fully cached prompt has no prefill rate. Print an em dash, not 0, and not ∞.
Absent is not zero. Daemons before 0.33.3 omit prompt_eval_cached_count entirely. Defaulting a missing field to 0 silently turns "I don't know" into a confident, wrong claim that nothing was cached. Leave it undefined and say so in the UI.

The same fact under three different names

Ollama exposes this through three API surfaces, and each calls it something else. All three verified on 0.34.0, same prompt, same warm cache:

endpoint	field	value
`/api/chat` (native)	`prompt_eval_cached_count`	317
`/v1/chat/completions` (OpenAI-compatible)	`usage.prompt_tokens_details.cached_tokens`	317
`/v1/messages` (Anthropic-compatible)	`usage.cache_read_input_tokens`	317

One trap on that last row. The Anthropic-compatible endpoint returned
input_tokens: 1 for a 318-token prompt. input_tokens there means total minus cache reads, not the total — so reconstructing prompt size means adding the two together. If you are summing input_tokens across turns to estimate load, warm turns will quietly contribute almost nothing.

Now that the cache is visible, it is worth measuring

The useful consequence of a readable cached_count is that you can finally see something most of us only assert: a local model reuses its KV cache only while the prompt matches from the very first token. One volatile value near the top forfeits everything after it, every single turn.

Same body text, 227 tokens, one timestamp. Turn one primes the cache; turn two sends the same layout with a fresh timestamp, which is what real traffic does:

timestamp position	tokens	reused	prefill
front	227	40	41.2 ms
back	227	211	16.3 ms

Identical words, identical model, 2.5x the prefill time, purely from where the changing value sits. The 40 tokens still reused in the front case are the chat template's own stable preamble, not your content — everything you wrote is recomputed.

So the rule is boring and worth following anyway: stable text first, volatile text last. Current date, session id, user name, retrieved chunks that change per turn — push them to the end of the prompt, below anything that stays put.

The measurement I got wrong first

I want to flag this, because I spent a day producing a confidently inverted result.

My first attempt asked: how much of layout A's cache does layout B reuse? I primed with one layout, sent the other, and read the hit count. It said the original prompt was getting 323 of 324 tokens cached and my improved rewrite was getting 3 — that the fix made things four times worse.

The question was wrong. Nobody alternates between two prompt layouts. Real traffic sends the same layout every turn with a fresh value in it, so each layout has to be measured against itself: send it twice, mutate the volatile value between the sends, and take the second send as the measurement. Measuring the first send only tells you how well a prompt matches itself, which is ~100% for every layout ever written and tells you nothing.

The thing I keep taking from this: the bug was not in the arithmetic, and no unit test would have caught it, because the code did exactly what I asked. Only the live daemon could tell me I was asking the wrong question. If you are measuring cache behaviour, prime it and mutate it — a single cold send measures nothing you care about.

Go and check your own numbers

The two scripts above are the whole point of this post; everything else is commentary on what they print. If you are running an observability dashboard, a benchmark harness, or anything that reports prefill tokens/sec for a local model, it is worth thirty seconds to find out whether it divides by prompt_eval_count or by prompt_eval_count - prompt_eval_cached_count. Mine divided by the wrong one for months.

Numbers here are from a single machine (Ollama 0.34.0, qwen2.5:7b, 2026-09-16) and yours will differ — the inflation factor in particular is just your cache hit ratio, so it scales with how long your system prompt is. The direction of the effect is not machine-specific.

I maintain LLMxRay, a local LLM observatory that now surfaces the cached counts across all three protocols and has a page for running the prime-and-mutate test against your own daemon. It is Apache-2.0, local-only, and there is nothing to sign up for. But the fields are in the API whether or not you use anything of mine, and
the one-term fix above is the part that actually matters.

Stop Guessing, Start Seeing: Multi-Model Observability with LLMxRay 🕵️‍♂️

Ivan Stankovic — Fri, 03 Apr 2026 20:55:02 +0000

Have you ever wondered why the same prompt costs more in one language than another? Or why a model feels "smarter" in English but struggles with Arabic or Chinese?

When working with LLMs, we often treat the response as a black box. We see the output, but we don't see the mechanics—the tokenization, the side-by-side comparison of different model families, or how different writing systems affect performance.

I built LLMxRay to pull back the curtain.

What is LLMxRay?

LLMxRay is an open-source observability tool designed to help developers inspect how different LLMs handle the exact same prompt in real-time. Whether you are using local models via Ollama/LM Studio or cloud-based APIs, LLMxRay gives you a "side-by-side" X-ray view of your prompt's journey.

Why use it?

Multi-Model Comparison: Run one prompt against multiple models simultaneously. See how Llama 3 compares to Mistral or GPT-4o in one view.

Multilingual Deep-Dive: This was a big focus for me. The tool supports 4 languages:

English 🇺🇸
French 🇫🇷
Arabic 🇸🇦 (RTL support)
Chinese 🇨🇳

Tokenization Transparency: See exactly how your text is being chopped up into tokens. This is crucial for debugging cost, context window limits, and model "reasoning" quality across different writing systems.

Why 4 Languages?

Tokenization isn't equal. A single concept might be 1 token in English but 3 tokens in another language. By supporting Latin, RTL (Arabic), and character-based (Chinese) scripts, LLMxRay lets you see the economic and technical difference of running multilingual apps.

Try it out

The project is early-stage and open for feedback! You can connect it to your local environment or use your API keys to start comparing models immediately.

👉 Check out the repo here:
https://github.com/LogneBudo/llmxray
or website and docs here:
https://lognebudo.github.io/llmxray/

I’d love to hear from the DEV community:

Which model families do you want to see compared next?

Are there specific visualizations that would help your LLM workflow?

Drop a comment below or open an issue on GitHub! 🚀

LLMxRay: A Local Observatory for Understanding How LLMs Think

Ivan Stankovic — Tue, 17 Mar 2026 11:47:10 +0000

Modern LLMs generate impressive results, but the most interesting part isn’t the final answer — it’s everything that happens before the answer appears.
Token probabilities, confidence shifts, reasoning traces, tool calls, divergences between models… all of this is usually hidden.
I wanted a way to see these internals clearly, locally, and without relying on cloud APIs.
That’s how LLMxRay started.

🧠 What LLMxRay does
LLMxRay is a local-first observability tool for LLMs.
It works with Ollama, LM Studio, llama.cpp, and any endpoint that streams tokens.
It gives you a real-time view of:

Token-by-token generation with confidence heatmaps
Reasoning traces (when the model exposes them)
Side-by-side model comparison
Tool/function call execution
Latency and cost breakdowns
Agent behavior introspection
A built-in Tools Workshop to design and test function-calling
flows

Everything runs locally.
No cloud, no telemetry, no accounts.

🔍 Why I built it

Working with local models, I often needed to answer questions like:
• Why did this model choose this token?
• Where did the reasoning diverge?
• Why does Q4_K_M behave differently from Q6_K?
• What exactly happened during a tool call?
• How do two models respond to the same prompt internally?

Existing UIs focus on chat experience, not introspection.
Debugging required custom scripts, logs, or guesswork.
LLMxRay tries to make this transparent.

🛠️ How it works

LLMxRay sits between you and your local model:

• It captures the token stream
• It records probabilities and reasoning (if available)
• It visualizes everything in a clean, interactive UI
• It stores traces so you can compare runs
• It supports multiple models and endpoints
You can run it with:

Or clone the repo and run it locally.

📦 Links

• GitHub: https://github.com/lognebudo/llmxray
• Demo / docs: https://lognebudo.github.io/llmxray/

💬 What’s next
I’m working on:

• better comparison tools
• an education pillar with kits for teachers and students
• improved reasoning visualization
• support for more local runtimes

If you work with local models, I’d love to hear how you debug or introspect them — and what features would help you the most.