DEV Community: Ivan Stankovic

Four numbers Ollama 0.40 gives you now, and which ones to trust

Ivan Stankovic — Sun, 11 Oct 2026 21:25:40 +0000

Ollama went from 0.33 to 0.40 in six weeks, and most of what changed is invisible unless you
read the responses with a magnifier. I did, because I maintain a tool that has to. Here are
four numbers the daemon reports now that it didn't before, what each one actually means, and
in one case why you should not believe it.

Everything below is two curl calls on your own machine. The tool is at the end.

1. A prefill rate the server computes for you — wrongly

Since 0.35.1 the OpenAI-compatible endpoint attaches a timings block to the final usage
chunk. Send one prompt twice, read the second answer:

"usage":   {"prompt_tokens": 31, "prompt_tokens_details": {"cached_tokens": 30}, "completion_tokens": 11},
"timings": {"prompt_n": 31, "prompt_ms": 8, "prompt_per_second": 3875,
            "predicted_n": 11, "predicted_ms": 83, "predicted_per_second": 132.5}

One token was evaluated, in 8 milliseconds. The block says 3875 tokens per second.

prompt_n is the whole prompt, cached tokens included. prompt_ms is the time spent on the
uncached ones only. Divide the first by the second and you get a rate inflated by exactly the
cache hit ratio: count / (count - cached), so 31x here, and anything up to a few hundred on a
long system prompt that matched. The honest figure sits in the same chunk:

(prompt_tokens - cached_tokens) / prompt_ms  =  1 / 8 ms  =  125 tok/s

If that looks familiar, it is the formula bug that the native API acquired in 0.33.3, when
prompt_eval_duration started timing only uncached tokens while prompt_eval_count stayed the
total. Now the server emits the result of that division itself, labelled as a rate. The decode
half, predicted_per_second, is fine; generated tokens are never cached.

What LLMxRay does with it: the Protocol Observatory's envelope diff now has three rows for
prefill: the time, the rate as the daemon states it, and the rate over evaluated tokens. Run a
prompt twice and the stated cell turns amber on the second run. That is the whole feature; the
point is to see the two numbers diverge on your own daemon.

2. A probability instead of a "yes"

0.35.1 added decision models: nimble, tev1, clef, clef-flash. They do not write text.
You give /v1/systemone a state and typed questions, and you get back the model's own
distribution over the candidates:

curl localhost:11434/v1/systemone -d '{
  "model": "nimble",
  "state": "Our checkout has returned 500 errors since 9am. Customers cannot pay.",
  "questions": {
    "label":    {"type": "choice", "instructions": "Which label fits this ticket?",
                 "criteria": {"billing": "Payments and refunds", "bug": "Software errors", "account": "Login and account access"}},
    "urgent":   {"type": "noul",   "instructions": "Is this ticket urgent?"},
    "severity": {"type": "score",  "instructions": "How severe is the impact?",
                 "criteria": ["cosmetic", "minor", "major", "full outage"]}
  }}'

"label":    {"type": "choice", "choice": "bug", "probabilities": {"billing": 0.038, "bug": 0.907, "account": 0.055}, "confidence": 0.66},
"urgent":   {"type": "noul", "noul": 0.9897},
"severity": {"type": "score", "score": 2.37, "probabilities": {"0": 0.005, "1": 0.022, "2": 0.569, "3": 0.403}}

Three things worth knowing before you build on it. score criteria must be an array; a map is
rejected with a clear message. Limits are 2 to 26 candidates and 1 to 64 questions per request.
And a score is the expected position on your scale, so 2.37 on a four-step scale means
"major, leaning to full outage", which is more information than either label alone.

The trap: these models report decision as their only capability, and /api/chat does not
refuse them. Ask nimble "hi" and it streams a reply about the schema it was not given. Any
client that builds its model list from /api/tags without reading capabilities will offer a
decision model for chat.

What LLMxRay does with it: decision models are kept out of the chat and embedding selectors
by capability, and a Decision Lab page sends the request above and draws each answer as
probability bars with the chosen candidate highlighted, plus the token usage. The sample on the
page is the ticket above, asked three ways.

3. Which runtime actually ran your model

0.40 made MLX the default on Apple Silicon for the architectures it supports, which means the
same model tag can now run on two different engines. The daemon says which:

/api/tags  ->  "details": {"runner": "mlx"}        // or "llamacpp", or "ggml" for models pulled before 0.40
/api/ps    ->  "runner": "mlx"

On the Mac mini I measured with, nomic-embed-text sat on llamacpp and embeddinggemma-2 on
mlx at the same time. A benchmark result that only records the model name is now ambiguous on
that machine.

What LLMxRay does with it: the runner is shown on each model and each running model, and
every new benchmark result is stamped with it, so two results under one name are only compared
when the runner matches. Results recorded before 0.7.0 carry no runner, and the page says so
rather than guessing.

4. An image costs 258 tokens to embed, on one platform

Also in 0.40: /api/embed accepts an image next to the text, on models that report both
embedding and vision:

curl localhost:11434/api/embed -d '{"model": "embeddinggemma-2",
  "input": {"text": "a cat on a windowsill", "image": "<base64 png>"}}'

Input	`prompt_eval_count`	Vector
short sentence	7	768 dims
same sentence + a 1x1 PNG	265	768 dims

A one-pixel image pays a ~258-token block, so the cost is per image, not per pixel. Sending an
image to a text-only embedding model returns HTTP 501, model does not support media embeddings, a status worth matching on rather than a generic 400.

The catch is in the model, not the API: embeddinggemma-2 is the only multimodal embedding
model in the library, and it requires MLX. On Windows the pull fails with "this model requires
MLX support". So in October 2026, image embeddings in Ollama are an Apple Silicon feature. I
measured the table above by forwarding a Mac mini's daemon over SSH.

What LLMxRay does with it: on the Embeddings page, when the selected model reports embedding
and vision, an image can be attached in the playground and on both sides of the similarity
calculator: photo against sentence, or photo against photo. Every result shows the prompt tokens
it cost. Text-only models show no image controls at all.

One more thing the daemon does quietly

0.40.2 upgrades models in the background the first time you run them and keeps the original
copy on disk, so a downgrade stays safe. The release notes give a jq pipeline to find and
delete the backups by digest. LLMxRay's Models tab does the same selection and offers a remove
button per backup. In fairness: no daemon I have access to has produced a backup yet, 0.40.3
paused the upgrades, so that part is built from Ollama's own recipe and tested against its shape,
not against a live backup. The section stays hidden until one exists.

The tool

LLMxRay is a local observability UI for Ollama:
Apache-2.0, nothing leaves localhost, npx llmxray or docker pull djovaneli/llmxray. Version
0.7.1 is the one with everything above, verified against Ollama 0.40.3. The guides are in
English, French and Chinese.

None of the four numbers needs it. What it adds is the habit of putting the daemon's figure and
the honest figure on the same screen, which is how the first one was caught.

Ollama says my model does 13,826 tokens/sec. It does 43.

Ivan Stankovic — Tue, 15 Sep 2026 22:14:04 +0000

That number is not a typo, and my GPU has not improved. Both figures came out of the same daemon, from the same prompt, thirty seconds apart.

If you have Ollama running, you can check this on your own machine before you read any further. Send the same prompt twice and print three fields:

import json, urllib.request

def chat(prompt, model="qwen2.5:7b"):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": False,
        "options": {"num_predict": 1, "temperature": 0},
    }).encode()
    req = urllib.request.Request("http://localhost:11434/api/chat", body,
                                 {"Content-Type": "application/json"})
    return json.load(urllib.request.urlopen(req, timeout=300))

prompt = "You are a helpful assistant. " + ("Context paragraph about nothing. " * 40) + "Say OK."

for label in ("cold", "warm"):
    d = chat(prompt)
    count, cached = d["prompt_eval_count"], d.get("prompt_eval_cached_count")
    seconds = d["prompt_eval_duration"] / 1e9
    print(f"{label:5} count={count} cached={cached} "
          f"rate={count / seconds:,.0f} tok/s")

Here is what I get. Ollama 0.34.0, qwen2.5:7b, one machine, nothing else running:

	`prompt_eval_count`	`prompt_eval_cached_count`	`prompt_eval_duration`	count ÷ duration
cold	318	0	139.97 ms	2,272 tok/s
warm	318	317	23.00 ms	13,826 tok/s

The second row claims the model prefilled 318 tokens at nearly fourteen thousand tokens per second. It did not. It prefilled one token and read the other 317 out of the KV cache. The honest prefill rate for that row is 43 tok/s.

Why the number comes out wrong

prompt_eval_duration times only the tokens that were actually computed.
prompt_eval_count still reports the total prompt size, cached tokens included. So the obvious calculation divides a whole prompt by the time taken to process a fraction of it, and the error is not a small constant — it is exactly the cache hit ratio:

inflation = prompt_eval_count / (prompt_eval_count - prompt_eval_cached_count)

With 317 of 318 tokens cached, that is 318x. Note what this means in practice: the better your cache is working, the more the number lies. A cold prompt reports truthfully. A perfectly warm one is off by the length of your system prompt. Any dashboard plotting prefill tokens/sec over a conversation is drawing a curve of its own cache hit rate and labelling it throughput.

The fix is one term:

uncached = count - (cached or 0)
rate = uncached / seconds if uncached > 0 else None

This is also what Ollama's own Metrics.Summary() does, so the daemon is not really disagreeing with itself — it just exposes two fields whose units stopped matching, and the older, more obvious one is the one everybody already had in their code.

Two details worth having:

When uncached is zero, report nothing. A fully cached prompt has no prefill rate. Print an em dash, not 0, and not ∞.
Absent is not zero. Daemons before 0.33.3 omit prompt_eval_cached_count entirely. Defaulting a missing field to 0 silently turns "I don't know" into a confident, wrong claim that nothing was cached. Leave it undefined and say so in the UI.

The same fact under three different names

Ollama exposes this through three API surfaces, and each calls it something else. All three verified on 0.34.0, same prompt, same warm cache:

endpoint	field	value
`/api/chat` (native)	`prompt_eval_cached_count`	317
`/v1/chat/completions` (OpenAI-compatible)	`usage.prompt_tokens_details.cached_tokens`	317
`/v1/messages` (Anthropic-compatible)	`usage.cache_read_input_tokens`	317

One trap on that last row. The Anthropic-compatible endpoint returned
input_tokens: 1 for a 318-token prompt. input_tokens there means total minus cache reads, not the total — so reconstructing prompt size means adding the two together. If you are summing input_tokens across turns to estimate load, warm turns will quietly contribute almost nothing.

Now that the cache is visible, it is worth measuring

The useful consequence of a readable cached_count is that you can finally see something most of us only assert: a local model reuses its KV cache only while the prompt matches from the very first token. One volatile value near the top forfeits everything after it, every single turn.

Same body text, 227 tokens, one timestamp. Turn one primes the cache; turn two sends the same layout with a fresh timestamp, which is what real traffic does:

timestamp position	tokens	reused	prefill
front	227	40	41.2 ms
back	227	211	16.3 ms

Identical words, identical model, 2.5x the prefill time, purely from where the changing value sits. The 40 tokens still reused in the front case are the chat template's own stable preamble, not your content — everything you wrote is recomputed.

So the rule is boring and worth following anyway: stable text first, volatile text last. Current date, session id, user name, retrieved chunks that change per turn — push them to the end of the prompt, below anything that stays put.

The measurement I got wrong first

I want to flag this, because I spent a day producing a confidently inverted result.

My first attempt asked: how much of layout A's cache does layout B reuse? I primed with one layout, sent the other, and read the hit count. It said the original prompt was getting 323 of 324 tokens cached and my improved rewrite was getting 3 — that the fix made things four times worse.

The question was wrong. Nobody alternates between two prompt layouts. Real traffic sends the same layout every turn with a fresh value in it, so each layout has to be measured against itself: send it twice, mutate the volatile value between the sends, and take the second send as the measurement. Measuring the first send only tells you how well a prompt matches itself, which is ~100% for every layout ever written and tells you nothing.

The thing I keep taking from this: the bug was not in the arithmetic, and no unit test would have caught it, because the code did exactly what I asked. Only the live daemon could tell me I was asking the wrong question. If you are measuring cache behaviour, prime it and mutate it — a single cold send measures nothing you care about.

Go and check your own numbers

The two scripts above are the whole point of this post; everything else is commentary on what they print. If you are running an observability dashboard, a benchmark harness, or anything that reports prefill tokens/sec for a local model, it is worth thirty seconds to find out whether it divides by prompt_eval_count or by prompt_eval_count - prompt_eval_cached_count. Mine divided by the wrong one for months.

Numbers here are from a single machine (Ollama 0.34.0, qwen2.5:7b, 2026-09-16) and yours will differ — the inflation factor in particular is just your cache hit ratio, so it scales with how long your system prompt is. The direction of the effect is not machine-specific.

I maintain LLMxRay, a local LLM observatory that now surfaces the cached counts across all three protocols and has a page for running the prime-and-mutate test against your own daemon. It is Apache-2.0, local-only, and there is nothing to sign up for. But the fields are in the API whether or not you use anything of mine, and
the one-term fix above is the part that actually matters.

Stop Guessing, Start Seeing: Multi-Model Observability with LLMxRay 🕵️‍♂️

Ivan Stankovic — Fri, 03 Apr 2026 20:55:02 +0000

Have you ever wondered why the same prompt costs more in one language than another? Or why a model feels "smarter" in English but struggles with Arabic or Chinese?

When working with LLMs, we often treat the response as a black box. We see the output, but we don't see the mechanics—the tokenization, the side-by-side comparison of different model families, or how different writing systems affect performance.

I built LLMxRay to pull back the curtain.

What is LLMxRay?

LLMxRay is an open-source observability tool designed to help developers inspect how different LLMs handle the exact same prompt in real-time. Whether you are using local models via Ollama/LM Studio or cloud-based APIs, LLMxRay gives you a "side-by-side" X-ray view of your prompt's journey.

Why use it?

Multi-Model Comparison: Run one prompt against multiple models simultaneously. See how Llama 3 compares to Mistral or GPT-4o in one view.

Multilingual Deep-Dive: This was a big focus for me. The tool supports 4 languages:

English 🇺🇸
French 🇫🇷
Arabic 🇸🇦 (RTL support)
Chinese 🇨🇳

Tokenization Transparency: See exactly how your text is being chopped up into tokens. This is crucial for debugging cost, context window limits, and model "reasoning" quality across different writing systems.

Why 4 Languages?

Tokenization isn't equal. A single concept might be 1 token in English but 3 tokens in another language. By supporting Latin, RTL (Arabic), and character-based (Chinese) scripts, LLMxRay lets you see the economic and technical difference of running multilingual apps.

Try it out

The project is early-stage and open for feedback! You can connect it to your local environment or use your API keys to start comparing models immediately.

👉 Check out the repo here:
https://github.com/LogneBudo/llmxray
or website and docs here:
https://lognebudo.github.io/llmxray/

I’d love to hear from the DEV community:

Which model families do you want to see compared next?

Are there specific visualizations that would help your LLM workflow?

Drop a comment below or open an issue on GitHub! 🚀

LLMxRay: A Local Observatory for Understanding How LLMs Think

Ivan Stankovic — Tue, 17 Mar 2026 11:47:10 +0000

Modern LLMs generate impressive results, but the most interesting part isn’t the final answer — it’s everything that happens before the answer appears.
Token probabilities, confidence shifts, reasoning traces, tool calls, divergences between models… all of this is usually hidden.
I wanted a way to see these internals clearly, locally, and without relying on cloud APIs.
That’s how LLMxRay started.

🧠 What LLMxRay does
LLMxRay is a local-first observability tool for LLMs.
It works with Ollama, LM Studio, llama.cpp, and any endpoint that streams tokens.
It gives you a real-time view of:

Token-by-token generation with confidence heatmaps
Reasoning traces (when the model exposes them)
Side-by-side model comparison
Tool/function call execution
Latency and cost breakdowns
Agent behavior introspection
A built-in Tools Workshop to design and test function-calling
flows

Everything runs locally.
No cloud, no telemetry, no accounts.

🔍 Why I built it

Working with local models, I often needed to answer questions like:
• Why did this model choose this token?
• Where did the reasoning diverge?
• Why does Q4_K_M behave differently from Q6_K?
• What exactly happened during a tool call?
• How do two models respond to the same prompt internally?

Existing UIs focus on chat experience, not introspection.
Debugging required custom scripts, logs, or guesswork.
LLMxRay tries to make this transparent.

🛠️ How it works

LLMxRay sits between you and your local model:

• It captures the token stream
• It records probabilities and reasoning (if available)
• It visualizes everything in a clean, interactive UI
• It stores traces so you can compare runs
• It supports multiple models and endpoints
You can run it with:

Or clone the repo and run it locally.

📦 Links

• GitHub: https://github.com/lognebudo/llmxray
• Demo / docs: https://lognebudo.github.io/llmxray/

💬 What’s next
I’m working on:

• better comparison tools
• an education pillar with kits for teachers and students
• improved reasoning visualization
• support for more local runtimes

If you work with local models, I’d love to hear how you debug or introspect them — and what features would help you the most.