maruhana

Posted on Nov 9 Originally published at zenn.dev

Ollama vs. llama.cpp on the Same GGUF: The 15% Speed Gap Was the Defaults, Not the Engine

#ai #llm #ollama #llamacpp

Measured on a Surface Laptop Ultra (Windows on Arm, RTX Spark): align the draft and sampling settings, and the gap mostly disappears.

This is an English adaptation of an article I originally published in Japanese on Zenn: Ollama と Llama-Windows を同じ GGUF で比べたら、差はエンジンではなく既定の設定だった. The measurements and conclusions are mine; I used an AI tool to help translate and restructure the text, and I checked the numbers against the original.

If you benchmark local LLMs, this is a log of how I got the comparison wrong twice before getting it roughly right. The fix was not a faster engine. It was making both sides use the same settings.

TL;DR

Same GGUF (Qwen3.6-35B-A3B, Q4_K_M), same laptop. With default settings, Llama-Windows ran at about 85% of Ollama's speed (roughly 15% slower).
Ollama is also running llama.cpp inside. Its bundled build was only 32 commits newer than the one in Llama-Windows.
Two defaults differed: Ollama uses MTP speculative decoding (2 draft tokens) by default, and the sampling defaults are different.
With both aligned, the speeds were about the same. The one remaining difference: with presence_penalty 1.5, only the llama.cpp side got about 7% slower. I have not investigated why.
2 draft tokens were enough. 3 was never faster.

Names first

Four names in this article look alike. They are different things:

Name	What it is
Llama-Windows	A Windows tray app from ggml-org (the llama.cpp developers). Not llama.cpp itself. github.com/ggml-org/Llama-Windows
llama.exe	The inference engine that Llama-Windows downloads and runs. It is a llama.cpp build (b11706, as recorded at measurement time).
llama.cpp	The engine project both tools are built on.
llama-server (bundled in Ollama)	The llama.cpp server that Ollama 0.40.7 ships and runs internally.

The first comparison below is "Ollama app vs. Llama-Windows app, defaults as shipped". Everything after that compares Ollama against llama.exe started by hand, so I could control its settings.

How the comparison flipped

I revised the conclusion twice while writing this:

Defaults as shipped: Llama-Windows is about 15% slower than Ollama.
Draft (speculative decoding) aligned: now llama.exe looked 7–8% faster than Ollama. That was also too early.
Draft and sampling aligned: about the same speed.

The rest of the article is what I forgot to align at each step.

Test setup

Item	Details
PC	Surface Laptop Ultra (Windows 11 Arm64, NVIDIA RTX Spark N1X)
Model	Qwen3.6-35B-A3B, Q4_K_M. The GGUF Ollama had downloaded, hard-linked into Llama-Windows (so no extra 22 GB on disk)
Ollama	0.40.7
Llama-Windows	0.13.0. Its engine is `llama.exe` (llama.cpp b11706, as recorded at measurement time; CUDA aarch64 build)
Context length	262144 (same on both)
Prompt	A single Japanese prompt asking for a three-sentence explanation of the capital of Japan. Thinking mode left at its default
Speed metric	Ollama: from `eval_count` / `eval_duration` in its API. `llama.exe`: `timings.predicted_per_second` from its server API (`/v1/chat/completions`). Unit is tokens/sec, including the thinking tokens

Step 1: with defaults, Llama-Windows was about 15% slower

	Ollama 0.40.7	Llama-Windows 0.13.0
Generation speed (3 runs, tokens/sec)	101.89 / 104.12 / 100.11	87.97 / 86.19 / 85.70
GPU utilization	69–75%	84–86%

Same file, and Llama-Windows ran at about 85% of Ollama's speed. It also had higher GPU utilization. Busier, yet slower.

Ollama is llama.cpp inside, too

Before blaming either engine, I checked what Ollama was actually running. Ollama 0.40.7 runs its bundled llama.cpp llama-server on CUDA:

The log shows using llama-server for model, and the RUNNER column in ollama ps says llamacpp.
The GPU library is ggml-cuda.dll from cuda_v13. No Vulkan build is included.
The bundled llama-server commit was only 32 commits newer than Llama-Windows' llama.exe (b11706). I compared the commits printed by --version using GitHub's compare view.

So this was close to llama.cpp vs. llama.cpp. If there is a 15% gap, the settings are the first suspect, not the engine.

Difference 1: Ollama drafts ahead by default (MTP)

Qwen3.6 has multi-token prediction (MTP) layers built into the model. They let the runtime guess upcoming tokens and verify them in one go: speculative decoding without a separate draft model.

Ollama uses this by default. ollama show lists draft_num_predict 2 under Parameters, meaning it drafts 2 tokens ahead. Setting draft_num_predict to 0 in the API options turns it off.
The Llama-Windows app does not draft by default. If you start llama.exe yourself, --spec-type draft-mtp --spec-draft-n-max 2 gives you 2-token drafting.

Drafting costs about 1.5–2 GB of extra GPU memory and lowers GPU utilization. That explains the "busier but slower" pattern in step 1: Llama-Windows was simply not drafting.

When I aligned only the drafting, llama.exe came out 7–8% faster than Ollama. That was my second conclusion, and it was also premature.

Difference 2: the sampling defaults were different

The sampling defaults did not match either:

	Ollama (this model's defaults)	Llama-Windows (llama.cpp defaults)
temperature	1.0	0.8
top_k	20	40
min_p	0	0.05
presence_penalty	1.5	0

So I started the app's engine (llama.exe) separately as a server and aligned both drafting and sampling. I used two sampling settings:

S1: the model's recommended values, which are the same as Ollama's defaults: temperature 1.0, top_k 20, top_p 0.95, min_p 0, presence_penalty 1.5, repeat_penalty 1.0.
S2: greedy decoding (temperature 0), which always picks the most likely token.

Other conditions:

Seed fixed at 42 for both.
Values passed per request. I did not change the app's or Ollama's configuration files.
Generation capped at 1024 tokens (num_predict for Ollama, max_tokens for llama.exe).
For each combination, one warm-up run discarded, then 5 measured runs. The engine not being measured was stopped.

Results (tokens/sec, mean of 5 runs; standard deviations 0.38–1.90):

Drafting	Sampling	Ollama 0.40.7	llama.exe (b11706)
None	S1 (recommended)	81.45	74.75
None	S2 (greedy)	81.69	80.44
MTP, 2 tokens	S1 (recommended)	99.05	97.30
MTP, 2 tokens	S2 (greedy)	104.04	110.85
MTP, 3 tokens	S1 (recommended)	97.17	97.01
MTP, 3 tokens	S2 (greedy)	101.31	106.00

What I read from this:

With both drafting and sampling aligned, the two were about the same speed (no drafting, greedy: 81.69 vs. 80.44).
The remaining difference was presence_penalty. At 1.5, only llama.exe got about 7% slower (80.44 → 74.75). Ollama barely changed (81.69 at presence_penalty 0 → 81.45 at 1.5).
Drafting 2 tokens sped both up by roughly 20–40% (Ollama 22–27%, llama.exe 30–38%). With the recommended settings, llama.exe made up for the presence_penalty slowdown through drafting and ended up roughly level with Ollama.
With greedy decoding and 2-token drafting, llama.exe came out ahead (110.85 vs. 104.04). But the output text was different, and so was the draft acceptance rate (0.835 vs. 0.819), so I can't call that an engine difference.

2 draft tokens were enough

Going from 2 to 3 draft tokens was not faster in any of the four combinations (the same or slower). The draft acceptance rate was 0.71–0.84 with 2 tokens and 0.65–0.71 with 3. The third guess is harder to hit, which pulls down the overall acceptance rate.

Before I fixed the seed, 3-token drafting swung between 103 and 113 tokens/sec, because each run generated a different text. With the seed fixed, all 5 runs produced the same number of tokens and the spread went away.

What I checked when installing Llama-Windows

Llama-Windows fetches its inference engine separately from the app itself (an MSIX package). The app downloads install.ps1, runs it in memory, and installs the llama.exe build matching your GPU (here, the CUDA aarch64 build) from Hugging Face.

The MSIX is signed, and its hash matched the one listed in the release.
The downloaded llama.exe was not signed, and there was no hash check. It is just fetched over HTTPS.
So I read install.ps1 before installing, and recorded the hash of llama.exe afterwards.
By default, the server listened on 127.0.0.1 only.

What I didn't measure

One prompt, one PC. With long contexts or other models, the direction and size of the gap could change.
Because the seed was fixed, the 5 runs are the same text repeated. Variation from different outputs is not included.
Even with greedy decoding, the two engines' outputs did not match. The thinking started the same, but the answer diverged partway, and the generated token counts ranged from 739 to 1024. Even the same engine produced different text with and without drafting.
I have not investigated why presence_penalty slows down only llama.exe. Ollama's bundled build is 32 commits newer, so it might be a change in between, or it might be a difference in how the engines are called. Those are candidates, not findings.
I did not compare answer quality.

Takeaways

Ollama runs llama.cpp too. With the same GGUF, the speed gap came from default settings rather than the engine.
Two defaults differed: Ollama drafts 2 tokens ahead with MTP, and its sampling values (presence_penalty 1.5 and others) are different.
Aligned, the speeds were about the same; only presence_penalty left a ~7% gap on the llama.cpp side. 2 draft tokens were enough.
When comparing speeds, align drafting, sampling, seed, and generation length. Leave any one of them out and a 10% gap is easy to get, sometimes pointing the wrong way.

All defaults above are as of the measured versions (Ollama 0.40.7, Llama-Windows 0.13.0 with llama.cpp b11706). Newer versions may have changed them.

Update note (Nov 2026): Ollama on this machine has since auto-updated from 0.40.7 to 0.40.10. All measurements above are from 0.40.7, and I have not re-checked whether the defaults changed in 0.40.10. Llama-Windows is still 0.13.0.

Llama-Windows and llama.cpp are MIT-licensed (ggml-org/Llama-Windows).

About this series

This is part of my Japanese-language series on the Surface Laptop Ultra (Windows on Arm with an NVIDIA GPU). Related articles on Zenn (Japanese):

The original of this article: Ollama と Llama-Windows を同じ GGUF で比べたら、差はエンジンではなく既定の設定だった.

DEV Community