You set temperature to 0, fix the seed, send the same prompt twice, and get two different answers. This post explains where the difference comes from, how to see it on your own machine, and what each engine offers if you need answers that repeat.
The symptom
One measured case: I ran lm-evaluation-harness on gsm8k (100 problems) against llama.cpp's server (build b11643, Qwen2.5-7B-Instruct Q8_0, -np 8), once with one request at a time and once with eight concurrent requests. Same model file, same settings, greedy decoding. The generated text differed on 41 of the 100 problems. On problem 0 the two runs agreed for 213 characters; then one wrote "so she makes" and the other "so the total amount she makes", and the rest of the solution went its own way. The strict-match score moved from 0.83 to 0.82.
It is not just one engine:
- Thinking Machines Lab sent the same prompt 1,000 times at temperature 0 and got 80 different completions (Defeating Nondeterminism in LLM Inference, 2025).
- Yuan et al. measured up to 9% accuracy variation for DeepSeek-R1-Distill-Qwen-7B in BF16 when only the number of GPUs, the GPU type or the evaluation batch size changed (arXiv 2506.09501).
- The issue trackers of llama.cpp, Ollama, LM Studio and vLLM all have reports of temperature-0 output that changes with the prompt cache, speculative decoding, the number of slots or the GPU.
What temperature 0 and the seed actually fix
Temperature and the seed control one step: picking the next token from the probabilities the model computed. At temperature 0 the engine takes the most probable token, and the seed stops mattering.
Neither touches the probabilities themselves. If the logits come out slightly different, greedy decoding faithfully picks from slightly different numbers. So the real question is why the same model computes slightly different logits for the same input.
Cause 1: floating-point addition depends on order
A computer rounds after every floating-point addition, so (a + b) + c is not always equal to a + (b + c). Each logit is the end of millions of multiply-adds, and the order in which a GPU kernel adds them up decides the last bits.
A few lines of NumPy show it:
import numpy as np
rng = np.random.default_rng(0)
x = rng.standard_normal(4096).astype(np.float32)
def fsum(v): # add in float32, front to back
s = np.float32(0)
for t in v:
s = np.float32(s + t)
return s
print(fsum(x) == fsum(x[::-1])) # False: a different order changes the last digits
n = np.rint(x * 2**13).astype(np.int64) # the same values on an integer grid
print(n.sum() == n[::-1].sum()) # True: integer sums don't depend on order
Keep the last two lines in mind. They come back at the end.
Cause 2: the engine changes the order for you
GPU kernels are written for speed, and the fastest way to add numbers depends on the shape of the work. Your request stays the same, but the order changes whenever the engine changes how it computes:
- Batching and concurrent requests. The server computes your request together with whatever else it is doing. A matrix multiplication for one row and for eight rows can use different kernels with different reduction orders, so your row comes out slightly different depending on who else was in the batch. Thinking Machines showed that this lack of batch invariance is the main cause in inference servers.
- The prompt cache and chunk size. A prompt computed in one piece, in chunks, or restored from a cache goes through different kernel paths. llama.cpp's server documentation says so directly: logits are not guaranteed to be bit-for-bit identical for different batch sizes, so the prompt cache (on by default) can cause nondeterministic results.
- Speculative decoding. The verify step scores several draft tokens in one batch, which is a different computation from generating them one by one.
- KV cache quantization, the GPU, the driver and library versions. Each can change which kernels run.
- Kernels that are not deterministic from run to run, such as reductions built on atomic additions. They are rarer in inference, but they exist.
Why a tiny difference changes the whole answer
Usually the top token wins by a wide margin, and a difference in the last bits changes nothing. But a long answer passes through steps where two tokens are nearly tied. There, a one-bit difference can change which token wins. From then on the model is continuing a different text, so everything after that point is regenerated: different wording, sometimes a different line of reasoning, sometimes a different final answer.
The flips go both ways, which is why scores move far less than answers. In one of my measurements (600 gsm8k problems, DeepSeek-R1-Distill-Qwen-7B Q8_0, greedy), switching between two engines that run the same weights with different numerics changed the text of most answers and flipped correctness on 53 problems: 25 became wrong and 28 became right, a net change of 3 problems. So when you compare two models or two quantizations, compare the outputs item by item, not just the average score.
See it on your own machine
Start llama.cpp's server with several slots (use your own model path):
llama-server -m qwen2.5-7b-instruct-q8_0.gguf -ngl 99 -np 8 -c 16384
Then run this script. It sends one prompt alone, then again while seven other prompts run at the same time, and compares the two answers. It works with any OpenAI-compatible server; for Ollama or LM Studio, pass the URL and a model name as arguments.
# Send one prompt alone, then again while 7 other prompts run at the same time,
# and compare the two answers. Standard library only.
import json
import sys
import urllib.request
from concurrent.futures import ThreadPoolExecutor
URL = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8080/v1/chat/completions"
MODEL = sys.argv[2] if len(sys.argv) > 2 else None # Ollama and LM Studio need a model name
TARGET = "Explain why the sky is blue in three short paragraphs."
OTHERS = [
"Write a short story about a lighthouse keeper.",
"List ten facts about octopuses.",
"Explain how a hash map works.",
"Summarize the plot of Hamlet.",
"Describe how vaccines train the immune system.",
"Write four haiku about autumn.",
"Explain the difference between TCP and UDP.",
]
def ask(prompt):
body = {
"messages": [{"role": "user", "content": prompt}],
"temperature": 0,
"seed": 42,
"max_tokens": 400,
}
if MODEL:
body["model"] = MODEL
req = urllib.request.Request(URL, data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=600) as r:
return json.load(r)["choices"][0]["message"]["content"]
alone = ask(TARGET)
with ThreadPoolExecutor(max_workers=len(OTHERS) + 1) as pool:
futures = [pool.submit(ask, TARGET)] + [pool.submit(ask, p) for p in OTHERS]
together = futures[0].result()
if alone == together:
print("identical")
else:
n = min(len(alone), len(together))
i = next((k for k in range(n) if alone[k] != together[k]), n)
print(f"different: first difference at character {i}")
print("alone: ...", alone[max(0, i - 40):i + 60].replace("\n", " "))
print("together: ...", together[max(0, i - 40):i + 60].replace("\n", " "))
On a GPU build you will often see different and the character where the answers part. How often depends on the model, the GPU and the build, so try a few prompts. Sending the same prompt twice in a row can differ as well: the second request may reuse the cached prompt and take another path. llama.cpp lets you turn that off per request with "cache_prompt": false.
What you can do today
What to set depends on the engine, and each option costs something.
| Engine | What to set | What it costs |
|---|---|---|
| llama.cpp | One slot (-np 1), "cache_prompt": false, no speculative decoding |
Concurrency and the prompt cache. The maintainers do not aim for identical results across batch sizes; one slot without the prompt cache is the way they point to |
| Ollama |
seed and temperature: 0 in the options, OLLAMA_NUM_PARALLEL=1
|
Concurrency |
| vLLM |
VLLM_BATCH_INVARIANT=1 (beta; NVIDIA compute capability 8.0+, AMD Instinct, Intel XPU) |
Some speed; parts of tensor parallelism are not covered yet |
| SGLang |
--enable-deterministic-inference (FlashInfer, FA3 or Triton attention) |
Some speed (34% on average in its 2025 announcement); no radix cache with FlashInfer |
| Hugging Face transformers | Batch size 1, torch.use_deterministic_algorithms(True), CUBLAS_WORKSPACE_CONFIG=:4096:8
|
Speed; holds only on the same GPU and library versions |
Two things none of these give you: identical results on a different GPU, let alone between NVIDIA and AMD, and a way for someone else to check your numbers without your hardware.
Making the order irrelevant
The NumPy example holds the other half of the answer: integer sums don't depend on order. That suggests a different approach. Instead of forcing every kernel to add in one fixed order, make the order not matter.
That is what I built GeulPU to do (disclosure: I'm its author). It is a local engine for GGUF models (Qwen2.5, Qwen3, Llama 3, Mistral) on NVIDIA (CUDA) and other GPUs (Vulkan). It turns activations into blocks of 32 integers with a power-of-two scale, sums the products exactly on int8 tensor cores, and fixes every remaining rounding point in a written precision contract. The GPU output matches a NumPy CPU reference bit for bit, so batching, the prompt cache and speculative decoding stop mattering:
| Change | GeulPU | llama.cpp |
|---|---|---|
| 1 vs 8 concurrent requests (lm-eval gsm8k, 100 items) | identical | 41 of 100 generations changed (b11643) |
| Speculative decoding on vs off | identical tokens and logits | 2 of 8 greedy answers changed (b11721, ngram-simple) |
| Prompt cache restored from disk vs recomputed (54 agent requests) | identical (1,728 steps) | — |
| RTX 3080 Ti / 4070 Ti / 5070 / Radeon RX 9060 XT (exact mode) | identical (82,673 steps) | — |
The deterministic modes of vLLM and SGLang above cost speed. Against llama.cpp, GeulPU's default mode doesn't. On an RTX 4070 Ti with Qwen2.5-7B Q4_K_M, version 0.11.0 took 0.76–0.97× llama.cpp's time at every prompt length (512 to 65,536 tokens) and session depth (8k to 64k) measured. The llama.cpp build was b11827, using whichever of its f16 and q8_0 KV caches was faster. A recorded 54-request coding-agent session took 0.59× the time of llama.cpp's fastest setting, a 0.5B draft model. Exact mode, which also makes NVIDIA and AMD agree, is slower: 1.19–1.40× llama.cpp's time on prompts up to 8k tokens. GeulPU is Windows-first (Linux builds from source), runs dense models only for now, and is a solo project: https://github.com/wwoosshh/GeulPUv2. The study behind the "flips go both ways" numbers, with its raw data, is at https://github.com/wwoosshh/GeulPU.
Checklist
- Set the seed anyway. Not every engine or API treats temperature 0 as strictly greedy, and it costs nothing.
- Temperature 0 and a seed fix the sampling step, not the numbers it samples from.
- If answers change, look first at concurrency, the prompt cache and speculative decoding, then at the GPU, driver and library versions.
- For repeatable runs with a standard engine: one request at a time, no prompt cache, no speculation, the same hardware and versions; or the deterministic modes of vLLM and SGLang.
- Compare outputs item by item, not just the final score. Most flips cancel out in the average.
- If you need identical results with concurrency and caching on, or across GPUs, you need an engine built for it.
This article was drafted with an AI assistant and checked by the author.
Top comments (5)
the 41 of 100 on gsm8k is the number that stuck for me. i hit the same thing on my own box last month — same prompt, same q8_0 7b, different text depending on whether one other request was in flight. i kept blaming the quant before i read that llama.cpp does not guarantee bit-for-bit logits across batch sizes. for anything i need to reproduce (evals, regression tests) i now run single-slot with the prompt cache off and just accept the throughput hit. the one thing i would add to the checklist: the seed is worth setting anyway, because it is the only knob that does something at temperature 0 on some engines, and it costs nothing.
Thanks for sharing this. Blaming the quant is the natural first guess, but the file is the same in both runs, so the difference has to come from how the engine computes. Single slot with the prompt cache off is the right setting for evals and regression tests on a stock engine.
Good point on the seed. In strict greedy decoding it doesn't change anything, but not every engine or API treats temperature 0 as strictly greedy, so it's free insurance. I've added it to the checklist.
I've been blaming most run-to-run flips in my evals on the judge, and 41 of 100 changing between one request and eight concurrent ones says some of it is the model side moving under load. The 25 wrong and 28 right netting out to 3 is the bit I'd show anyone who trusts the average. Next time my scores wobble I'm rerunning with requests sent one at a time before I touch the judge.
Thanks. The judge is the obvious suspect, but there are two moving parts here, and you can separate them:
One catch with "one request at a time": eval prompts usually share a long few-shot prefix, so after the first item the server's prompt cache kicks in, and a cached prefix takes a different path than a fresh one. For a clean rerun, also send
"cache_prompt": false(in llama-server) and keep speculative decoding off.On the 25 vs 28: an exact McNemar test on those 53 flipped items gives p ≈ 0.78. So a score that moves by 3 can be pure noise even when 53 individual answers changed, which is why the per-item view matters.
The pair of numbers worth keeping is 41 of 100 generations changed and, in the 600-problem engine comparison, 25 problems turning wrong against 28 turning right. With 53 flips and a net change of 3, an exact sign test on those discordant items gives p of about 0.78, so the score difference is indistinguishable from none while most of the text differs. That argues for reporting paired flips with an interval instead of two averages. One thing the post leaves open: does the 41 of 100 appear with the prompt cache off and one slot versus eight, or does the cache path add flips of its own? Separating those would tell readers which setting to change first.