DEV Community: Moca

Why Your Local LLM Gives Different Answers at Temperature 0.

Moca — Fri, 23 Oct 2026 05:57:30 +0000

You set temperature to 0, fix the seed, send the same prompt twice, and get two different answers. This post explains where the difference comes from, how to see it on your own machine, and what each engine offers if you need answers that repeat.

The symptom

One measured case: I ran lm-evaluation-harness on gsm8k (100 problems) against llama.cpp's server (build b11643, Qwen2.5-7B-Instruct Q8_0, -np 8), once with one request at a time and once with eight concurrent requests. Same model file, same settings, greedy decoding. The generated text differed on 41 of the 100 problems. On problem 0 the two runs agreed for 213 characters; then one wrote "so she makes" and the other "so the total amount she makes", and the rest of the solution went its own way. The strict-match score moved from 0.83 to 0.82.

It is not just one engine:

Thinking Machines Lab sent the same prompt 1,000 times at temperature 0 and got 80 different completions (Defeating Nondeterminism in LLM Inference, 2025).
Yuan et al. measured up to 9% accuracy variation for DeepSeek-R1-Distill-Qwen-7B in BF16 when only the number of GPUs, the GPU type or the evaluation batch size changed (arXiv 2506.09501).
The issue trackers of llama.cpp, Ollama, LM Studio and vLLM all have reports of temperature-0 output that changes with the prompt cache, speculative decoding, the number of slots or the GPU.

What temperature 0 and the seed actually fix

Temperature and the seed control one step: picking the next token from the probabilities the model computed. At temperature 0 the engine takes the most probable token, and the seed stops mattering.

Neither touches the probabilities themselves. If the logits come out slightly different, greedy decoding faithfully picks from slightly different numbers. So the real question is why the same model computes slightly different logits for the same input.

Cause 1: floating-point addition depends on order

A computer rounds after every floating-point addition, so (a + b) + c is not always equal to a + (b + c). Each logit is the end of millions of multiply-adds, and the order in which a GPU kernel adds them up decides the last bits.

A few lines of NumPy show it:

import numpy as np

rng = np.random.default_rng(0)
x = rng.standard_normal(4096).astype(np.float32)

def fsum(v):                      # add in float32, front to back
    s = np.float32(0)
    for t in v:
        s = np.float32(s + t)
    return s

print(fsum(x) == fsum(x[::-1]))   # False: a different order changes the last digits

n = np.rint(x * 2**13).astype(np.int64)   # the same values on an integer grid
print(n.sum() == n[::-1].sum())           # True: integer sums don't depend on order

Keep the last two lines in mind. They come back at the end.

Cause 2: the engine changes the order for you

GPU kernels are written for speed, and the fastest way to add numbers depends on the shape of the work. Your request stays the same, but the order changes whenever the engine changes how it computes:

Batching and concurrent requests. The server computes your request together with whatever else it is doing. A matrix multiplication for one row and for eight rows can use different kernels with different reduction orders, so your row comes out slightly different depending on who else was in the batch. Thinking Machines showed that this lack of batch invariance is the main cause in inference servers.
The prompt cache and chunk size. A prompt computed in one piece, in chunks, or restored from a cache goes through different kernel paths. llama.cpp's server documentation says so directly: logits are not guaranteed to be bit-for-bit identical for different batch sizes, so the prompt cache (on by default) can cause nondeterministic results.
Speculative decoding. The verify step scores several draft tokens in one batch, which is a different computation from generating them one by one.
KV cache quantization, the GPU, the driver and library versions. Each can change which kernels run.
Kernels that are not deterministic from run to run, such as reductions built on atomic additions. They are rarer in inference, but they exist.

Why a tiny difference changes the whole answer

Usually the top token wins by a wide margin, and a difference in the last bits changes nothing. But a long answer passes through steps where two tokens are nearly tied. There, a one-bit difference can change which token wins. From then on the model is continuing a different text, so everything after that point is regenerated: different wording, sometimes a different line of reasoning, sometimes a different final answer.

The flips go both ways, which is why scores move far less than answers. In one of my measurements (600 gsm8k problems, DeepSeek-R1-Distill-Qwen-7B Q8_0, greedy), switching between two engines that run the same weights with different numerics changed the text of most answers and flipped correctness on 53 problems: 25 became wrong and 28 became right, a net change of 3 problems. So when you compare two models or two quantizations, compare the outputs item by item, not just the average score.

See it on your own machine

Start llama.cpp's server with several slots (use your own model path):

llama-server -m qwen2.5-7b-instruct-q8_0.gguf -ngl 99 -np 8 -c 16384

Then run this script. It sends one prompt alone, then again while seven other prompts run at the same time, and compares the two answers. It works with any OpenAI-compatible server; for Ollama or LM Studio, pass the URL and a model name as arguments.

# Send one prompt alone, then again while 7 other prompts run at the same time,
# and compare the two answers. Standard library only.
import json
import sys
import urllib.request
from concurrent.futures import ThreadPoolExecutor

URL = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8080/v1/chat/completions"
MODEL = sys.argv[2] if len(sys.argv) > 2 else None  # Ollama and LM Studio need a model name
TARGET = "Explain why the sky is blue in three short paragraphs."
OTHERS = [
    "Write a short story about a lighthouse keeper.",
    "List ten facts about octopuses.",
    "Explain how a hash map works.",
    "Summarize the plot of Hamlet.",
    "Describe how vaccines train the immune system.",
    "Write four haiku about autumn.",
    "Explain the difference between TCP and UDP.",
]

def ask(prompt):
    body = {
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
        "seed": 42,
        "max_tokens": 400,
    }
    if MODEL:
        body["model"] = MODEL
    req = urllib.request.Request(URL, data=json.dumps(body).encode(),
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=600) as r:
        return json.load(r)["choices"][0]["message"]["content"]

alone = ask(TARGET)
with ThreadPoolExecutor(max_workers=len(OTHERS) + 1) as pool:
    futures = [pool.submit(ask, TARGET)] + [pool.submit(ask, p) for p in OTHERS]
    together = futures[0].result()

if alone == together:
    print("identical")
else:
    n = min(len(alone), len(together))
    i = next((k for k in range(n) if alone[k] != together[k]), n)
    print(f"different: first difference at character {i}")
    print("alone:    ...", alone[max(0, i - 40):i + 60].replace("\n", " "))
    print("together: ...", together[max(0, i - 40):i + 60].replace("\n", " "))

On a GPU build you will often see different and the character where the answers part. How often depends on the model, the GPU and the build, so try a few prompts. Sending the same prompt twice in a row can differ as well: the second request may reuse the cached prompt and take another path. llama.cpp lets you turn that off per request with "cache_prompt": false.

What you can do today

What to set depends on the engine, and each option costs something.

Engine	What to set	What it costs
llama.cpp	One slot (`-np 1`), `"cache_prompt": false`, no speculative decoding	Concurrency and the prompt cache. The maintainers do not aim for identical results across batch sizes; one slot without the prompt cache is the way they point to
Ollama	`seed` and `temperature: 0` in the options, `OLLAMA_NUM_PARALLEL=1`	Concurrency
vLLM	`VLLM_BATCH_INVARIANT=1` (beta; NVIDIA compute capability 8.0+, AMD Instinct, Intel XPU)	Some speed; parts of tensor parallelism are not covered yet
SGLang	`--enable-deterministic-inference` (FlashInfer, FA3 or Triton attention)	Some speed (34% on average in its 2025 announcement); no radix cache with FlashInfer
Hugging Face transformers	Batch size 1, `torch.use_deterministic_algorithms(True)`, `CUBLAS_WORKSPACE_CONFIG=:4096:8`	Speed; holds only on the same GPU and library versions

Two things none of these give you: identical results on a different GPU, let alone between NVIDIA and AMD, and a way for someone else to check your numbers without your hardware.

Making the order irrelevant

The NumPy example holds the other half of the answer: integer sums don't depend on order. That suggests a different approach. Instead of forcing every kernel to add in one fixed order, make the order not matter.

That is what I built GeulPU to do (disclosure: I'm its author). It is a local engine for GGUF models (Qwen2.5, Qwen3, Llama 3, Mistral) on NVIDIA (CUDA) and other GPUs (Vulkan). It turns activations into blocks of 32 integers with a power-of-two scale, sums the products exactly on int8 tensor cores, and fixes every remaining rounding point in a written precision contract. The GPU output matches a NumPy CPU reference bit for bit, so batching, the prompt cache and speculative decoding stop mattering:

Change	GeulPU	llama.cpp
1 vs 8 concurrent requests (lm-eval gsm8k, 100 items)	identical	41 of 100 generations changed (b11643)
Speculative decoding on vs off	identical tokens and logits	2 of 8 greedy answers changed (b11721, ngram-simple)
Prompt cache restored from disk vs recomputed (54 agent requests)	identical (1,728 steps)	—
RTX 3080 Ti / 4070 Ti / 5070 / Radeon RX 9060 XT (exact mode)	identical (82,673 steps)	—

The deterministic modes of vLLM and SGLang above cost speed. Against llama.cpp, GeulPU's default mode doesn't. On an RTX 4070 Ti with Qwen2.5-7B Q4_K_M, version 0.11.0 took 0.76–0.97× llama.cpp's time at every prompt length (512 to 65,536 tokens) and session depth (8k to 64k) measured. The llama.cpp build was b11827, using whichever of its f16 and q8_0 KV caches was faster. A recorded 54-request coding-agent session took 0.59× the time of llama.cpp's fastest setting, a 0.5B draft model. Exact mode, which also makes NVIDIA and AMD agree, is slower: 1.19–1.40× llama.cpp's time on prompts up to 8k tokens. GeulPU is Windows-first (Linux builds from source), runs dense models only for now, and is a solo project: https://github.com/wwoosshh/GeulPUv2. The study behind the "flips go both ways" numbers, with its raw data, is at https://github.com/wwoosshh/GeulPU.

Checklist

Set the seed anyway. Not every engine or API treats temperature 0 as strictly greedy, and it costs nothing.
Temperature 0 and a seed fix the sampling step, not the numbers it samples from.
If answers change, look first at concurrency, the prompt cache and speculative decoding, then at the GPU, driver and library versions.
For repeatable runs with a standard engine: one request at a time, no prompt cache, no speculation, the same hardware and versions; or the deterministic modes of vLLM and SGLang.
Compare outputs item by item, not just the final score. Most flips cancel out in the average.
If you need identical results with concurrency and caching on, or across GPUs, you need an engine built for it.

This article was drafted with an AI assistant and checked by the author.

Geul: A Native Compiled Language Designed Around Korean Grammar (SOV, Particle Binding, Self-Hosting)

Moca — Sat, 11 Apr 2026 10:11:22 +0000

Every non-English programming language I've seen does the same thing: translate
the keywords. if, while, return become local words, but the grammar stays
identical to C or Python.

That never felt right to me for Korean.

Korean isn't just "English with different words." It's a fundamentally different
grammatical structure — SOV word order, agglutinative morphology, and a particle
system that carries meaning independently of word position.

So I built 글 (Geul) — a natively compiled, self-hosting programming language
that actually models Korean grammar instead of just wearing it as a skin.

The Core Problem: Korean is SOV, Not SVO

English (and most programming languages) follow SVO order:

Subject Verb Object → add(5, 3)

Korean follows SOV:

Subject Object Verb → 나는 밥을 먹다 (I rice eat)

But more importantly, Korean uses grammatical particles to mark roles:

Particle	Role
을/를	Object marker
에	Location / target
로	Direction / method
이/가	Subject marker

This means word order is flexible in Korean — the particles carry the meaning,
not the position.

What This Looks Like in Code

In Geul, a function call follows Korean SOV structure:

5를 3에 더하다

Breaking it down:

5를 — 5 marked with 를 (object marker)
3에 — 3 marked with 에 (target marker)
더하다 — "to add" (verb form)

Compare this to the equivalent in C:

add(5, 3)

The function declaration mirrors this structure:

[정수 왼쪽을 정수 오른쪽에 더하기]는 -> 정수 {
    반환 왼쪽 + 오른쪽.
}

왼쪽을 declares that the parameter 왼쪽 (left) expects the 를/을 particle.

오른쪽에 declares that 오른쪽 (right) expects the 에 particle.

The parser resolves argument-to-parameter binding through particle matching —
not positional order.

The Morpheme Problem

Korean verbs conjugate heavily. The call site uses a conjugated verb form, but
the function is declared with a nominalized (noun) form:

Form	Korean	Meaning
Function name	`더하기`	"adding" (noun form)
Call site	`더하다`	"to add" (verb form)

The compiler needs to know that 더하다 → 더하기 at parse time.

To solve this, I built a 3-level recursive morpheme analyzer that strips
Korean verb conjugation suffixes and maps call-site verb forms to their
declared function names automatically.

Compiler Architecture

Geul has two backends:

.글 source
    │
    ▼
Morpheme Analysis   ← 3-level recursive affix splitting
    │
    ▼
Recursive Descent Parser   ← SOV + particle-aware
    │
    ▼
AST → 3-Address IR + Static Type Checking
    │
    ├──────────────────────┐
    ▼                      ▼
C Transpiler          x86-64 Native
(cl/gcc)              (PE direct write)
    │                      │
    ▼                      ▼
  .exe                   .exe
                    (zero external deps)

The native backend writes PE headers and x86-64 machine code directly —
no cl.exe, no gcc, no external assembler required.

Optimizations

Compare-branch fusion — cmp + jcc direct emit
Register pinning — loop variables pinned to callee-saved registers
Peephole — dead store elimination, redundant copy removal
Loop-invariant code motion — constant loads hoisted out of loops
Tail call optimization — self-recursion converted to jmp

Self-Hosting

In v0.5, Geul became self-hosting — the compiler is written in Geul and
compiles itself.

Verification: bootstrap binary → compile source → output binary SHA256
fixed-point matches. ✓

This was the hardest milestone. Getting a compiler to compile itself means
every language feature used in the compiler itself must be fully functional
and stable.

Performance

Measured on Intel Ultra 5 226V, 5-run median. C compiled with MSVC /O2.

Benchmark	Geul	C /O2	vs C	vs Python
Recursive fib(40)	464ms	321ms	1.45x	15.9x faster
Sieve of 1M primes	10ms	8ms	1.30x	16.7x faster
Bubble sort 30k	1564ms	564ms	2.77x	24.5x faster

The bubble sort gap (2.77x) is the most notable — nested loop register
allocation pressure is the likely cause. It's on the roadmap.

Quick Syntax Tour

Variables and types:

정수 나이 = 25.          (* int *)
실수 원주율 = 3.14159.   (* double *)
문자열 이름 = "홍길동".  (* string *)
참거짓 활성 = 참.        (* bool *)

String interpolation:

"{이름}은 {나이}세입니다\n"을 쓰다.

Variables are embedded directly with automatic type formatting.

Control flow:

값 > 10이면 {
    "크다\n"을 쓰다.
}
아니면 {
    "작다\n"을 쓰다.
}

(번호 < 10)동안 {
    번호를 증가하다.
}

Structs:

묶음 점은 정수 가로, 정수 세로.

점 위치.
위치의 가로 = 10.
위치의 세로 = 20.

Current State

✅ Windows 10/11 (Linux ELF planned)
✅ VS Code extension (syntax highlight, F5 run, LSP autocomplete)
✅ 9 stdlib modules, 62 functions
✅ Static type checking at compile time
✅ Self-hosting since v0.5
🔲 Package manager (planned)
🔲 Debug info / PDB (planned)
🔲 Error handling syntax (시도/실패)

Currently at v0.7.1 (18 releases since March 2026, MIT license).

Try It

No install needed — browser playground:

👉 https://geul-web.vercel.app/playground

Source:

👉 https://github.com/wwoosshh/geul-lang

Curious what approaches others have seen for non-English or non-SVO language
design — are there other languages that go beyond keyword translation?