Temperature 0 Isn't Deterministic: Batch Invariance in LLM Inference

My snapshot test had one job: send a fixed prompt at temperature=0, compare the answer to a saved string, fail on any diff. It passed on my laptop every time. In CI it failed on some runs and not others. Same prompt, same model, same seed, same everything. I spent an afternoon hunting a race condition in my own code before accepting the embarrassing truth: temperature 0 isn't deterministic, and the reason has nothing to do with sampling.

The reason is batch invariance, or rather the lack of it. Your request's output depends on who else was on the GPU at the same time.

TL;DR

• Temperature 0 means greedy decoding (always pick the highest-probability token). That part is deterministic. The logits feeding it are not.
• Floating-point addition isn't associative. (a + b) + c can differ from a + (b + c) in the last bits, and bf16 has only 7 explicit mantissa bits.
• GPU kernels for matmul, normalization and attention pick different reduction strategies depending on batch size. A server's batch size depends on load. So your logits change with traffic.
• A tiny logit wobble only matters at a near-tie between two tokens. But once one token flips, every following token is conditioned on different text, and the outputs diverge completely.
• Fixes: don't string-compare LLM outputs in tests, record and replay responses, and if you self-host, use a batch-invariant inference mode and pay the throughput cost.

Why isn't temperature 0 deterministic?

Temperature 0 isn't deterministic because the sampler is deterministic but the numbers it reads are not. Greedy decoding does argmax(logits). If the logits for the same prompt come out slightly different on two runs, and the top two tokens were almost tied, argmax picks different tokens.

The common folk explanation is "GPUs are parallel, threads race, atomics add in random order." That's partly true for some kernels, but it's not the main story for inference. Most kernels in a modern LLM forward pass are run-to-run deterministic: give them the exact same input tensor twice and you get bit-identical output twice. Thinking Machines made this point well in their public write-up on defeating nondeterminism in LLM inference.

The catch is "the exact same input tensor." On an inference server, your prompt is never alone.

What does floating-point associativity have to do with it?

Everything. Run this in any Python shell:

>>> (0.1 + 0.2) + 0.3
0.6000000000000001
>>> 0.1 + (0.2 + 0.3)
0.6

Same three numbers, different order, different answer. Each addition rounds to the nearest representable value, and rounding errors depend on what you've accumulated so far.

That's float64, with 52 mantissa bits. LLM inference mostly runs in bf16, which has 7 explicit mantissa bits. That's roughly 2 to 3 significant decimal digits. A dot product over a 4096-dimensional hidden state is thousands of additions, and the order you sum them in leaves fingerprints in the result.

So the real question isn't "is the GPU random?" It's "does the kernel always add things up in the same order?"

Why does batch size change my LLM's output?

Because kernels choose their reduction strategy based on tensor shape, and batch size is part of the shape. When a server has 3 requests in flight, your prompt runs in a matmul with 3 rows. When it has 40, it runs with 40 rows. The math is "the same" for your row, but the kernel isn't.

Here's where it bites, layer by layer:

1. Matmuls (split-K). With a small batch there aren't enough output tiles to keep every GPU core busy. So kernel libraries split the reduction dimension (K) across multiple cores and add the partial sums at the end. With a big batch, there's enough parallelism already, so no split. Different split, different summation order, different bits.

2. RMSNorm / LayerNorm. Same idea. A normalization kernel can assign one core per row when there are many rows, and split a row's reduction across cores when there are few. Reduction order changes with batch size.

3. Attention with a KV cache. This one is sneaky. During decoding, attention reduces over every cached key. Kernels in the FlashDecoding family split the KV sequence into chunks to parallelize, and the chunk count often depends on how much work is in flight. Chunked prefill makes it worse: your prompt might be processed as one chunk in one run and three chunks in another, so the attention sums happen in a different grouping.

You can see the first effect directly with PyTorch on a GPU:

import torch

torch.manual_seed(0)
A = torch.randn(2048, 4096, device="cuda", dtype=torch.bfloat16)
B = torch.randn(4096, 4096, device="cuda", dtype=torch.bfloat16)

row_alone    = A[:1] @ B          # batch of 1
row_in_batch = (A @ B)[:1]        # same row, batch of 2048

print((row_alone - row_in_batch).abs().max())

Mathematically that should print zero. On most GPU and library combinations it doesn't, because the batch-of-1 and batch-of-2048 calls take different kernel paths. Run each line twice and each one is stable against itself. That's the whole bug in four lines: run-to-run deterministic, not batch invariant.

Now connect it to a server. Your request's batch size is set by other users' traffic, which you don't control and can't see. From your side, that's indistinguishable from randomness.

Why does a tiny logit difference change the whole answer?

A tiny logit difference only changes the answer at a near-tie, but long generations hit near-ties constantly, and autoregressive decoding turns one flipped token into a completely different continuation.

Most tokens aren't close calls. After "The capital of France is", the top logit wins by a mile, and a last-bit wobble does nothing. But generations are full of genuine coin-flips: "However" vs "But", a comma vs a period, "5" vs "five", which of two equally valid list items to write first. When the top two logits are separated by less than the numerical noise, argmax becomes a function of server load.

And then the cascade. Token 47 flips from "However" to "But". Token 48 is now conditioned on a different prefix. So is every token after it. By token 300 you're reading a different essay. That's why the diff you see isn't "one character changed." It's "the second half of the answer is unrecognizable."

In the Thinking Machines experiment, they sampled the same prompt 1,000 times at temperature 0 on a vLLM-served Qwen model and got 80 unique completions. The completions agreed word for word for a long opening stretch, then split apart at a near-tie.

Do hosted APIs like OpenAI and Claude guarantee determinism?

No. Hosted LLM APIs run batched inference on shared hardware, so temperature 0 on a hosted API isn't deterministic either. Providers say so in their docs. OpenAI's seed parameter is documented as best-effort, and the system_fingerprint field exists to tell you when the backend configuration changed underneath you. Anthropic's docs note that even temperature 0 won't produce fully deterministic results.

Also remember the backend isn't frozen. Hardware pools, kernel versions and serving configs change over time. Even a perfectly batch-invariant server can give you a different answer next month because the model was redeployed on a different GPU type.

How do I make LLM outputs reproducible?

Pick based on what you actually need.

1. Stop string-comparing outputs in tests. This was my bug. Assert on properties instead: valid JSON, required keys present, the extracted number is correct, the classification label is in the allowed set. If you need a semantic check, compare meaning, not bytes.

2. Record and replay. For integration tests, cache the raw response keyed by a hash of the request, and replay it. You're testing your code's handling of a response, not the provider's floating-point pipeline.

import hashlib, json, pathlib

CACHE = pathlib.Path(".llm_cassettes")

def cached_call(client, **req):
    key = hashlib.sha256(json.dumps(req, sort_keys=True).encode()).hexdigest()
    path = CACHE / f"{key}.json"
    if path.exists():
        return json.loads(path.read_text())
    resp = client.chat.completions.create(**req).model_dump()
    CACHE.mkdir(exist_ok=True)
    path.write_text(json.dumps(resp))
    return resp

3. Measure near-ties instead of guessing. If an OpenAI-compatible endpoint exposes logprobs, you can find the fragile tokens:

resp = client.chat.completions.create(
    model=MODEL, messages=msgs, temperature=0,
    logprobs=True, top_logprobs=2,
)
for t in resp.choices[0].logprobs.content:
    top, second = t.top_logprobs[0], t.top_logprobs[1]
    if top.logprob - second.logprob < 0.05:
        print(f"near-tie: {top.token!r} vs {second.token!r}")

Every line this prints is a place where your output can flip under load. If your prompt produces a near-tie on the exact token your parser cares about, that's a prompt problem worth fixing.

4. Treat evals as distributions. If a single run of your eval suite decides whether a prompt change shipped, you're partly measuring server traffic. Run each case several times and report the agreement rate alongside the score.

5. Self-hosting? Use batch-invariant kernels. This is the only true fix, and it lives in the inference engine. The idea: force every kernel to use one reduction strategy regardless of batch size. No shape-dependent split-K, a fixed split size for attention over the KV cache instead of a fixed split count, and a consistent reduction order in normalization. Thinking Machines released batch-invariant kernel ops, and inference engines including vLLM and SGLang have added deterministic or batch-invariant modes. Check your engine version's docs for the exact flag.

The cost is throughput. You're deliberately giving up the kernel choices that were optimized for each batch shape, so expect it to run slower than the default path. For RL training, where the sampler and trainer need to agree bit-for-bit, or for regulated workloads that need audit replay, that trade is worth it. For a chatbot, it usually isn't.

So, is temperature 0 deterministic?

No. Temperature 0 makes the sampling step deterministic by always choosing the top token, but the logits come from GPU kernels whose floating-point reduction order depends on batch size, and on a shared inference server batch size depends on everyone else's traffic. When two tokens are nearly tied, that last-bit noise flips the choice, and autoregressive decoding turns one flipped token into a completely different response. To get reproducible LLM outputs you need batch-invariant kernels on hardware you control. Everywhere else, design your tests and evals to expect variation instead of pretending it isn't there.

---

Written by the developer behind Preterview, an interview prep platform.