Set temperature=0, send the exact same prompt to Claude Opus 4.x or a self-hosted Qwen twice, and you can still get two different completions. Not subtly different — sometimes a whole different answer. Most engineers blame the sampler, add a seed, and move on when it doesn't help. The seed was never the problem. Temperature 0 does not make your LLM deterministic because the numerics of your request depend on what else is in the batch next to it on the server.
This is the batch-invariance problem, and it's one of the most misunderstood sources of nondeterminism in LLM inference.
TL;DR
- Temperature 0 = greedy decoding (pick the argmax logit). There is no RNG left to seed, so a seed can't fix the variation you see.
-
The real cause is floating-point non-associativity plus non-batch-invariant kernels.
(a + b) + c ≠ a + (b + c)in float32, and the order of those additions changes with server batch size. - Your request gets batched with other users' requests. Batch size shifts with load, which shifts the reduction order in matmul and attention, which occasionally flips one token's argmax. One flip cascades into a different completion.
- The fix is batch-invariant kernels: force matmul, attention, and RMSNorm to use the same reduction order regardless of batch size. This buys bitwise-reproducible output.
- It costs throughput, because you give up the adaptive tiling and split reductions that make batched inference fast.
Isn't temperature 0 supposed to be deterministic?
Logically, yes — and that's exactly why the bug is confusing. At temperature 0, decoding is argmax(logits) at every step. There is no sampling distribution to draw from, no random seed that matters. Given identical logits, you get identical tokens forever.
So if two runs diverge, the logits must have differed. And they did — by a few units in the last place (ULPs) of float32. Usually those tiny differences don't matter, because the top token wins by a comfortable margin. But when two candidate tokens are nearly tied, a ULP-level wobble flips the argmax. That flipped token changes the context for every subsequent step, and greedy decoding faithfully amplifies the divergence from there. One coin-flip near a tie, and the rest of the paragraph is different.
The question becomes: why do the logits wobble at all if the input is identical?
Why does floating-point math break determinism?
Floating-point addition is not associative. Reorder the terms of a sum and the rounding changes. You can watch it happen in a few lines:
import numpy as np
rng = np.random.default_rng(0)
x = rng.standard_normal(1_000_000).astype(np.float32)
fwd = np.float32(0.0)
for v in x:
fwd += v
bwd = np.float32(0.0)
for v in x[::-1]:
bwd += v
print(fwd, bwd, fwd - bwd)
# 251.0908 251.09048 3.0517578e-04
Same numbers, same operation, different order — different result. This isn't a bug in NumPy; it's IEEE 754. Every accumulation into a fixed-width float rounds, and rounding errors depend on the order you accumulate.
Now scale that up. A transformer forward pass is billions of these reductions: every matmul sums over the contraction dimension, every attention op sums over the KV sequence, every RMSNorm sums over the hidden dimension. If any of those reductions happens in a different order between two runs, the output logits differ by a few ULPs. That's your wobble.
What is batch invariance, and why does it matter?
A kernel is batch-invariant if the result computed for a given sequence is bitwise identical no matter what else is in the batch or how big the batch is. Row 3 of the output should be the same whether you ran a batch of 1 or a batch of 512.
Production inference kernels are not batch-invariant by default, and that's the crux. GPU kernels pick their execution strategy — tile sizes, how many thread blocks split a reduction, whether to use a split-K GEMM — based on the shape of the problem. Change the batch size and the shape changes, so the kernel picks a different strategy, so the reduction order changes, so the rounding changes.
Here's the part everyone misses: you don't control your batch size. On a shared endpoint, the server groups concurrent requests into a batch to keep the GPU busy. How many requests land in the same batch depends on traffic at that instant. Send your prompt when the server is idle, it runs in a batch of 1. Send it during a spike, it rides in a batch of 200. Different batch → different kernel path → different reduction order → different ULPs → occasionally a flipped token.
Your output depends on other people's traffic. That is the actual mechanism, and no seed parameter touches it.
Where does the reduction order actually change?
Three places dominate, and each needs its own fix.
Matmul. For efficiency, GEMM kernels sometimes split the contraction dimension across thread blocks and combine partial sums (split-K), often with atomic adds whose completion order is nondeterministic. Whether split-K activates, and how many splits, depends on the M/N/K shape — which moves with batch size. A batch-invariant matmul must commit to a single reduction layout regardless of shape, even when a different one would be faster.
Attention. FlashAttention-style kernels tile over the KV sequence and combine per-tile running softmax statistics. The number of KV splits (and thus the combine order) can depend on sequence length and batch. Long-context and paged/chunked prefill make this worse, because the KV dimension gets split differently depending on how the cache is laid out. Batch-invariant attention pins the split strategy so the online-softmax combination happens in a fixed order.
RMSNorm. The reduction over the hidden dimension is the easy one, but a naive data-parallel implementation that changes how it distributes rows across blocks under different batch sizes can still leak nondeterminism. Fix it the same way: one reduction layout, always.
Note what is not on this list: GPU concurrency and atomic race conditions per se. The forward pass is run-to-run deterministic on a fixed batch size on most stacks. The variance comes specifically from the batch size changing the kernel's plan. That reframing — from "GPUs are just nondeterministic" to "our kernels aren't batch-invariant" — is the whole insight, and it's the argument the Thinking Machines team laid out in their 2025 write-up on defeating nondeterminism in LLM inference.
How do you make an LLM deterministic?
You swap in batch-invariant implementations of the three kernels above so the reduction order never depends on batch size. In practice you don't hand-write these; you enable a mode.
At the framework level, the pattern looks like this — pin every source of nondeterminism, then use batch-invariant kernels:
import os
# 1) Deterministic algorithms where cuBLAS/cuDNN offer them.
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
import torch
torch.use_deterministic_algorithms(True)
torch.manual_seed(0)
# 2) Enable batch-invariant kernels (matmul / attention / RMSNorm).
# e.g. the batch-invariant-ops approach patches the relevant torch ops
# so the reduction layout is fixed regardless of batch shape.
from batch_invariant_ops import set_batch_invariant_mode
set_batch_invariant_mode(True)
# 3) Greedy decoding: no sampling entropy on top.
gen_kwargs = dict(do_sample=False, temperature=0.0)
For serving stacks like vLLM, the equivalent is a deterministic/batch-invariant execution mode plus a fixed attention backend — check the current docs for the exact flags, since these are actively moving. The important properties to verify are: (a) the same prompt returns byte-identical output across repeated calls, and (b) it stays identical when you vary the concurrent load. If (a) holds but (b) fails, your kernels still aren't batch-invariant.
If you're on a hosted API like Claude or GPT-5.x, you can't install kernels. You get a seed parameter that constrains sampling but not the server-side batch numerics, which is why hosted endpoints document their output as "mostly" deterministic rather than guaranteed. For those, the pragmatic answer is different: don't depend on exact-string reproducibility. Assert on parsed structure and semantics, not on the literal token stream.
What does batch invariance cost?
Throughput, mostly. The kernels that make batched inference fast are fast precisely because they adapt their reduction strategy to the shape in front of them — split-K when K is large, more KV splits when the sequence is long. Freezing that strategy leaves performance on the table for the shapes where the adaptive choice would have won. In published experiments the batch-invariant path runs slower than the tuned default, though the gap narrows as the kernels mature. Whether it's worth it depends on why you need reproducibility.
The cases where it clearly pays off: on-policy RL where you need the sampler's numerics to match the trainer's, otherwise your "on-policy" updates are quietly off-policy; scientific or regulatory workloads that must reproduce a result exactly; and debugging, where a nondeterministic forward pass makes every bug a heisenbug. For a typical chat product, eat the variance and test on semantics instead.
Bottom line
Temperature 0 doesn't make your LLM deterministic because temperature 0 only removes sampling randomness — it doesn't remove floating-point non-associativity, and it doesn't stop your request from being batched with others. Your logits are computed by kernels whose reduction order shifts with batch size, and batch size shifts with server load, so a near-tied token occasionally flips and the completion diverges. The fix is batch-invariant kernels for matmul, attention, and RMSNorm, which give you bitwise reproducibility at some throughput cost. If you're on a hosted API, you can't force that, so stop asserting on exact strings and assert on meaning instead.
Top comments (0)