Temperature=0 Doesn't Mean Deterministic. Your Batch Size Does.

The problem

A team I worked with had an eval suite that called their model at temperature=0 and diffed the output against a golden completion, on the theory that greedy decoding meant identical input in, identical output out — perfect for catching regressions. Most of the time it worked. A few times a week, a completion that had passed for months would suddenly diff as "failed," with no code change, no prompt change, no model version bump. They spent real engineering hours convinced their harness had a race condition.

It didn't. The model was doing exactly what it was supposed to do. Their assumption about what temperature=0 guarantees was the bug.

Why it happens

temperature=0 makes exactly one thing deterministic: the sampling step. Instead of drawing from a probability distribution, the model always picks the highest-probability token — greedy decoding. That's it. That's the whole guarantee.

It says nothing about whether the forward pass that produces those probabilities is deterministic, and on a real GPU inference server, it usually isn't. Matmuls, attention, and normalization layers all involve summing large numbers of floating-point values, and floating-point addition is not associative — (a + b) + c can produce a different result than a + (b + c) once you're dealing with rounding error, and at model scale you're summing thousands of terms per reduction. The order those sums happen in is decided by the kernel's reduction strategy, which depends on how work gets scheduled across the GPU.

Here's the part that actually causes the flakiness: inference servers batch concurrent requests together to keep GPU utilization high. The exact set of requests riding alongside yours in a batch changes from one call to the next, based on real-time load — and that changes the batch size and shape the kernel sees, which changes the reduction order, which changes the tiny rounding error in the output logits. Normally this is invisible — a difference in the 6th decimal place of a logit doesn't change which token has the highest probability. But when the top two candidate tokens are extremely close, that noise is occasionally enough to flip the argmax. One flipped token early in a completion cascades into a completely different rest-of-sequence, because everything after it is now conditioned on a different prefix.

Your request didn't change. The model didn't change. The batch of other people's requests sharing the GPU with you at that exact millisecond did — and that was enough.

What to do about it

• Stop treating temperature=0 as a promise of bit-identical output. It's a promise about the sampling step only. Forward-pass determinism is a separate, much harder property.
• If you genuinely need bit-identical reproducibility — for compliance, security-sensitive audits, or debugging a subtle regression — know that it requires batch-invariant kernels: reduction implementations for matmul, attention, and normalization that are written to produce the same result regardless of batch composition. This is an active area of inference research, and early implementations that guarantee it have shown real throughput costs, with newer optimized versions cutting that overhead from roughly 60% down to around 34%. Determinism is purchasable, but it isn't free.
• For everyday eval and regression suites, design for tolerance instead of exact match: compare semantic equivalence, run each case N times and check for stability within a distribution, or flag "near-tie" completions (where the model's top-2 token probabilities were close) as inherently higher-variance rather than expecting them to byte-diff clean forever.
• Treat a flipped early token as a signal, not just noise — if your eval is riding a razor-thin margin between two candidate tokens, that's useful information about how confidently your prompt is actually constraining the model, independent of the flakiness it causes.

Key takeaways

• temperature=0 guarantees deterministic token selection (always argmax) — it does not guarantee a deterministic forward pass.
• Floating-point addition isn't associative; GPU reduction kernels sum in an order that depends on batch composition.
• Batch composition shifts with real-time server load from other concurrent requests, which is why identical calls can occasionally produce different completions.
• Bit-identical determinism is achievable with batch-invariant kernels, but at a real, currently non-trivial throughput cost.
• Build evals that tolerate token-level variance instead of assuming byte-for-byte reproducibility.