Temperature 0 Isn't Deterministic: Why Your LLM Still Drifts

Our CI had one test that failed roughly once a week. Same prompt, same model snapshot, temperature=0, seed pinned, snapshot assertion on the output string. Nothing in the diff touched it.

I did what every engineer does with a weekly flake: reran it, watched it go green, and blamed the network.

Then I got annoyed enough to loop the exact same request 500 times and diff the results. Twelve of them came back different. Not "slightly reworded" different in the wishy-washy sense — one of them classified a refund ticket as billing instead of fraud, which is a real behavior change from a byte-identical request.

Temperature 0 isn't deterministic. It never was. And the reason is not the model being creative behind your back.

TL;DR

• temperature=0 makes sampling greedy (always pick the top token), but it does not guarantee identical logits between runs, so the top token can change.
• The logits shift because floating point addition is not associative, and GPU kernels change their reduction order based on batch shape — which depends on who else is hitting the server at that millisecond.
• This is called a lack of batch invariance: your result depends on other people's requests sharing your batch.
• One flipped token near a near-tie gets amplified by autoregressive decoding, so a 1e-6 numeric wobble can rewrite an entire paragraph.
• Fix your tests, not the math: assert on parsed fields and invariants, pin model snapshots, and keep a flake budget. Bitwise reproducibility only exists on your own hardware at batch size 1.

Why isn't temperature 0 deterministic in LLMs?

Because greedy decoding is deterministic given identical logits, and you never get identical logits from a shared inference server.

Walk the pipeline. Your prompt becomes a matmul-heavy forward pass that ends in a vector of scores over the vocabulary. temperature=0 collapses the sampler to argmax. That part is genuinely deterministic — argmax over the same floats returns the same index every time.

The floats are the problem.

Floating point addition is not associative. (a + b) + c and a + (b + c) can differ in the last bits. On a CPU running one thread you would never notice, because the order never changes. On a GPU, a reduction (summing across a hidden dimension, softmax denominators, RMSNorm) gets split across blocks and the results combined. The way it splits depends on the tensor shape the kernel was handed.

And the shape depends on the batch.

What is batch invariance and why don't inference kernels have it?

Batch invariance means a single request produces the same output no matter what else is batched alongside it. Most production inference kernels do not have this property, and it's a performance decision, not a bug.

Serving stacks batch aggressively. Your request at 3pm on a Tuesday lands in a batch of 48 other requests. The same request at 4am lands in a batch of 3. Different batch size, different tiling strategy, different split-K choice inside the matmul, different reduction order, different last-bit rounding on your logits.

Nobody is randomizing anything. Every individual run is fully deterministic given its batch. You just don't control your batch, and you can't see it.

A few other things stack on top of this on real endpoints:

• Mixture-of-experts routing. With capacity limits, which tokens get routed to which expert can depend on the other tokens competing for that expert in the same batch. Your token's neighbors are strangers.
• Prefix caching. Whether your prompt hits a cached KV prefix changes where the compute boundary sits, which changes the reduction grouping for the tokens after it.
• Speculative decoding. Designed to be output-equivalent, but the verification path is a different numeric path, and equivalence holds only up to those same last bits.
• Silent fleet heterogeneity. Two GPU generations behind one endpoint means two kernel choices for identical input.

The seed parameter on the major APIs is documented as best effort for exactly this reason. It pins the sampler's RNG. It does not pin the arithmetic.

Why does one flipped bit change an entire answer?

Because decoding is autoregressive, so a single token swap changes the input to every token after it.

Most positions are not close calls. When the model writes "the capital of France is Paris", the gap between the top token and the runner-up is enormous, and a 1e-7 perturbation cannot touch it. Those tokens are effectively locked.

But a small share of positions are near-ties: "however" vs "but", { vs [", fraud vs billing on an ambiguous ticket. At those positions the top-2 gap is smaller than your numeric noise, and the argmax genuinely coin-flips.

Then the divergence compounds. Once the model has committed to "however", it is conditioning every future token on "however". You don't get a one-word diff. You get a different second half.

This is also why the flakes cluster on your hardest inputs. Ambiguous, low-confidence prompts are exactly where near-ties live. Your test suite's easy cases stay green forever and your nastiest edge case fails once a week, which reads like a haunting.

How do I make LLM output reproducible?

Mostly you don't, on a hosted API. You make your tests robust and reserve bitwise determinism for cases where you own the whole stack.

What actually worked for us:

1. Stop asserting on strings. Parse the output and assert on the fields you care about. Our fraud test now checks parsed.label in ALLOWED and parsed.label == "fraud", not the surrounding prose.
2. Pin the dated model snapshot, not the alias. Aliases roll under you. That's a separate source of drift that people misdiagnose as this one.
3. Shrink the output space for anything you assert on. A classifier that emits one enum token has far fewer near-tie positions than one that writes a paragraph and buries the label in it.
4. Measure the flip rate instead of pretending it's zero. Run your eval set N times, record the disagreement rate per item, and treat items that flip as low-confidence rather than as passing or failing.
5. Use K-of-N for decisions that matter. If three runs disagree on a refund, that's a routing signal to a human, not a bug to retry away.
6. Log the response fingerprint (system_fingerprint or equivalent) so you can tell "backend changed" apart from "we hit a near-tie."

If you truly need bitwise reproducibility, run locally: batch size 1, fixed library and driver versions, deterministic kernel flags, one GPU model. That is achievable, and it is still only reproducible against that box. Batch-invariant kernels exist and are being built into serving stacks, but they cost throughput, so you should assume you don't have them unless a vendor says so explicitly.

What to tell your team when temperature 0 isn't deterministic

Say it plainly: temperature=0 controls the sampler, not the arithmetic, and the arithmetic depends on server load.

The common misread is that the model is "still being creative" and someone proposes cranking temperature to -0 or adding top_p=0 (which does nothing here) or retrying until the output matches. I've watched a team add a retry loop that reran the model until it produced the expected string. That is not determinism. That is rejection sampling with extra steps and a much larger bill.

The honest framing: an LLM call is a statistical dependency, not a pure function. You wouldn't snapshot-test a call to a third-party ranking service and expect byte equality across a year. Same posture here.

So, is temperature 0 deterministic?

No. Temperature 0 isn't deterministic in practice, because it only makes token selection greedy while leaving the underlying computation free to vary. Floating point addition is non-associative, GPU kernels choose their reduction order based on batch shape, and batch shape on a hosted API depends on concurrent traffic you don't control. Identical prompts therefore produce slightly different logits, and at the small fraction of positions where the top two tokens are nearly tied, argmax flips — after which autoregressive decoding amplifies that single token into a different answer. You can get bitwise reproducibility on your own hardware at batch size 1 with pinned versions, or with genuinely batch-invariant kernels; on a shared endpoint you should design for a measured flip rate instead.

If your LLM test suite has one test that fails once a week, it isn't cursed. It's just standing on a near-tie.