Is temperature 0 deterministic? How does your team handle LLM randomness?

Friday AI Question. This one comes up the first time a test that passed yesterday fails today with the exact same prompt:

How do you handle LLM randomness in your app and your tests?

• A. Temperature 0. Set temperature to 0 and assume same input = same output.
• B. Temperature 0 + a fixed seed. Pin the seed and every other parameter, and watch the provider's backend fingerprint.
• C. Record and replay. Store the output once and reuse it wherever you need the exact same answer (audits, fixtures, cached responses).
• D. Accept the variance. Run each eval case several times and assert a pass rate instead of an exact string.

Drop your letter in the comments. Here's what the sources actually say, and where I land.

Temperature 0 is not a determinism switch

The intuition behind A is reasonable: at temperature 0 the model should always pick the most likely next token, so the output should be fixed. In practice it isn't.

• Anthropic's API reference says it directly: "Note that even with temperature of 0.0, the results will not be fully deterministic." The same reference now marks temperature as deprecated: models released after Claude Opus 4.6 don't support setting it, and any value other than 1.0 is rejected with a 400 error.
• OpenAI's GPT-5.2 guide says temperature, top_p and logprobs are only supported when reasoning effort is set to none. Requests to GPT-5.2 or GPT-5.1 with any other reasoning effort, or to older GPT-5 models such as gpt-5, gpt-5-mini or gpt-5-nano, that include these fields raise an error.

So on several current models, "just set temperature 0" isn't even an option you can pick.

Why the same request gives different answers

Thinking Machines Lab (Horace He and colleagues, Defeating Nondeterminism in LLM Inference, Sep 2025) dug into this. Their experiment: 1,000 completions from Qwen/Qwen3-235B-A22B-Instruct-2507 at temperature 0, same prompt ("Tell me about Richard Feynman"), 1,000 tokens each. Result: 80 unique completions, with the most common one appearing 78 times. Every completion was identical for the first 102 tokens; at token 103, 992 completions said "Queens, New York" and 8 said "New York City".

Their explanation is not the usual "GPUs are parallel, floating point is messy" story. The forward pass is run-to-run deterministic for a fixed batch, but most kernels are not batch-invariant: your request's numbers depend on how many other requests the server batched with it. Server load changes the batch size, so from your side the output changes. With batch-invariant kernels they got 1,000 identical completions out of 1,000, at a cost: in their unoptimized vLLM test, 1,000 sequences took 26 s by default versus 55 s deterministic (42 s with an improved attention kernel).

That's a fix for people who run their own inference. If you call a hosted API, you don't control the batch.

A. Temperature 0 and assume

Good: less variety in outputs, which is often what you want for extraction and classification.

Bad: it reduces variance; it doesn't remove it. Tests that compare exact strings will flake, and on some newer models the parameter is rejected anyway.

B. Temperature 0 + seed

Good: OpenAI's cookbook on the seed parameter shows it makes outputs much more consistent. Their advice: same seed, same parameters, and check system_fingerprint, which changes when the backend configuration changes.

Bad: the same cookbook says determinism "is not guaranteed" and that there is "a small chance that responses differ even when request parameters and system_fingerprint match". Use it to reduce noise, not as a contract.

C. Record and replay

Good: the only option on this list that gives you the exact same text every time, because you're not calling the model again. Right for audit trails ("what did the bot tell this customer?"), regression fixtures for code downstream of the LLM, and caching answers you've already approved.

Bad: it tests your code, not the model. A recorded answer says nothing about what the model will say tomorrow.

D. Accept the variance, test with a pass rate (my pick for quality)

Good: it matches how the system really behaves. Run each eval case several times, check the property you care about (valid JSON, right label, cites the right document) rather than the exact wording, and fail the build when the pass rate drops.

Bad: more calls per test run, so more cost and time. Keep the "run it N times" set small and focused on the cases that matter.

Where I land

D for testing quality, C for anything that must be exactly repeatable. Use A or B as a noise reducer where the model supports it, never as a guarantee. And if your app breaks when the model words the same answer differently, the fix is usually in your parsing (structured outputs, validators), not in the sampler.

Rule of thumb: low temperature to reduce noise, store outputs to repeat them, sample several times to test them.

References

• Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference" (Horace He et al., Sep 2025): https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
• Anthropic API reference, Messages (temperature note and deprecation): https://platform.claude.com/docs/en/api/cli/messages/create
• OpenAI, "Using GPT-5.2" (parameter compatibility): https://developers.openai.com/api/docs/guides/latest-model/gpt-5.2
• OpenAI Cookbook, "How to make your completions outputs reproducible with the new seed parameter": https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter

So, A, B, C or D? And has a temperature-0 test ever flaked on you?