Temperature Zero, Fixed Seed, Different Answers

I was discussing reliability in AI agents when a familiar question came up: if you set an LLM's temperature to zero, doesn't that make it deterministic?

It sounds reasonable. Remove random sampling, always pick the most likely next token, and you get the same answer every time.

So I tried it on my laptop.

Same model. Same prompt. Temperature zero. Fixed seed. Five identical responses, then a different story on request six.

It happened on the very first execution of the script.

A small test with Ollama

I used llama3.1:8b through Ollama's local API, with these settings:

{
  "temperature": 0,
  "seed": 42,
  "num_ctx": 4096,
  "num_predict": 512
}

The prompt asked for a short story in Spanish about a programmer who finds a strange message in the server logs.

The Python script serializes the request body once and sends those exact bytes repeatedly. Requests are sequential, and no conversation history is passed between them.

It compares the generated text exactly and prints a short SHA-256 hash for each response. Timestamps and execution timings are excluded, since comparing those would produce meaningless differences.

Request six changed the story

Here are the response hashes from the captured output, with the diff omitted for readability (variantes counts the distinct responses seen so far):

01/30  hash=e8150452110a716b  variantes=1  fin=stop
02/30  hash=e8150452110a716b  variantes=1  fin=stop
03/30  hash=e8150452110a716b  variantes=1  fin=stop
04/30  hash=e8150452110a716b  variantes=1  fin=stop
05/30  hash=e8150452110a716b  variantes=1  fin=stop
06/30  hash=bf54c4fc8b524784  variantes=2  fin=stop
07/30  hash=e8150452110a716b  variantes=2  fin=stop
08/30  hash=e8150452110a716b  variantes=2  fin=stop
09/30  hash=e8150452110a716b  variantes=2  fin=stop

The first story featured the message "La respuesta está en el número 42" ("The answer is in the number 42").

The sixth response changed it to "La sombra se mueve en el rincón" ("The shadow moves in the corner") and developed a different plot.

Requests seven through nine returned to the original text.

This was a change in content, not just whitespace or formatting. The excerpt stops at request nine; it is not a completed 30-request report.

The complete script, settings and output diff are in this Gist.

It only uses Python's standard library. With Ollama running and the model installed, you can paste the script into a terminal and try it yourself.

What temperature zero actually controls

With temperature zero, Ollama falls back to greedy decoding: the model selects the token with the highest score. Given identical scores and consistent tie handling, that selection is deterministic.

The important qualification is given identical scores.

Those scores come from floating-point computation. Execution details such as GPU kernels, the order of operations and batching can slightly change the numbers. If two candidate tokens are close enough, a tiny difference can change which one wins.

Once a different token is selected, the next step receives a different prefix. From there, the rest of the answer can diverge substantially.

That is a plausible mechanism, not a diagnosis of my run. I did not trace Ollama's internals or record token scores, so I cannot attribute this result to a particular kernel, cache or scheduling decision.

For context, vLLM documents explicit batch invariance controls involving deterministic kernels and consistent numerical behavior. Reproducibility requires attention to the execution system, not just the decoding settings.

A fixed seed does not settle that problem either. It controls the random number generator used for sampling, not the arithmetic that produces the scores.

A counterexample, not a benchmark

The observation is simple: my local setup returned different text for identical requests specifying temperature=0 and seed=42.

It does not establish a general failure rate. Nor does it mean every model or runtime will behave this way.

Your installation might return the same answer every time. That would be a useful result too. This test checks repeatability under your conditions; it does not guarantee you will reproduce my divergence.

Repeatable is not the same as correct

There is another distinction worth keeping in mind when building agents: an answer can be perfectly repeatable and consistently wrong. Two different answers can also both be acceptable.

Exact reproducibility and task correctness need separate checks.

For my own systems, I want to measure repeatability where it matters, validate what the output means, and keep execution permissions and constraints in code.

Temperature zero is a setting. Reproducibility is a property of the system you actually run.