You set temperature=0. You pinned the seed. Same prompt, same weights, greedy decoding, same everything you can put in a request body.
The reply comes back different, and the seed changed nothing at all.
Run it: https://dev48.infy.uk/ai/days/day75-inference-nondeterminism.html
What actually decides it
Nothing here simulates language. The engine simulates the arithmetic:
- a logit is a sum of many terms
- floating-point addition is not associative: (a+b)+c and a+(b+c) differ in general
- the order the machine adds them in is picked by the size of the batch you happened to land in
Different batch, different reduction tree, different last bit on a logit. When two candidate tokens are close, that last bit selects a different argmax — and greedy decoding faithfully propagates it into a different sentence.
batch of 8 -> logits summed in one order -> token A
batch of 32 -> same numbers, other order -> token B
temperature and seed are identical in both
Both controls aim at the wrong layer
| control | governs | where the divergence is |
|---|---|---|
| temperature | the sample | — |
| seed | the sampler | — |
| batch composition | the sum | ← here, upstream of both |
That is why turning temperature to 0 feels like it should work and does not. It removes the randomness you chose; it cannot touch the reassociation you did not.
Every number on the page is computed in your browser on load from integers, so the exact answer is exact and not an estimate of one.
801 verifier asserts, 174 in-page checks, 0 failures.
Top comments (0)