Why temperature=0 Still Gives Different Answers: Batch Invariance

Set temperature=0, send the same prompt twice, get two different answers. Most engineers blame "GPU nondeterminism" and move on. That explanation is wrong, and being wrong about it means you'll never fix it.

The real cause is batch invariance — or rather, the lack of it. Your request's logits depend on what other requests happened to be in the same batch, because the kernel picked a different reduction order for a different batch shape. Nothing about your request changed. The floating-point arithmetic did.

This is now visible at the API layer too: on Claude Opus 5, Sonnet 5, and Opus 4.7/4.8, temperature is no longer an accepted parameter at all — sending it returns a 400. Anthropic's own migration guidance says the quiet part out loud: temperature = 0 never guaranteed identical outputs on prior models either.

TL;DR

• Greedy decoding is deterministic; your serving stack is not. argmax over identical logits always returns the same token. The logits aren't identical run to run.
• Floating-point addition is non-associative. (a+b)+c != a+(b+c) in fp16/bf16. Every matmul, RMSNorm, and softmax is a reduction, and the summation order is chosen by kernel heuristics keyed to batch shape.
• Continuous batching makes batch shape depend on server load. Your request gets batched with whatever else arrived that millisecond, so the reduction tree — and the last bits of every logit — changes with unrelated traffic.
• A 1e-6 logit perturbation is harmless until two tokens are nearly tied. Then argmax flips, and autoregressive feedback amplifies one flipped token into a completely different answer.
• The fix is batch-invariant kernels (fixed split sizes, no atomics, one config per op) at a real throughput cost — or, for hosted APIs, accepting nondeterminism and testing semantically instead of byte-wise.

Why does temperature=0 still produce different outputs?

Because "deterministic sampling" and "deterministic inference" are different claims. temperature=0 makes token selection a pure function of the logit vector. It says nothing about whether the logit vector is reproducible.

Consider what the forward pass actually is: a very long chain of reductions. A single logit is a dot product over the hidden dimension. RMSNorm sums squares across the row. Attention normalizes over the KV sequence. Every one of these sums thousands of floating-point values.

And floating-point addition is not associative:

import numpy as np

x = (np.random.randn(4096) * 0.02).astype(np.float32)

seq   = np.float32(0.0)
for v in x: seq += v                     # strictly sequential

tree4 = x.reshape(4, -1).sum(axis=1, dtype=np.float32).sum(dtype=np.float32)
tree8 = x.reshape(8, -1).sum(axis=1, dtype=np.float32).sum(dtype=np.float32)

print(f"{seq:.10f}  {tree4:.10f}  {tree8:.10f}")
print("4-way vs 8-way differ:", tree4 != tree8)

Same numbers, same values, three answers differing in the last few bits. The only thing that changed was how the sum was partitioned. That partition is exactly what a GPU kernel picks at launch time.

What is batch invariance in LLM inference?

A kernel is batch-invariant if the result computed for a given row is bitwise identical regardless of what other rows are in the batch. Most production LLM kernels are not.

The reason is performance tuning. To saturate a GPU, a matmul kernel splits work across streaming multiprocessors. When the batch is small, there aren't enough output tiles to fill the device, so the kernel splits along the reduction dimension instead (split-K) and combines partial sums afterward. Larger batches don't need the split. Which strategy runs — and how many splits — is chosen by a heuristic or an autotuner reading the tensor shapes.

Same for attention decode kernels: FlashDecoding-style implementations partition the KV cache into chunks, run online softmax per chunk, then rescale and merge. The number of KV splits is commonly derived from batch size, head count, and sequence length so the kernel fills all SMs. Different batch → different split count → different merge order → different last bits.

Note what's not on this list: race conditions. Individual kernels are typically run-to-run deterministic for fixed inputs and fixed launch config. Atomics-based split-K reductions are a real second source of nondeterminism, but they're the smaller problem. The dominant one is that your batch composition is set by other people's traffic.

Additional shape-dependent sources worth knowing about:

• Prefix caching. A cache hit reuses KV computed under a different chunking than a cache miss recomputes. Same prompt, different numerics, before you've sampled a single token.
• Chunked prefill boundaries. Where a long prompt gets sliced depends on scheduler state.
• Tensor parallelism degree. A different TP size means a different all-reduce tree.
• MoE routing. Expert assignment and per-expert batching shift with batch composition, which is why MoE models feel noticeably flakier than dense ones.

How does a 1e-6 logit difference change the whole answer?

It usually doesn't. Argmax is robust when the top-1 margin is comfortable. The failure mode is narrow but structural.

Define the margin at step t as logit[top1] - logit[top2]. Numerical noise on the order of 1e-6 relative flips the decision only when the margin falls below that noise floor. On confident tokens — punctuation, the second half of a word, a memorized fact — the margin is enormous. On genuinely uncertain tokens — a synonym choice, a formatting decision, whether to open a code block — the margin can be near zero.

Two properties make this bite in production:

1. Ties are not rare over long outputs. A per-token flip probability of even 0.1% compounds to roughly a 40% chance of at least one divergence across 500 tokens.
2. Divergence is absorbing. One different token becomes part of the context for every subsequent token. There's no reconvergence mechanism. A single flipped connective can send an agent down a different tool call.

That's the whole story: a last-bit arithmetic difference, amplified by argmax at a near-tie, amplified again by autoregressive feedback.

How do I measure nondeterminism in my own stack?

Don't argue about it — quantify it. The metric you want is divergence rate under load, plus the index of first divergence. This probe works against any endpoint, hosted or self-hosted:

import asyncio, hashlib
from collections import Counter

async def one(client, prompt, model):
    r = await client.messages.create(
        model=model, max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    return "".join(b.text for b in r.content if b.type == "text")

async def probe(client, prompt, model, n=64, concurrency=16):
    sem = asyncio.Semaphore(concurrency)
    async def guarded():
        async with sem:
            return await one(client, prompt, model)
    outs = await asyncio.gather(*(guarded() for _ in range(n)))

    counts = Counter(hashlib.sha256(o.encode()).hexdigest()[:12] for o in outs)
    base = outs[0]
    firsts = [
        next((i for i, (a, b) in enumerate(zip(base, o)) if a != b), None)
        for o in outs[1:]
    ]
    firsts = [f for f in firsts if f is not None]

    print(f"unique outputs : {len(counts)}/{n}")
    print(f"modal share    : {counts.most_common(1)[0][1] / n:.1%}")
    if firsts:
        firsts.sort()
        print(f"first divergence char (min/median): {firsts[0]} / {firsts[len(firsts)//2]}")

Run it twice: once at concurrency=1, once at concurrency=32. If unique-output count climbs with concurrency, you have a batch-invariance problem, not a sampling problem. On a self-hosted stack, also run it with prefix caching on and off — the delta tells you how much of your instability is cache-boundary numerics.

How do I make LLM inference reproducible?

If you control the server, you have three tiers of fix, in increasing order of cost.

Tier 1 — pin the shape. Batch size 1, one replica, fixed TP degree, prefix caching off, CUDA graphs off. This is what your reproducibility test suite should run against, not production.

vllm serve <model> \
  --max-num-seqs 1 \
  --tensor-parallel-size 1 \
  --no-enable-prefix-caching \
  --enforce-eager

Throughput will be terrible. That's fine — this configuration exists to make regression diffs meaningful, not to serve users.

Tier 2 — batch-invariant kernels. Replace the shape-sensitive ops with versions whose reduction structure is fixed. The core rule is to make the split a function of a constant chunk size rather than a target split count: process the reduction dimension in fixed 256-element chunks and combine in a fixed order, so a row's reduction tree is identical whether it ships alone or with 63 neighbors. Concretely that means data-parallel RMSNorm (one block per row, no cross-block reduce), a single matmul tile configuration with no atomic split-K, and an attention kernel with a fixed KV chunk size that treats cached and freshly-computed KV identically.

Thinking Machines published this analysis and a batch_invariant_ops library in late 2025; vLLM has since grown a batch-invariant mode behind an environment flag (check your build — the flag name and coverage have moved). The honest tradeoff: the naive deterministic path was substantially slower, and an optimized attention kernel recovered much of the gap without fully closing it. You are trading throughput for bitwise reproducibility. Buy it deliberately, for eval and RL-rollout paths, not for your whole fleet.

Tier 3 — accept it, and change what you assert. Which brings us to hosted APIs.

What about hosted APIs like Claude and GPT-5?

You have zero control over batch composition on a hosted endpoint, so bitwise reproducibility is not on the menu. The API surface now reflects this. Frontier Claude models (Opus 5, Sonnet 5, Opus 4.8/4.7) reject temperature, top_p, and top_k outright — the documented guidance is to steer behavior with prompting instead. Anthropic's API has never exposed a seed. OpenAI's seed plus system_fingerprint is explicitly best-effort, and the fingerprint exists precisely to tell you when the backend changed underneath you.

So stop writing tests that hash the completion. Write these instead:

• Structured output, field-level assertions. Constrain the response to a schema and assert on the fields that matter. Prose varies; {"risk": "high"} shouldn't.
• Divergence rate as an SLO, not a bug. Measure it, set a threshold, alert when it moves. A jump in divergence rate is a genuine signal that something changed on the provider side.
• Never cache on output hash. Key caches on the request, not the response.
• Budget for it in evals. Run n samples per item and report a confidence interval. A 1-point accuracy difference between two prompts, measured at n=1, is noise.

The short answer

temperature=0 isn't deterministic because greedy decoding only guarantees the same token for the same logits — and your logits are not reproducible. Floating-point addition is non-associative, GPU kernels choose their reduction order from batch shape, and continuous batching makes that shape depend on unrelated concurrent traffic. The result is a last-bit perturbation that flips argmax whenever two candidate tokens are nearly tied, and autoregressive decoding turns one flipped token into a completely different response. Fixing it requires batch-invariant kernels with fixed reduction structure, at a real throughput cost — an investment that makes sense for evals, regression tests, and RL rollouts. Everywhere else, and on every hosted API, the right move is to measure divergence rate and assert on meaning instead of bytes.