Why temperature=0 Isn't Deterministic: LLM Batch Invariance

Same weights. Same prompt. Same server. temperature=0. You run it twice and token 217 comes back different, and from there the two outputs have nothing in common. Nothing in your stack sampled anything. Your seed didn't matter because greedy decoding never drew a random number.

The culprit is not your code. It's that the floating-point reduction order inside your GEMM and attention kernels depends on how many other people's requests were in the batch alongside yours.

TL;DR

• Why temperature=0 isn't deterministic: greedy decoding is deterministic given the logits, but the logits aren't deterministic given the server. Kernels pick reduction/split strategies based on runtime batch shape and occupancy, and floating-point addition is not associative, so the last bits move.
• The perturbation is tiny (roughly 1e-6 to 1e-3 in logit units for a bf16 pipeline), but argmax is discontinuous. One near-tie flips, and after that the two trajectories are sampling from different contexts entirely.
• Main sources: split-K GEMM, split-KV attention (FlashDecoding), MoE expert grouping, chunked-prefill boundaries, prefix-cache hits, and tensor-parallel all-reduce order.
• Fix at the kernel level with batch-invariant kernels (fixed split counts, no atomics, fixed merge order) — real bitwise reproducibility, at a throughput cost. There is no fix through a hosted API.
• If you're doing RL, this is not cosmetic: sampler and trainer disagree on logprobs for the same tokens, so your "on-policy" GRPO/PPO step is quietly off-policy at step 0.

Why temperature=0 isn't deterministic: the short version

Greedy decoding is a pure function of the logit vector. So the question is whether the forward pass is a pure function of (weights, tokens). It isn't — it's a function of (weights, tokens, batch shape, occupancy, kernel selection).

A GPU matmul doesn't sum the K dimension in one thread. It splits the work, produces partials, and combines them. How it splits depends on the shape: with M=1 (one decode row) there's no row parallelism, so the library reaches for split-K to fill the SMs; with M=256 it doesn't need to. Different split → different summation order → different rounding.

In bf16, an element carries 8 explicit mantissa bits. Accumulation is usually in fp32, so end-to-end logit agreement lands somewhere around 1e-6 to 1e-3 absolute. That is invisible in the text — until it isn't.

Worth killing a common misdiagnosis: this is usually not nondeterministic atomics in the forward pass. Most inference kernels are run-to-run deterministic for a fixed shape. That's exactly why the bug feels haunted: rerun it alone at 3am and it reproduces fine, then it drifts under production traffic, because your batch composition is a function of other users.

Where does the batch dependence actually come from?

Five places, in rough order of how often they bite:

1. Split-K GEMM. Decode-time M equals the number of sequences currently decoding. That number is traffic. Kernel heuristics switch on it.
2. Split-KV attention. FlashDecoding chunks the KV cache and merges partial attention outputs with a log-sum-exp rescale. The number of chunks is picked from sequence length and available parallelism, so a long sequence alone on the GPU gets a different split than the same sequence sharing with 31 others. The merge is a sum of rescaled partials — order matters.
3. Chunked prefill. Whether token 2048 is computed at the tail of a prefill chunk or in a mixed prefill+decode batch changes the kernel path it goes through.
4. Prefix caching. A cache hit reuses KV computed under some earlier batch shape. Cache hit and cache miss are numerically different runs of the same prompt. This one surprises people because prefix caching is supposed to be semantically transparent — it is, to within rounding.
5. MoE routing. Per-expert GEMMs have M = number of tokens routed to that expert in this batch. Co-tenants change your expert's tile shape.

And one config-level source that isn't load-dependent but breaks reproducibility across deployments: tensor parallelism degree. TP=2 and TP=4 build different all-reduce trees, so they sum partial hidden states in a different order. Same model, same prompt, different numbers. Pin TP if you compare across clusters.

Why does a 1e-6 logit change flip a whole answer?

Because argmax is discontinuous, and because divergence compounds.

The flip only needs the top-2 logit gap to be smaller than the perturbation. Most positions are safe — the model is confident and the gap is >1 nat. But near-ties are not uniformly distributed. They cluster exactly where you'd expect: ", " vs " ", list-marker choices, synonym pairs, the first token after a heading, whether to open a code fence now or after one more sentence.

Once one token flips, the two runs no longer share a context. Everything after is a legitimately different generation. So measure divergence-free rate over N runs and the index of first divergence, not average per-token difference — the latter saturates immediately and tells you nothing.

How do I measure this on my own stack?

Run the same request repeatedly and look at two things: whether outputs diverge, and how thin the top-2 margins are. Run the probe at temperature=1 so the reported logprob differences are interpretable in logit units (log p_i − log p_j = z_i − z_j, since softmax's shift cancels).

import asyncio, collections
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="x")
PROMPT = "Summarize the tradeoffs of split-K GEMM in three sentences."

async def probe(load=0):
    """One measured request, optionally with `load` junk requests co-scheduled."""
    async def junk(i):
        await client.chat.completions.create(
            model="m", messages=[{"role": "user", "content": f"count to 200, run {i}"}],
            max_tokens=256, temperature=0)

    task = client.chat.completions.create(
        model="m", messages=[{"role": "user", "content": PROMPT}],
        max_tokens=256, temperature=0, logprobs=True, top_logprobs=2, seed=0)
    # co-tenants change the batch shape our request is computed in
    r, *_ = await asyncio.gather(task, *[junk(i) for i in range(load)])
    toks = [c.token for c in r.choices[0].logprobs.content]
    gaps = [c.top_logprobs[0].logprob - c.top_logprobs[1].logprob
            for c in r.choices[0].logprobs.content if len(c.top_logprobs) > 1]
    return toks, gaps

async def main():
    runs = [await probe(load=l) for l in (0, 0, 8, 32, 32)]
    base = runs[0][0]
    for i, (toks, gaps) in enumerate(runs):
        first = next((k for k, (a, b) in enumerate(zip(base, toks)) if a != b), None)
        thin = sum(g < 1e-3 for g in gaps)
        print(f"run{i}: first_divergence={first}  thin_margins(<1e-3)={thin}/{len(gaps)}")

asyncio.run(main())

first_divergence=None on every row means you're batch-invariant for these shapes. What you normally see instead is None for the two idle runs and an integer for the loaded ones — that's the signature. The thin_margins count tells you how exposed the prompt is: a response with 40 sub-1e-3 margins is a coin flip waiting for traffic.

Can I actually make it deterministic?

Yes, on hardware you control, with batch-invariant kernels. The rule is that the reduction strategy must be a function of the shape you promise, not of runtime occupancy:

• fixed split-K count (or none) regardless of M;
• fixed-size KV chunks in attention, so the number of partials depends only on sequence length;
• one block per row for RMSNorm instead of switching strategies at small batch;
• no atomic accumulation anywhere in the path;
• a fixed-order tree merge for partials.

This is what the batch-invariant kernel work that appeared in 2025 does, and inference stacks have started shipping toggles for it — vLLM has a batch-invariant mode in recent versions; check your version's flag rather than trusting a name from a blog post. Expect a throughput hit, concentrated at small batch and long context, because you're deliberately declining the occupancy-driven tuning that made those shapes fast.

Determinism also requires freezing everything else that selects a kernel:

# reproducibility profile — every one of these changes the numbers
vllm serve $MODEL \
  --tensor-parallel-size 4 \        # TP degree changes all-reduce order
  --dtype bfloat16 \                # not "auto"
  --max-num-seqs 64 \
  --no-enable-prefix-caching \      # cache hit != cache miss, numerically
  --no-enable-chunked-prefill       # chunk boundary changes the kernel path
# plus: pin the vLLM version, the attention backend, and the GPU generation

Turning off prefix caching and chunked prefill is a large performance sacrifice. Do it for an eval or repro rig, not for your serving fleet.

What about hosted models like Claude Opus 4.x or GPT-5.x?

You can't fix it, and you shouldn't build as if you could. temperature=0 on any hosted endpoint is best-effort, not a contract. Providers batch across tenants, route across hardware generations, and update kernels without telling you. OpenAI's seed is explicitly best-effort and pairs with a fingerprint field precisely so you can detect that the backend moved; Anthropic's API doesn't offer a seed at all. Treat model output as a distribution you sample, not a function you call.

Design accordingly:

• Cache at the application layer, keyed by a hash of the exact rendered prompt. That is the only real determinism available to you.
• Evaluate with n≥5 and report an interval. A single-run "regression" between two prompt versions is usually noise. Use paired sampling on the same items.
• Never gate CI on exact string equality. Assert on parsed fields, schema validity, or a judge score.
• Make agent tools idempotent and keyed by a request id, because a retried step can silently take a different branch.
• Log tokens and top logprobs, not just the final text. Without margins you can't tell a real behavior change from a coin flip.

Why this breaks RL fine-tuning specifically

This is the case where "it's only the last bits" stops being true. Your rollout worker (paged attention, small decode batches) and your trainer (large fused prefill-style batches) compute different logprobs for the exact same token sequence. So the importance ratio π_train/π_sample is not 1 at step 0, even though the data is genuinely on-policy.

Symptoms: nonzero KL against the reference at the very first step, PPO ratio clipping firing before the policy has moved, and a gradient bias that no hyperparameter sweep will explain. Either record the behavior logprob from the sampler and use it as the denominator (with truncated importance sampling), or make the two paths bitwise identical with batch-invariant kernels so the ratio is exactly 1. Averaging the discrepancy away is not an option; it's systematic, not zero-mean.

The short answer

temperature=0 isn't deterministic because greedy decoding only removes the sampler's randomness, not the numerical variability of the forward pass. GPU kernels choose their reduction and split strategy from the runtime batch shape, floating-point addition isn't associative, so logits shift by roughly 1e-6 to 1e-3 depending on how many co-tenant requests shared your batch — and any position where the top-2 logit gap is thinner than that shift can flip, after which the two generations diverge completely. On your own hardware you can get bitwise reproducibility with batch-invariant kernels plus a pinned TP degree, dtype, backend, and version, paying throughput for it. On a hosted API you cannot, so cache prompts at the application layer, evaluate over multiple samples with intervals, and stop asserting exact strings.