vLLM Temperature 0 Isn't Deterministic: 23 of 200 Answers Changed

My regression test for a prompt change failed on a Tuesday. I hadn't touched the prompt yet.

Same model, same weights, same vLLM server, same 200 eval prompts, temperature=0. I ran the suite twice in a row and 23 of the 200 answers came back different. Not paraphrased-different. Some of them reached a different conclusion. I spent an hour convinced someone had redeployed the server behind my back. Nobody had.

Here's the uncomfortable fact I'd been ignoring: temperature 0 isn't deterministic on a real inference server. Greedy decoding is deterministic. The math that produces the logits it picks from is not, as soon as other people's requests share the GPU with yours.

TL;DR

• temperature=0 means greedy decoding: always pick the highest-probability token. That step is deterministic. The logits feeding it are not.
• vLLM uses continuous batching, so your request's batch size depends on whatever else is in flight. Different batch sizes make GPU kernels sum numbers in a different order.
• Floating-point addition isn't associative, so a different order gives slightly different logits. When the top two tokens are nearly tied, the winner flips.
• One flipped token changes the context for every token after it, so the rest of the answer diverges.
• Fix for evals: pin batch composition (serialize requests or use a batch-invariant mode), compare outputs semantically, and log the top-1 vs top-2 logprob gap so you can see where flips will happen.

Why does temperature 0 give different outputs on vLLM?

Because your request is never computed alone. vLLM's continuous batching packs every in-flight sequence into each forward pass, and the kernels pick different reduction strategies for different batch shapes. Different reduction order means different rounding, which means slightly different logits for the exact same input.

My setup was boring on purpose: Llama 3.1 8B Instruct in bf16 on one GPU, vLLM's OpenAI-compatible server, default settings. The test that caught it looked like this:

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="x")

async def ask(prompt):
    r = await client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=400,
    )
    return r.choices[0].message.content

async def run_suite(prompts, concurrency):
    sem = asyncio.Semaphore(concurrency)
    async def one(p):
        async with sem:
            return await ask(p)
    return await asyncio.gather(*(one(p) for p in prompts))

The first time it "passed," I'd run it with concurrency=1 out of laziness. Two runs, 0 of 200 different. The day it failed, I'd bumped it to concurrency=32 to make CI faster. Two runs, 23 of 200 different.

Same prompts. Same server. The only thing that changed was who my requests were sitting next to on the GPU.

Isn't the GPU just random because of parallel threads?

Mostly no, and this surprised me. A given kernel with a given input shape usually produces the same bits every time. The problem is that the shape isn't fixed. The batch your request lands in is decided by traffic, not by you.

Thinking Machines Lab published a well-known write-up on this in 2025 that put a name on it: the culprit is a lack of batch invariance. Kernels are tuned for speed, and the fastest way to reduce a big sum depends on how much work there is:

• Matmuls may split the reduction dimension across thread blocks (split-K) for small batches and not for large ones.
• Attention during decode may split the KV sequence into chunks (flash-decoding style) depending on how many queries are in the batch.
• Normalization layers like RMSNorm reduce across the hidden dimension, and the reduction strategy can change with batch size.

Each strategy is correct. They just add the same numbers in a different order.

Why does the order of addition matter?

Because floating-point addition is not associative. You can see it in plain Python, no GPU required:

>>> (0.1 + 0.2) + 0.3
0.6000000000000001
>>> 0.1 + (0.2 + 0.3)
0.6

That's float64, with 52 bits of mantissa. Model inference runs in bf16, which has 7. Rounding error per operation is enormous by comparison, and a single logit is the result of thousands of multiply-adds stacked through 32 layers.

So the same prompt in a batch of 1 and a batch of 29 produces logits that agree to two or three significant digits and then disagree. Usually that doesn't matter at all.

When does a tiny logit difference change the token?

Only when the top two candidates are nearly tied. Greedy decoding is an argmax. If token A has a logit of 18.40 and token B has 18.39, a rounding wobble of 0.02 is enough to swap the winner.

You can see these near-ties directly. Ask for logprobs and record the gap between the top two candidates at every step:

r = await client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": prompt}],
    temperature=0,
    max_tokens=400,
    logprobs=True,
    top_logprobs=2,
)

for i, tok in enumerate(r.choices[0].logprobs.content):
    alts = tok.top_logprobs
    if len(alts) == 2:
        gap = alts[0].logprob - alts[1].logprob
        if gap < 0.05:
            print(f"step {i}: {alts[0].token!r} vs {alts[1].token!r}  gap={gap:.4f}")

When I went back through my 23 flipped answers and found the first token where the two runs disagreed, every one I inspected had a tiny top-1/top-2 gap at that position. The flips weren't happening on confident tokens. They were happening on coin tosses the model was already making.

The usual suspects: "However" vs "But", whether to open a list with "1." or a bullet, "is" vs "can be". Stylistically trivial. That's the trap.

Why does one flipped token change the whole answer?

Because generation is autoregressive. Every token becomes input for the next one. Flip "However" to "But" at step 40 and every step after 40 is computed from a different context. Those steps have their own near-ties, which now resolve differently too.

That's how a rounding error in the fourth significant digit turned into a different conclusion on 3 of my 23 diverged answers. The model didn't change its mind. It started a sentence slightly differently and then followed that sentence somewhere else.

Doesn't the seed parameter fix this?

No. seed controls the random number generator used for sampling. At temperature=0 there is no sampling, just argmax, so the seed has nothing to do. The nondeterminism lives upstream in the forward pass, which the seed never touches.

Hosted APIs have the same issue for the same reason. Providers batch your requests with everyone else's, and most of them say plainly in their docs that temperature=0 gives "mostly" deterministic output, not guaranteed. If your eval assumes byte-identical answers from an API, it's assuming something nobody promised.

Are there other sources besides batch size?

Yes, and they all share the theme of "the same input took a different compute path":

• Prefix caching. If part of your prompt's KV cache was reused from an earlier request, those values were computed in a different batch than a fresh prefill would use.
• Chunked prefill. Long prompts get split into chunks scheduled alongside other work, which changes the shapes again.
• Tensor parallel setups. All-reduce across GPUs adds its own reduction order.
• Different hardware or library versions. A different GPU model or CUDA version may select different kernels entirely.

Batch size is the one that bites in steady state, because it changes every second with traffic.

How do I make LLM evals reproducible?

Pick the level of determinism you actually need and pay for exactly that.

1. Stop asserting exact string equality. For most evals, the question is "is this answer correct," not "is this answer identical." Grade with a checker, a parser, or a rubric. This alone made my suite stable.

2. Serialize when you need bit-identical runs. Running with one request in flight made my two runs match 200 of 200. You can enforce it server-side:

vllm serve meta-llama/Llama-3.1-8B-Instruct --max-num-seqs 1

This kills throughput, so use a dedicated eval server, not production.

3. Look for a batch-invariant mode. Recent versions of vLLM and SGLang have added experimental deterministic or batch-invariant options that use kernels with a fixed reduction order regardless of batch size. They're slower. Check the docs for the exact flag in your version before trusting any blog post, this one included.

4. Log the logprob gap. If a test case has a top-1/top-2 gap under ~0.05 somewhere early, it's going to flake. Flag it, or rewrite the prompt so the model isn't balanced on a knife edge.

5. Run each eval case more than once. If a case passes 2 of 3 times, that's real information about your prompt's fragility. Hiding it behind one lucky greedy run is worse.

The checklist I use now

• Eval server: separate from prod, --max-num-seqs 1 when I'm comparing prompt versions.
• Grading: semantic or programmatic, never == on full text.
• Every run stores logprobs with top_logprobs=2 for the first 100 tokens.
• Cases with a near-tie in the first 50 tokens get tagged fragile and reported separately.
• Prompt changes are judged on pass rate across 3 runs, not one.

My prompt change, by the way, was fine. It just took me a day to be able to tell.

So why isn't temperature 0 deterministic?

Temperature 0 isn't deterministic because greedy decoding only removes randomness from the sampling step, not from the forward pass that computes the logits. On vLLM and most hosted APIs, continuous batching puts your request in a batch whose size depends on other traffic, and GPU kernels change their reduction order with batch size. Since floating-point addition isn't associative, the logits shift slightly; when the top two tokens are nearly tied, the argmax flips, and autoregressive generation carries that one flip into a completely different answer. To get reproducible results, fix the batch composition (serialize requests or use a batch-invariant kernel mode), grade outputs semantically instead of by exact match, and use logprob gaps to find the prompts that are one rounding error away from changing.

---

Written by the developer behind Preterview, an interview prep platform.