Temperature 0 Is Not a Reproducibility Setting: Five Model Myths and a Drift Probe

You shipped a prompt. It returned a great answer.

Two days later, same input, same code, worse answer. Nothing in your repo moved.

So what changed? That question is the whole article.

Below are five claims I keep hearing in review threads. Each one sounds reasonable. Each one quietly breaks your ability to tell "the model drifted" apart from "my code regressed."

Myth 1: "Temperature 0 means deterministic output"

What developers repeat: set temperature: 0 and the model becomes a pure function.

It does not. Temperature controls how flat or sharp the sampling distribution is. It says nothing about the arithmetic that produced that distribution.

Several things can still vary between two identical requests:

• Batching. Your request may share a GPU pass with strangers.
• Reduction order in floating-point sums. Addition is not associative.
• Kernel selection under load. Different kernels, different rounding.
• Routing inside mixture-of-experts models.
• Quantized paths chosen per batch shape.

Corrected mental model: temperature 0 reduces sampling variance. It does not remove infrastructure variance.

That distinction matters. If you assume determinism, every failed assertion looks like a code bug. Most of the time it is not.

Myth 2: "The model name is a version pin"

What developers repeat: model: "something-4" is stable, so my baseline holds.

Model names are usually aliases, not immutable artifacts. Providers re-point them. Rollouts are not always announced in your dashboard.

Ask yourself: if the weights changed overnight, would any test in your pipeline fail? For most teams, the honest answer is no.

Corrected mental model: a name is a routing label. If the provider exposes a fingerprint, log it. If not, you need a canary — a few stable prompts whose output you hash and store.

That is what the probe below does.

Myth 3: "One good run is a regression test"

One good run proves the endpoint was up. That is it.

It does not prove the output is stable, and it definitely does not prove quality. A single sample cannot separate signal from variance.

Corrected mental model: model behavior needs a distribution, not a datapoint. Run the same prompt N times and count distinct normalized outputs.

Myth 4: "Retrying is idempotency"

A retry is a second generation, not a replay. If the first call succeeded on the server side and the response was lost, you paid twice and now you have two different answers.

Write-side effects are the dangerous case. If the call inserts a row, sends a message, or charges a card, a blind retry duplicates it.

Corrected mental model: put your own idempotency key on the operation. Treat the model call as a nondeterministic side effect, not a pure function.

Myth 5: "Streaming and non-streaming return the same text"

They can diverge. Streaming changes the serving path, and the serving path can influence batching and kernel choice.

Is it always different? No. Is it something you should assume? Also no.

Corrected mental model: pick one mode for your test harness, record which one, and never compare hashes across modes.

The artifact: a drift probe you can diff

Here is a small probe. It is stdlib-only, and the HTTP block assumes an OpenAI-compatible /chat/completions shape — adapt it if your endpoint differs.

# drift_probe.py — run with: python drift_probe.py
import hashlib, json, os, statistics, urllib.request

ENDPOINT = os.environ["LLM_ENDPOINT"]      # e.g. https://host/v1/chat/completions
API_KEY  = os.environ["LLM_API_KEY"]
MODEL    = os.environ["LLM_MODEL"]
RUNS     = int(os.environ.get("RUNS", "5"))

# Canaries: short, boring, stable. Not your product prompts.
CANARIES = [
    "Reply with exactly: ok",
    "List the integers 1 to 5, comma separated, no spaces.",
    "What is 17 + 26? Answer with the number only.",
]

def call(prompt: str) -> str:
    body = json.dumps({
        "model": MODEL,
        "temperature": 0,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        ENDPOINT, data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {API_KEY}"},
    )
    with urllib.request.urlopen(req, timeout=60) as r:
        data = json.load(r)
    return data["choices"][0]["message"]["content"]

def norm(text: str) -> str:
    # Collapse whitespace so we compare content, not formatting.
    return " ".join(text.split())

def digest(text: str) -> str:
    return hashlib.sha256(norm(text).encode()).hexdigest()[:12]

ledger = {}
for prompt in CANARIES:
    hashes = []
    for _ in range(RUNS):
        try:
            hashes.append(digest(call(prompt)))
        except Exception as e:
            hashes.append(f"ERROR:{type(e).__name__}")
    ledger[prompt] = hashes

print(json.dumps(ledger, indent=2))
for prompt, hashes in ledger.items():
    distinct = len(set(hashes))
    print(f"{distinct}/{len(hashes)} distinct | {prompt[:40]}")

overall = statistics.mean(
    len(set(h)) / len(h) for h in ledger.values()
)
print(f"mean distinct ratio: {overall:.2f}")

Save that JSON output somewhere durable. That file is your baseline. Diff it next week.

The probe does not care where it runs, but it does need to keep running after your laptop closes. A small always-on box is enough — I run mine as a cron job rather than in a notebook. If you would rather not spend credits while you are still setting baselines, MonkeyCode offers free model access and a free server option, which is a reasonable place to run the first pass. I have not tested it under sustained load, so treat that as an option to evaluate, not a benchmark.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Reading the numbers

Distinct ratio	What it probably means	Next step
0.0	Your canaries are too easy, or you got lucky	Add a canary with free-form output
0.2 – 0.4	Normal sampling-adjacent variance	Widen assertions; stop using exact match
0.6 – 1.0	High variance or genuine serving noise	Cache responses; test structure, not text
Sudden jump vs baseline	Possible model or config change	Re-run, then check fingerprints and changelog
ERROR:* mixed in	Transport, auth, or rate limit	Fix the harness before trusting ratios

One warning. A ratio of 1.0 is not proof that the provider is unstable. It may just mean your prompt has many valid answers. Boring canaries beat clever ones here.

Limitations, honestly

• Five runs is a smoke test, not statistics. Bump RUNS before drawing conclusions.
• Normalizing whitespace hides formatting drift. That is a feature for content checks and a bug for UI tests.
• Hashing cannot detect a consistently wrong answer. A model that always says 43 to 17 + 26 hashes perfectly.
• Fingerprints, when exposed, are metadata you have to trust. Log them; do not depend on them.
• This measures stability. It says nothing about quality.

Who should not use this

Skip the probe if you need hard determinism. In that case, run a local model with a fixed seed and fixed kernel, or record responses once and replay them as cassettes in CI.

Also skip it if your only model path is a human reviewing chat output. That is a different problem. This probe is for people who already have a test suite that occasionally lies to them.

What I would do on Monday

1. Pick three canaries. Keep them dull.
2. Run the probe ten times. Save the JSON.
3. Add it to CI as a report, not a gate.
4. Diff ratios weekly. Alert on steps, not on absolute values.
5. Log the model fingerprint wherever you log the request.

The myth is that a free-tier model is a stable dependency. It is a moving one. Measure the movement before you blame your code — that is the only way the diff stays readable.

What would your baseline look like if you captured it today?