Jordan Huang

Posted on Oct 19

Temperature 0 Is a Suggestion: Five LLM Determinism Myths and a Rerun Harness

#ai #testing #python #llm

Your test passes locally. It fails in CI maybe once a week. The prompt did not change. So what exactly changed?

Most teams answer 'the model is flaky.' That answer is lazy, and it hides a fixable bug: your assertion. Let's walk through five claims developers repeat about determinism, then build a small harness that measures what your setup actually does.

Why the myth survives

Model output looks stable in demos. You send a prompt, you get the same JSON shape twice. Two samples is not evidence.

Determinism is a property you measure over many runs. It is not a vibe you pick up from a playground tab.

Myth 1: Temperature 0 means deterministic

Not quite. OpenAI's reproducibility guidance describes low temperature as mostly deterministic, not guaranteed. The docs also warn that identical requests can return different outputs.

Why? Batching, GPU kernels, and routing between backend configurations. Your request lands wherever capacity exists.

The corrected mental model: temperature 0 narrows the distribution. It does not pin one sample.

Myth 2: A seed makes your experiment reproducible

A seed helps. It is best-effort, not a guarantee. The same guidance says to log system_fingerprint alongside your seed.

If the fingerprint changes, the backend changed. Your 'same' run is now a different experiment.

Corrected model: seed plus fingerprint plus model version is your reproducibility key. Any missing piece makes replay a guess.

Myth 3: Same prompt means same tokens

Your prompt string is not the prompt the model sees. Chat templates inject system text. Some SDKs add defaults.

Tokenizers also differ per model. A string that costs 40 tokens on one model may cost 43 on another.

Corrected model: compare rendered prompts, not source strings. Log the rendered payload hash.

Myth 4: A different string means different logic

Often the logic is identical. Only formatting drifted.

Key order in JSON objects.
1 versus 1.0 for the same number.
Extra whitespace or a trailing newline.
Markdown fences around otherwise valid JSON.

Compare normalized output. Or better, compare parsed fields.

Corrected model: assert on schema and field values. Do not assert on the raw string.

Myth 5: Divergence means the model is broken

Sometimes it is. Often it is not. Ask one question first: did the divergence change a downstream decision?

If both outputs parse and pass your validator, your pipeline is fine. If one output fails validation, you found a real defect.

Corrected model: divergence is data. Only decision changes are incidents.

The harness: small, boring, repeatable

Here is a rerun harness. It sends one prompt N times and reports how many distinct normalized outputs you get.

I run this kind of probe against free model access, because repetition is the whole point. MonkeyCode offers free model access and a free server option; treat both as availability claims and verify them against your own workload. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

import hashlib, json, os, re
from collections import Counter
from openai import OpenAI

client = OpenAI(
    base_url=os.environ['LLM_BASE_URL'],
    api_key=os.environ['LLM_API_KEY'],
)
MODEL = os.environ['LLM_MODEL']
PROMPT = 'Return one JSON object with exactly two keys: name, age.'
N = int(os.environ.get('N', '20'))

def normalize(text: str) -> str:
    text = re.sub(r'^```

(?:json)?|

```$', '', text.strip(), flags=re.M)
    return re.sub(r'\s+', ' ', text).strip()

def digest(text: str) -> str:
    return hashlib.sha256(normalize(text).encode()).hexdigest()[:12]

def schema_ok(text: str) -> bool:
    try:
        obj = json.loads(normalize(text))
    except json.JSONDecodeError:
        return False
    return isinstance(obj, dict) and set(obj) == {'name', 'age'}

def run_once(seed=None):
    kwargs = {
        'model': MODEL,
        'temperature': 0,
        'messages': [{'role': 'user', 'content': PROMPT}],
    }
    if seed is not None:
        kwargs['seed'] = seed
    resp = client.chat.completions.create(**kwargs)
    text = resp.choices[0].message.content or ''
    return text, getattr(resp, 'system_fingerprint', None)

def main():
    hashes, valid, fps = Counter(), 0, Counter()
    for _ in range(N):
        text, fp = run_once(seed=7)
        hashes[digest(text)] += 1
        valid += schema_ok(text)
        fps[fp] += 1
    print(json.dumps({
        'runs': N,
        'distinct_normalized_outputs': len(hashes),
        'schema_valid_share': round(valid / N, 3),
        'system_fingerprints': dict(fps),
    }, indent=2))

if __name__ == '__main__':
    main()

Any OpenAI-compatible endpoint works. Set three environment variables and run it.

I am not printing sample numbers here. You have not run them, so they would be fiction.

How to read the output

Field by field:

distinct_normalized_outputs above 1 means string-level nondeterminism exists.
schema_valid_share below 1 means you have a production bug, not a flake.
More than one system_fingerprint means the backend changed mid-run.
A single fingerprint with many distinct outputs points at inference nondeterminism.
A single fingerprint with one output still proves nothing at N=20.

Decision table

Your goal	Assert on	Never assert on
CI stability	schema plus field constraints	exact raw string
Regression tests	recorded fixtures with recorded fingerprint	live model calls
Prompt tuning	pass rate over 20+ runs	one sample
Debugging a failure	rendered prompt hash plus fingerprint	memory of the prompt

Limitations and who should skip this

This harness measures your endpoint, not determinism in general. N=20 is a smoke test, not a rate estimate. Provider-side caching can hide variance and inflate your stability numbers.

Skip this approach if you need bit-exact replay for audits, billing, or regulated output. For those cases, log every response and compare against recorded fixtures, never against a live model.

Also skip it if your prompt has no validator. Collecting twenty unparsed strings teaches you nothing.

The one habit worth keeping

Log three things on every call: rendered prompt hash, model version, and fingerprint. When a test fails next month, you can answer 'what changed' in minutes.

Try the harness on whatever free access you already have. Then tell me your distinct-output count, not your gut feeling.

Further reading: OpenAI Cookbook, 'Reproducible outputs with the seed parameter' (https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter). Confirm current wording in your provider's own docs before quoting it.

DEV Community