Temperature 0 Is Not Determinism: Four Sampling Myths and a Repeat-Run Probe

My CI turned red on a test that passed yesterday. Same prompt. Same temperature: 0. Different output.

So I asked the boring question. Is temperature 0 a promise, or just a request? It is a request. The endpoint decides how literally to honor it.

This post is a myth-busting FAQ. Four claims I kept hearing, the evidence that breaks them, and a small probe you can run in ten minutes.

Myth 1: temperature=0 gives the same output every time

This is the claim that cost me a morning. It sounds mathematical. It is not.

What temperature actually does

Temperature scales logits before sampling. At 0, most stacks take the argmax instead of sampling. That part is real.

But argmax over which floats? That is the part nobody pins down.

Why identical inputs still drift

• Floating-point addition is not associative. Batch order changes the sum.
• Serving stacks batch your request with strangers. Batch composition changes.
• Kernel and library versions change between deploys.
• A provider can swap the model behind the same name.

Ask yourself: did you pin the weights, or did you pin a string? You pinned a string.

Corrected mental model

temperature=0 means "as greedy as this endpoint allows". It is not a seed. It is not a cache key. Treat it as a hint toward stability, not a guarantee of identity.

Myth 2: max_tokens caps the length of my answer

It caps tokens, not characters. Those are different units, and the mapping is model-specific.

Two things bite people here.

1. finish_reason tells you why generation stopped. If it is "length", you were truncated. Check it.
2. usage.completion_tokens may not match the visible text. Depending on the provider, it can include tokens you never see, or omit tokens it still counted.

If your parser assumes "the answer ended because the model finished", you will eventually parse a half sentence. Ask for the finish reason. Store it next to the text.

Myth 3: A stop sequence cuts cleanly

Stop handling is not standardized. Two endpoints, two behaviors.

• Some return the text up to the stop string. Some return it including the stop string.
• With streaming, the stop string can straddle two chunks. Naive clients leak it.
• Some endpoints drop trailing whitespace when they trim. Your diff notices.

So do not assume. Probe it with one request and read the raw repr.

Myth 4: The same model name behaves the same everywhere

A model name is a label, not a contract. The endpoint is the contract.

Free access and paid access can differ in version, quantization, default parameters, or limits. Even two endpoints with the same label on the same day can differ. That is not a scandal. It is just two deployments.

Why does this matter in practice? Because your retry logic, your eval baseline, and your prompt tuning are all calibrated against one endpoint. Move the base URL and you moved the ground.

The probe: repeat the same request eight times

Here is the artifact. It is a probe, not a benchmark. It measures one endpoint, one prompt, one parameter set.

# probe_sampling.py -- repeat-run probe for an OpenAI-compatible chat endpoint
import hashlib, json, os, time, urllib.request

BASE_URL = os.environ['PROBE_BASE_URL'].rstrip('/')
API_KEY  = os.environ.get('PROBE_API_KEY', '')
MODEL    = os.environ['PROBE_MODEL']
RUNS     = int(os.environ.get('PROBE_RUNS', '8'))
PROMPT   = 'Name one prime number between 10 and 20. Reply with the number only.'

def chat(payload):
    headers = {'Content-Type': 'application/json'}
    if API_KEY:
        headers['Authorization'] = 'Bearer ' + API_KEY
    req = urllib.request.Request(BASE_URL + '/chat/completions',
                                 data=json.dumps(payload).encode(),
                                 headers=headers, method='POST')
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.loads(r.read())

def text_of(resp):
    return resp['choices'][0]['message'].get('content') or ''

seen, rows = {}, []
for i in range(RUNS):
    payload = {'model': MODEL, 'temperature': 0,
               'messages': [{'role': 'user', 'content': PROMPT}]}
    if os.environ.get('PROBE_SEED'):
        payload['seed'] = int(os.environ['PROBE_SEED'])

    t0 = time.time()
    resp = chat(payload)
    out = text_of(resp)
    fp = hashlib.sha256(out.encode()).hexdigest()[:12]
    seen[fp] = seen.get(fp, 0) + 1
    rows.append({'run': i, 'fp': fp, 'chars': len(out),
                 'finish': resp['choices'][0].get('finish_reason'),
                 'usage': resp.get('usage', {}),
                 'ms': round((time.time() - t0) * 1000)})
    with open('probe_runs.jsonl', 'a') as f:
        f.write(json.dumps(rows[-1]) + '\n')

print(json.dumps({'model': MODEL, 'runs': RUNS, 'distinct': len(seen),
                  'top_share': round(max(seen.values()) / RUNS, 3)}, indent=2))

Run it like this:

export PROBE_BASE_URL='https://your-endpoint.example/v1'
export PROBE_API_KEY='only-if-needed'
export PROBE_MODEL='your-model-id'
export PROBE_RUNS=8
python3 probe_sampling.py

The output shape looks like this. I am showing the shape, not measured values from someone else's endpoint:

{"model": "<model-id>", "runs": 8, "distinct": <n>, "top_share": <0..1>}

distinct above 1 at temperature 0 means the endpoint is not locked. That is your finding. Now you can design around it.

Probe the cap and the stop string too

Two more checks, thirty seconds each:

def probe_cap(n):
    r = chat({'model': MODEL, 'temperature': 0, 'max_tokens': n,
              'messages': [{'role': 'user',
                            'content': 'Count from 1 to 40 in words, comma separated.'}]})
    c = r['choices'][0]
    return {'max_tokens': n, 'finish': c.get('finish_reason'),
            'chars': len(text_of(r)),
            'completion_tokens': r.get('usage', {}).get('completion_tokens')}

for n in (16, 64, 256):
    print(probe_cap(n))

r = chat({'model': MODEL, 'temperature': 0, 'stop': ['END'],
          'messages': [{'role': 'user', 'content': 'Write: alpha END beta'}]})
out = text_of(r)
print(repr(out), 'ends_with_stop=', out.rstrip().endswith('END'))

That repr is the whole point. It removes ambiguity about trimming and inclusion.

The decision table

Read the probe output, then pick a row.

Observation	What it means	What to change
distinct: 1	Stable for this prompt, today	Exact-match tests may hold; re-run after any deploy
distinct > 1 at temperature 0	Sampling is not locked	Assert on properties, not strings
finish: "length"	You hit the cap	Raise the cap or shorten the instruction
finish is missing or null	Endpoint differs from the spec you assumed	Do not branch retry logic on it
Output contains the stop string	Inclusive stop	Strip it yourself before parsing
usage is empty	No accounting available	Count tokens on the client

One row is not a verdict. Re-run after every endpoint change, and after every model update you did not request.

Where to run the loop

The probe is a small loop with sleeps and retries. It does not need your laptop.

I ran my copy on a free server so the terminal stayed free for real work. MonkeyCode offers free model access and a free server option, so I pointed PROBE_BASE_URL at the free model endpoint and ran the loop there. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Three limitations matter here.

• A free server can be interrupted. That is why the script appends every row to probe_runs.jsonl immediately.
• Free model access comes with limits you should read before looping. Keep PROBE_RUNS small.
• The probe result describes one endpoint on one day. It is evidence, not a benchmark, and it says nothing about model quality.

Do not turn this into a load test. Eight runs answer the question. Eight thousand runs get you rate limited and teach you nothing new.

Who should not use this

• If your tests can assert on structure instead of exact strings, do that first. Skip the probe.
• If you need to compare providers rigorously, you need many prompts and multiple seeds. This is not that tool.
• If your endpoint is stateful or multimodal, the probe shape needs rework before it fits.
• If you cannot spend ten minutes per endpoint change, pick one property test and stop there.

The corrected model, in one line

Output is a function of endpoint, version, parameters, and load. Not of the model name.

So what does your retry logic actually trust? If the answer is "the string I saw yesterday", you have a probe to write.