FAQ: Four Myths About Reproducible Runs on a Free Model Endpoint

Your eval passed on Tuesday. It failed on Wednesday. Same prompt, same commit, same endpoint.

So what changed? Most of us answer with a version string. That answer is often wrong.

This is a myth-busting FAQ about what a free model endpoint really guarantees. Each myth below comes with a check you can run yourself. No vibes, no screenshots, just bytes you can compare.

Myth 1: "The model string is the version"

The claim: model: "some-model" pins behavior like requests==2.31.0 pins a dependency.

Why it breaks: aliases resolve. A gateway accepts a friendly name and serves whatever it currently routes to. Providers such as OpenAI document dated snapshot names and report the resolved snapshot in the response body, so the model field you read back is not guaranteed to equal the string you sent. Some providers also return a system_fingerprint. Many return nothing at all. A missing fingerprint means unknown, not stable.

The check: never hardcode an id you have not listed from the endpoint. Pull the model list into a lockfile, then compare the echoed field against your request.

curl -s "$MC_BASE_URL/models" -H "Authorization: Bearer $MC_API_KEY" \
  | jq -r '.data[].id' | sort > models.lock

That file is a snapshot, not a contract. Re-run it whenever a result surprises you.

Myth 2: "temperature=0 is a determinism switch"

The claim: set temperature to zero and identical inputs give identical outputs.

Why it breaks: temperature is a sampling parameter, not a reproducibility mode. At zero, most implementations take the highest-probability token, so you see less variance. You do not see a guarantee. Batching, floating-point kernels, mixture-of-experts routing, prompt caching, and provider-side deploys all live below your request. Sampling was never the only source of variation.

Corrected mental model: temperature=0 means "lower variance on this endpoint, today". If you need bit-identical output, sample nothing and control the weights, seeds, and kernels yourself.

# Lower variance, not a promise.
payload = {"model": MODEL, "messages": msgs, "temperature": 0}

Myth 3: "If I log the prompt, I have provenance"

The claim: the prompt is the record. Store it and you can explain any result later.

Why it breaks: the prompt tells you what you asked. It says nothing about where the request landed, which model answered, which parameters applied, or what bytes came back. That is not provenance. That is a grocery list.

Corrected mental model — record five things per call:

1. Requested model string and full parameter set.
2. Base URL host you actually called.
3. The model field echoed in the response.
4. system_fingerprint when present, null when not.
5. A hash of the normalized output content.

Five fields, one timestamp. That is enough to answer "what ran?" six weeks later.

Myth 4: "A retry that passes proves the first failure was flaky"

The claim: call it again, it works, so the earlier run was noise.

Why it breaks: a retry changes the sample. It does not change the environment. You learn that a second draw succeeded, not why the first draw failed. Worse, many free-tier failures are transport-level: connection resets, read timeouts, and 429 rate limits. A 429 is a limit, not a flake. Retry loops that hammer it make the problem worse, not better.

The check: classify before you retry.

• Transport error → back off with jitter, then retry once.
• Echoed model mismatch → stop. Your assumptions are wrong.
• Output differs with no transport error → that is sampling variance, not flakiness.

A probe you can run in ten minutes

You need an OpenAI-compatible base URL and a key. If you do not have one handy, the free server option and free model access from MonkeyCode are enough to run this end to end without provisioning anything first.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Save this as provenance_probe.py. It is standard library only.

#!/usr/bin/env python3
"""Record what a model endpoint actually served.
Labeled as a proposal: run it, then keep the JSON output with your results.
"""
import hashlib, json, os, time, urllib.request

BASE = os.environ["MC_BASE_URL"].rstrip("/")
KEY  = os.environ["MC_API_KEY"]
MODEL = os.environ.get("MC_MODEL", "")  # fill from models.lock, never guess

def call(payload):
    req = urllib.request.Request(
        BASE + "/chat/completions",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + KEY},
        method="POST")
    t0 = time.monotonic()
    with urllib.request.urlopen(req, timeout=60) as r:
        raw = r.read()
    return raw, round(time.monotonic() - t0, 2)

def probe(n=5, prompt="Reply with the single word: ready"):
    rows = []
    for i in range(n):
        raw, ms = call({"model": MODEL,
                        "messages": [{"role": "user", "content": prompt}],
                        "temperature": 0})
        body = json.loads(raw)
        content = body["choices"][0]["message"]["content"].strip()
        rows.append({
            "run": i,
            "requested": MODEL,
            "served": body.get("model"),
            "fingerprint": body.get("system_fingerprint"),
            "content": content,
            "normalized_sha256": hashlib.sha256(content.encode()).hexdigest()[:16],
            "ms": ms,
        })
    return rows

if __name__ == "__main__":
    rows = probe()
    print(json.dumps(rows, indent=2))
    print("served ids :", sorted({r["served"] for r in rows}))
    print("output hashes:", sorted({r["normalized_sha256"] for r in rows}))

Read the output like this

• One served id, one hash → the endpoint behaved consistently for this prompt.
• Several hashes, one served id → sampling or infrastructure variance.
• More than one served id → you are not running what you think you are running.

One trap worth naming: hash the content, not the raw response body. Raw bodies contain an id and created timestamp, so the raw hash changes on every single call and tells you nothing.

Decision table

Use case	Safe on an unpinned free endpoint?	What to record
One-off script or chat	Yes	Nothing
Prompt iteration	Yes	Prompt, served id, output
Comparing two prompts	Mostly	Five provenance fields per call
CI gate on model output	Risky	Everything, plus a retry policy
Published benchmark	No	Pin a snapshot, publish the lockfile
Regulated or sensitive data	Only after review	Data-flow review first

Limitations, and who should skip this

This probe detects change. It does not explain it. A hash mismatch is a signal to investigate, not a diagnosis.

It also says nothing about quality. Two runs can match byte for byte and both be wrong. Keep your real eval on top of it.

A lockfile is a snapshot, not a guarantee. Providers can move under you between runs.

Skip this approach if you need audited determinism. That means fixed weights, fixed seeds, fixed kernels, and a host you control. Skip it too if your CI job cannot tolerate transport retries or rate limits, or if you cannot send the data to an endpoint you have reviewed.

The question to ask your own pipeline

If someone asked you which model answered yesterday's failing call, could you answer with evidence?

Or would you answer with a version string and a shrug?