Same Prompt, Different Answer: Five Reproducibility Myths in Free-Model Pipelines

Two runs. Same prompt. Different output. Is that a bug, or is that just how the endpoint works?

I keep seeing the same claims in review threads. The model is deterministic. A retry is harmless. A 200 means it ran once. Let's test those claims instead of repeating them.

Below is a small harness you can run in ten minutes. It uses stdlib Python only. It gives you evidence, not vibes.

The setup: a stub you control

First, a local server that is deliberately nondeterministic. Real endpoints vary too. This one just makes the variation obvious.

# stub_model_server.py
import json, random
from http.server import BaseHTTPRequestHandler, HTTPServer

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers.get("Content-Length", 0))
        body = json.loads(self.rfile.read(n) or b"{}")
        prompt = body.get("prompt", "")
        answer = f"answer-{random.randint(1000, 9999)} for: {prompt[:20]}"
        payload = json.dumps({"output": answer}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

HTTPServer(("127.0.0.1", 8765), H).serve_forever()

Run it in one terminal: python3 stub_model_server.py

Now the runner. It writes a two-phase receipt: one row before the call, one row after the side effect.

# job_runner.py
import hashlib, json, os, time, urllib.request, uuid

BASE_URL = os.environ.get("MODEL_BASE_URL", "http://127.0.0.1:8765/generate")
LEDGER = os.environ.get("LEDGER_PATH", "receipts.jsonl")

def key_for(prompt, params):
    blob = json.dumps({"prompt": prompt, "params": params}, sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()[:32]

def call_model(prompt, params):
    req = urllib.request.Request(
        BASE_URL,
        data=json.dumps({"prompt": prompt, **params}).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.loads(r.read())["output"]

def record(row):
    with open(LEDGER, "a") as f:
        f.write(json.dumps(row) + "\n")

def applied(key):
    if not os.path.exists(LEDGER):
        return False
    with open(LEDGER) as f:
        return any(
            json.loads(l)["key"] == key and json.loads(l)["status"] == "applied"
            for l in f if l.strip()
        )

def run(prompt, params):
    key = key_for(prompt, params)
    if applied(key):
        print("replay detected, skipping side effect:", key)
        return key
    record({"key": key, "status": "started", "ts": time.time()})
    out = call_model(prompt, params)
    record({
        "key": key,
        "status": "applied",
        "attempt": str(uuid.uuid4()),
        "output_sha": hashlib.sha256(out.encode()).hexdigest()[:16],
        "ts": time.time(),
    })
    print("output_sha:", hashlib.sha256(out.encode()).hexdigest()[:16])
    return key

if __name__ == "__main__":
    run("summarize ticket 42", {"temperature": 0})

Run it twice: python3 job_runner.py. Compare the two output_sha values. That is myth one, dead.

Myth 1: "Temperature 0 makes it repeatable"

Temperature 0 narrows sampling. It does not promise an identical string.

My two runs above differ. Same prompt, same params, same server. Different hash.

So what do you actually need? A fingerprint of the output, stored next to the request. Not a promise of sameness. Then you can detect drift instead of arguing about it.

What to record instead

• Request key: hash of prompt plus params, sorted.
• Output hash: short SHA of the returned text.
• Attempt id: a UUID per call, so duplicates are visible.
• Status: started and applied, never one row.
• Timestamp, in UTC, from the runner.

Five fields. That's the whole trick.

Myth 2: "A retry only costs time"

Does it, though? A retry can be a second side effect.

Picture this. Your worker calls the model, gets a timeout, retries. Both calls actually succeeded. Now you have two summaries, two emails, two tickets.

The guard is an idempotency key. Mine is key_for(prompt, params). Same input, same key. The ledger refuses the second apply.

But look at my code. The side effect runs after the model call. If the process dies in between, you get an orphan: started, never applied. That's myth two's real cost — not the call, the ambiguity.

# audit.py
import json
rows = [json.loads(l) for l in open("receipts.jsonl") if l.strip()]
started = {r["key"] for r in rows if r["status"] == "started"}
applied = {r["key"] for r in rows if r["status"] == "applied"}
print("orphaned keys:", sorted(started - applied))

Run python3 audit.py and kill the runner mid-flight. You'll see the gap. An orphan is not a failure. It's a question your logs now ask out loud.

Myth 3: "HTTP 200 means it ran exactly once"

A 200 tells you one response arrived. It says nothing about how many attempts happened upstream.

Timeouts, proxy retries, and client reconnects all break the one-to-one story. So treat every response as at least once. Design your write path for duplicates.

Ask yourself: if this job ran twice, what breaks? If the answer is "nothing," good. If it's "the invoice," you have work to do.

Myth 4: "The model name pins the behavior"

Model names are labels. Endpoints get updated behind labels.

I'm not claiming a specific provider changed anything on a specific date — check their changelog today, not my memory. The general lesson holds: a name is not a hash.

So log a response fingerprint on a fixed canary prompt. Run it daily. When the hash shifts, you learn about it from your ledger, not from a user.

This is where free access gets useful. MonkeyCode's free model access gives you a low-pressure place to run that canary (operator-supplied availability; confirm current terms yourself). Cheap enough to test daily, which is the point.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Myth 5: "Local logs are a ledger"

The ledger must outlive the process that wrote it. Files on ephemeral disk are not a ledger. They are a rumor.

MonkeyCode also offers a free server option. If you run the runner there, keep LEDGER_PATH on durable external storage — object storage, a managed database, or a volume the provider documents as persistent. Otherwise audit.py finds nothing after a restart.

I learned this the boring way, so you don't have to.

Who should not use this

Be honest about the fit.

1. Throwaway scripts. If a duplicate costs nothing, skip the ledger. It's overhead.
2. Strict exactly-once needs. A file ledger gives at-least-once plus detection. It is not a transaction. Use a database with unique constraints.
3. Multi-writer queues. Two writers appending to one file will interleave rows. Move to SQL first.
4. Teams without cleanup. Orphans pile up. If nobody runs the audit, you built a diary, not a control.

Run it in ten minutes

Start the stub. Run the job twice. Diff the hashes. Kill it once. Run the audit.

That's four commands and one honest conclusion: reproducibility is something you measure, not something you assume.

Which myth breaks first in your logs? I'd bet on the retry.