Field Notes: Temperature 0 Did Not Reproduce My Outputs, so I Built a Drift Ledger

The assumption I started with

My nightly job reads a batch of support tickets and emits a JSON plan describing which tickets to merge, which to escalate, and who owns each step. It had been running from my laptop for a few weeks, and I trusted it because I set temperature: 0 and the answers "looked the same" whenever I skimmed them. Isn't that how everyone validates a model-backed job — read two outputs, nod, ship it?

Three scheduled runs over one weekend were enough to show me that "looks the same" was a story I told myself. The plans were structurally identical and verbally different, and one of them referenced a ticket ID that did not exist in the input batch. Here are my field notes: what I tried, what broke, and what I would repeat if I had to do it again next weekend.

A note on honesty: the values in the code examples below are illustrative placeholders, not measurements from a benchmark. The artifact is real and runnable; the numbers in it are yours to fill in.

Hours 0–4: the exact-match test that failed on day one

My first instinct was the obvious one — hash the output and assert the hash never changes between runs. I wrote a nine-line test, ran it three times, and got three different hashes. Nothing was wrong with the model call; the problem was that I had confused determinism of a single request with reproducibility of a pipeline.

A hosted endpoint can change what serves your request without changing your code. Model aliases move, routing changes, and the decode path may not be bit-identical even at zero temperature. If your regression test compares strings, it is really testing the vendor's deployment schedule, not your prompt. So I deleted the hash assertion and kept the hash as data.

Hours 4–12: the drift ledger

Instead of asserting on hashes, I started writing them down. Every call now appends one line to runs.jsonl, which is the only piece of this workflow I would defend in a code review.

# drift_ledger.py
import hashlib, json, os, time, urllib.request

LEDGER = os.environ.get("LEDGER_PATH", "runs.jsonl")

def sha(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]

def call(prompt: str, extra_params: dict | None = None) -> dict:
    base = os.environ["MODEL_BASE_URL"].rstrip("/")
    key = os.environ["MODEL_API_KEY"]
    body = {
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
        **(extra_params or {}),
    }
    req = urllib.request.Request(
        f"{base}/chat/completions",
        data=json.dumps(body).encode("utf-8"),
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {key}",
        },
    )
    started = time.monotonic()
    with urllib.request.urlopen(req, timeout=90) as resp:
        payload = json.loads(resp.read())
    latency_ms = int((time.monotonic() - started) * 1000)

    text = payload["choices"][0]["message"]["content"]
    record = {
        "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "prompt_sha": sha(prompt),
        "params": body,
        "served_model": payload.get("model"),
        "output_sha": sha(text),
        "latency_ms": latency_ms,
    }
    with open(LEDGER, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(record) + "\n")
    return {"text": text, "record": record}

Two fields earned their keep immediately. served_model is whatever the response claims served the request — it can be absent, and it can differ from what I asked for. prompt_sha lets me group history later, so I can ask the only question that matters after a bad night: did the prompt change, or did the answer?

# did the same prompt get answered by the same thing all weekend?
jq -r 'select(.prompt_sha=="a1b2c3d4") | .served_model' runs.jsonl | sort | uniq -c

Hours 12–24: invariants instead of golden files

Once I accepted that the wording would drift, golden-file testing became pointless. What never drifts is the contract: the plan must be parseable JSON, every step must reference a ticket that actually exists in the input, and no step may contain placeholder text. Those are properties, not strings.

# invariants.py
import json, re

PLACEHOLDERS = re.compile(r"\bTBD\b|\bFIXME\b|\bXXX\b")

def validate_plan(text: str, allowed_ids: set[str]) -> list[str]:
    try:
        plan = json.loads(text)
    except json.JSONDecodeError as exc:
        return [f"not json: {exc}"]

    steps = plan.get("steps")
    if not isinstance(steps, list) or not steps:
        return ["steps missing or empty"]

    problems = []
    for i, step in enumerate(steps):
        action = step.get("action")
        if not isinstance(action, str) or not action.strip():
            problems.append(f"step[{i}]: empty action")
        elif PLACEHOLDERS.search(action):
            problems.append(f"step[{i}]: placeholder text in action")

        ref = step.get("ticket")
        if ref not in allowed_ids:
            problems.append(f"step[{i}]: unknown ticket {ref!r}")
    return problems

The ticket-reference check is the one that caught my real bug. A plan that invents OPS-99 is not a stylistic variation — it is a plan an on-call engineer cannot execute, and it would have shipped silently under any substring or similarity assertion I could think of.

Hours 24–36: moving the schedule off my laptop

A nightly job that only runs when I remember to open the lid is a hobby, not a job. I moved the schedule onto MonkeyCode's free server option, which meant the script had to stop assuming my shell, my working directory, and my leftover files. I rewrote it to treat everything as absent on start: config from environment variables, ledger path from LEDGER_PATH, prompt read from disk at call time.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free model access is what made the weekend experiments cheap enough to run in threes rather than ones — I could afford to burn two extra calls per prompt to learn whether a plan was stable. To be explicit about what I am not claiming: I have no data on quotas, uptime, hardware, or how long the free option stays free, and none of those belong in a reproducibility argument anyway. What I needed was somewhere the cron could live, and the design above assumes nothing about that host except Python and outbound HTTPS.

Hours 36–48: the three-run gate

One run tells you nothing, and two runs tell you almost nothing. Three runs is the smallest number where a disagreement has to be explained rather than dismissed as an outlier, so the gate below is what actually blocks a bad night. It does not compare outputs to each other; it checks each one against the invariants and reports how many distinct outputs appeared.

# gate.py
import json, sys
from drift_ledger import call
from invariants import validate_plan

PROMPT = open("prompt.txt", encoding="utf-8").read()
ALLOWED = {"OPS-11", "OPS-14", "OPS-27"}  # built from the real ticket batch

runs = [call(PROMPT) for _ in range(3)]
failures = {i: validate_plan(r["text"], ALLOWED) for i, r in enumerate(runs)}
bad = {i: p for i, p in failures.items() if p}

report = {
    "distinct_outputs": len({r["record"]["output_sha"] for r in runs}),
    "served_models": sorted({r["record"]["served_model"] for r in runs}),
    "failing_runs": bad,
}
print(json.dumps(report, indent=2))
sys.exit(0 if not bad else 1)

python gate.py; echo "exit=$?"

Wiring this into the cron wrapper is the whole point: if the gate exits non-zero, the nightly plan is not produced, and a human gets a notification instead of a plausible-looking artifact. Silence is a feature here.

What broke, and what I would repeat

1. Broke: exact-output hashing. Any test that fails when the vendor redeploys is a test nobody trusts by week two.
2. Broke: my mental model of temperature: 0. It reduces sampling variance; it does not pin a model, a version, or a tokenizer.
3. Broke: a served_model-based assertion. That field is informational and may be missing entirely, so I log it and never branch on it.
4. Would repeat: writing the ledger before writing any assertion, because it turned an argument about opinions into a query against a file.
5. Would repeat: invariants built from the input batch rather than from a previous output, since invented references are the expensive failure and formatting is the cheap one.
6. Would repeat: the three-run gate in front of the scheduled job, not inside it — the job should never be able to decide it is fine.

Where this approach is the wrong one

This is a workflow for teams that need to notice drift and fail safely, not for teams that need byte-identical output. If you are generating artifacts under a regulatory requirement that a specific output be reproducible on demand, a free endpoint with a moving alias is the wrong building block and you should be pinning a version you control. If your nightly plan takes forty minutes per call, the three-run gate triples a cost you may not be able to pay, and you should sample one run per night and gate weekly instead.

And if nobody reads the failure notification, none of this matters — the gate just moves the silent failure one step later. Who is on the other end of that alert in your setup?

The three files above are the entire harness, and they will run against MonkeyCode's free model access or anything else that speaks the same request shape — start by logging a week of runs before you assert on a single one.