Temperature Zero Is Not a Reproducibility Guarantee: Five Agent Myths, Tested

Your agent opens a pull request. The checks are green. You merge it.

Ten minutes later the same commit fails on a second run. Same prompt, same repo, same model name.

Was the fix real? Or did one lucky sample fool you? That question runs through everything below.

I keep a short list of claims developers repeat about agent runs. Most of them survive contact with a chat window. Few survive contact with twenty repeated runs.

Here are five myths, what the evidence actually looks like, and a corrected mental model you can test today.

Myth 1: "Temperature 0 makes the model a pure function"

Setting temperature=0 reduces sampling randomness. It does not remove run-to-run variance.

Batching, kernel selection, MoE routing, and silent model revisions all move the output. Some providers expose a system_fingerprint field for exactly this reason.

So the honest claim is: temperature 0 makes output more stable. It never promises the same bytes twice.

The fix is boring. Hash your outputs and count.

#!/usr/bin/env bash
# determinism-check.sh — TEMPLATE, not executed for this article
set -euo pipefail

PROMPT_FILE="${1:?usage: determinism-check.sh prompt.txt [n]}"
N="${2:-20}"
OUT="runs"
mkdir -p "$OUT"

for i in $(seq 1 "$N"); do
  # Swap in your own CLI call or curl request. Keep one variable per run.
  agent-run --prompt "$(cat "$PROMPT_FILE")" >"$OUT/run-$i.txt" 2>&1
done

sha256sum "$OUT"/run-*.txt | awk '{print $1}' | sort | uniq -c | sort -rn

Read the last line only. One bucket with the full count means stable. Three buckets means you have variance you were not accounting for.

This loop is where MonkeyCode's free model access fits the workflow. At twenty samples per prompt, a paid key turns a sanity check into a budget decision.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The corrected model

No model is deterministic across calls. Measure variance, then gate on it. If your pipeline needs bit-for-bit output, you need a pinned, self-hosted model, not a temperature flag.

Myth 2: "Green locally means green on the runner"

Local green proves your laptop works. It says nothing about the machine that will run your code.

Agents amplify this because they touch environment details nobody reviews. Timezones, locales, glibc versions, CPU flags, egress rules, and cache state all change behavior.

Stop guessing. Fingerprint both sides and diff.

#!/usr/bin/env bash
# env-fingerprint.sh — run locally AND on the runner, then diff
fingerprint() {
  uname -srmo
  printf 'TZ=%s\n' "$(cat /etc/timezone 2>/dev/null || echo unset)"
  printf 'LANG=%s\n' "${LANG:-unset}"
  node -v 2>/dev/null || true
  python3 -c 'import sys,platform;print(sys.version.split()[0], platform.libc_ver())' 2>/dev/null || true
  awk '/^flags/{print "cpu-flags:" $0; exit}' /proc/cpuinfo 2>/dev/null || true
  printf 'egress=%s\n' "$(curl -s -o /dev/null -w '%{http_code}' https://example.com || echo blocked)"
}
fingerprint | sort

./env-fingerprint.sh > local.txt
# Same script on the remote machine, then:
diff -u local.txt runner.txt

Every unexpected line in that diff is a candidate cause for your next flaky failure. This is also the second useful place for a free server option: you get a cheap second environment to diff against, without provisioning anything. Treat that availability as operator-supplied and re-check it before you depend on it.

Myth 3: "Retrying to green means it is fixed"

Retries hide flakiness. They do not repair it, and they quietly raise your cost per merged change.

Run the same job twenty times on purpose. Then read the flake rate as a signal, not a nuisance.

Flake rate over 20 runs	Likely cause	Action
0 failures	Nothing yet	Keep, re-check next week
1 failure	Clock, ordering, or network	Log a seed and the failing step
3–5 failures	Shared resource or race	Reproduce serially with a fixed seed
More than 5	Broken environment or bad test	Fix or delete the test, never retry-to-green

A test that fails once in twenty runs is not passing. It is sampling.

Myth 4: "A warm cache makes the build reproducible"

The cache key is not the build. The lockfile is.

Agents happily run an install that rewrites the lockfile, then commit the change with a vague message. Now two machines resolve different dependency trees from the same commit.

# Fail loudly if the install mutated the lockfile.
git diff --exit-code -- package-lock.json pnpm-lock.yaml poetry.lock \
  || { echo "lockfile drifted during install"; exit 1; }

The corrected model

Reproducibility comes from pinned inputs, not warm caches. A cache accelerates a correct build. It cannot make an unpinned one correct.

Myth 5: "Free compute means no cost"

Every free tier has a budget. It is usually wall-clock time and concurrency instead of dollars.

Track the number that matters:

• Agent minutes per merged PR, including every failed and retried run.
• Queued minutes, which consume human patience rather than machine time.
• Retry multiplier: total runs divided by successful runs.

A retry multiplier of 3 means your agent burns triple the time you think it does. That is a real engineering cost, even at zero dollars.

# From a CSV of run events: id,outcome,seconds
awk -F, '{t+=$3; n++; if($2!="success") f++} END {printf "runs=%d minutes=%.1f retry_mult=%.2f\n", n, t/60, n/(n-f)}' runs.csv

Replace the column numbers with your own. The formula is the point, not the exact field order.

What this approach does not prove

Be clear about the limits. A twenty-run sample bounds variance for one prompt on one day. It does not certify a model, a version, or a future run.

The scripts above are templates. I have not executed them against every provider for you, and the printed values are placeholders until you run them yourself.

Also, availability claims change. Free model access and the free server option are operator-supplied details, so confirm current terms in the product before building a dependency on either.

Who should not use this

• Teams needing bit-for-bit reproducible builds should pin and self-host, not sample.
• Regulated pipelines requiring versioned endpoints and written SLAs should skip free tiers entirely.
• Anyone who treats a chat log as a diff should fix that habit before adding more tooling.

The one-line correction

Green means one sample passed. It never means the system is deterministic.

So next time an agent job goes green on the first try, ask the uncomfortable question: would it be green on the tenth? If you want a cheap place to start measuring, MonkeyCode's free model access is enough for the loop in Myth 1. Start with twenty runs and one prompt.