Freeze the Model Graph Before Merge: A Fail-Closed Identity Checklist

If you cannot name the exact model identifier, prompt bytes, and tool schemas that produced a passing eval, you do not have a releasable agent. Aliases are not versions. A green CI run against latest is a screenshot of a moving target.

You already gate on tests, timeouts, and rollback windows. This checklist covers a different failure: identity drift. The agent still compiles. The fixtures still parse. The model behind the friendly name moved.

The failure unit tests will not show you

Agent PRs fail in production for boring reasons. A provider retargets an alias. A free endpoint swaps the checkpoint behind the same label. A teammate edits a system prompt and forgets that CI loads a different path.

Your unit tests keep passing. They never asked who answered.

Tool-calling stacks make this worse. The model is only one node. Prompts, tool JSON schemas, retrieval snapshots, and decoding knobs are the rest of the graph. If any node is unpinned, the eval is not replayable. If it is not replayable, it is not evidence. Merge evidence, not vibes.

Freeze this graph, not a dashboard default

Treat the runtime the way you treat a lockfile. You would not ship lodash@latest next to a payment charge. Do not ship chat-model@latest next to a tool that can write.

Pin all five:

1. Model identity — a concrete model id. Not default, auto, latest, or an undocumented alias.
2. Prompt bytes — SHA-256 of the exact system and developer prompts loaded at eval time.
3. Tool schemas — SHA-256 of the serialized allowlist the model was allowed to see.
4. Decoding knobs — temperature, top_p, max output tokens, and seed if the API exposes one.
5. Corpus snapshot — a git SHA or object digest. Not “whatever is in the bucket today.”

Missing any one of those is a merge blocker. Do not warn. Fail closed.

Copy-paste gates, evidence, and fail-closed rules

Use this as a PR template. Every row is a gate. “We’ll watch it in staging” is not evidence.

Gate A — Identity file exists

Evidence: agent-lock.json is committed in the same PR as the agent change.

Fail closed if: the file is missing, untracked, or generated after the eval instead of before it.

Gate B — No alias tokens

Evidence: model_id is a concrete vendor identifier, not a moving label.

Fail closed if: the value is empty or matches latest, default, auto, free, or current.

Gate C — Digests match the git tree

Evidence: CI recomputes hashes from the checkout and diffs them against the lock file.

Fail closed if: lock hash and tree hash disagree, even by one byte.

Gate D — Eval ran against the lock

Evidence: artifacts/eval-report.json echoes model_id, hashes, and knobs verbatim.

Fail closed if: the report omits a lock field, or the echoed values do not match.

Gate E — Replay fixture stored

Evidence: one redacted golden request/response plus the tool-call trace for the critical path.

Fail closed if: you cannot replay the eval locally with the lock file and the fixture alone.

Gate F — Tool allowlist is explicit

Evidence: the tool list in the lock file is an allowlist. Runtime cannot inject unknown names.

Fail closed if: the agent discovers tools from the environment without a lock-file match.

A reproducible CI artifact

Label this as a proposed gate. Wire it to your runner. It does not call a vendor. It only refuses to merge an unpinned graph.

Create agent-lock.json before you run evals:

{
  "model_id": "REPLACE_WITH_CONCRETE_ID",
  "prompt_sha256": "",
  "tools_sha256": "",
  "temperature": 0,
  "max_output_tokens": 1024,
  "seed": 7,
  "corpus_git_sha": "none",
  "eval_suite": "tests/agent_eval"
}

Empty strings are illegal. If you have no retriever, write none on purpose. Implicit nothing is how aliases sneak back in.

Hash the prompts and tools from the tree, not from memory:

#!/usr/bin/env bash
set -euo pipefail

prompt_sha=$(cat prompts/system.md prompts/developer.md | sha256sum | awk '{print $1}')

tools_sha=$(python - <<'PY'
import hashlib, json, pathlib
raw = pathlib.Path("tools/allowlist.json").read_text()
data = json.loads(raw)
canonical = json.dumps(data, sort_keys=True, separators=(",", ":"))
print(hashlib.sha256(canonical.encode()).hexdigest())
PY
)

echo "prompt_sha256=${prompt_sha}"
echo "tools_sha256=${tools_sha}"

Paste those digests into the lock file, commit it, then fail closed in CI:

#!/usr/bin/env bash
# proposed gate: reject unpinned agent graphs
set -euo pipefail

lock="agent-lock.json"
test -f "${lock}" || { echo "missing ${lock}"; exit 1; }

python - "${lock}" <<'PY'
import hashlib, json, pathlib, re, sys

lock = json.loads(pathlib.Path(sys.argv[1]).read_text())
required = [
    "model_id", "prompt_sha256", "tools_sha256",
    "temperature", "max_output_tokens", "corpus_git_sha", "eval_suite",
]
missing = [k for k in required if lock.get(k) in (None, "")]
if missing:
    raise SystemExit(f"lock missing fields: {missing}")

model = str(lock["model_id"])
if re.search(r"latest|default|auto|free|current", model, re.I):
    raise SystemExit(f"alias refused: {model}")

prompt = (
    pathlib.Path("prompts/system.md").read_bytes()
    + pathlib.Path("prompts/developer.md").read_bytes()
)
if hashlib.sha256(prompt).hexdigest() != lock["prompt_sha256"]:
    raise SystemExit("prompt hash mismatch")

tools = json.dumps(
    json.loads(pathlib.Path("tools/allowlist.json").read_text()),
    sort_keys=True,
    separators=(",", ":"),
).encode()
if hashlib.sha256(tools).hexdigest() != lock["tools_sha256"]:
    raise SystemExit("tools hash mismatch")

print("agent-lock.json: pinned and matched")
PY

Add a contract so the eval cannot “pass” while talking to a different model:

# tests/test_eval_echoes_lock.py
# proposed test: the eval runner must echo the lock, not a UI default
import json
from pathlib import Path


def test_eval_report_echoes_lock():
    lock = json.loads(Path("agent-lock.json").read_text())
    report = json.loads(Path("artifacts/eval-report.json").read_text())
    for key in ("model_id", "prompt_sha256", "tools_sha256", "temperature"):
        assert report.get(key) == lock[key], f"{key} drifted between lock and eval"

Run the same sequence on every PR:

mkdir -p artifacts
# your eval runner must write artifacts/eval-report.json
# including the lock fields it actually used
pytest tests/test_eval_echoes_lock.py tests/agent_eval -q

If the eval runner cannot print the model id it used, it is not a runner you can merge with. Throw it out.

Decision table you can paste into the PR

Symptom	What you probably did not pin	Merge?
Pass locally, fail in CI	prompt path or working directory	No
Pass Monday, fail Thursday, no code change	model alias	No
Same model id, different tool calls	tool schema or retrieval corpus	No
Golden trace will not replay	temperature, seed, or hidden preamble	No
New tool appeared at runtime	allowlist not enforced	No

Print the table. If the author cannot point to a lock field for the symptom, the PR stays closed.

Where a free eval lane belongs

You still need somewhere to run the suite when a laptop cannot. That is an eval lane. It is not production. Keep customer data, production credentials, and write-capable tools off it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. Point the same pin-and-prove job at that lane if you want hashes computed and eval-report.json published off your laptop. Do not promote that server to a write-enabled agent. If the lane cannot echo a concrete model id into the report, fail closed. Steal the lock file even if you never use that lane. The gate is the point.

Limitations, and who should skip this

This approach assumes you can read a durable model identifier and that your eval is cheaper than a production incident. Some hosted APIs never expose a stable id. If that is your vendor, this checklist cannot save you. Switch, or wrap the call behind a proxy you control.

A free eval lane can also disagree with production. Different hardware, different batching, different hidden preambles. Pinning does not make two endpoints identical. It only makes which endpoint you tested an explicit merge fact. If you cannot afford a production-shaped replay, do not pretend the free lane is one.

Do not use this as a substitute for:

• security review of tool side effects
• PII redaction in traces
• load tests
• a rollback plan

A pinned bad prompt is still a bad prompt. Hashing it only makes the mistake reproducible.

Skip the full lock file if your “agent” is a single offline classifier with no tools and no network. You are not in the failure mode this gate exists for. Add it the moment the model can call anything with a side effect.

Merge rule

No lock file, no echoed eval, no merge. Aliases are a paging incident you scheduled for yourself. Pin the graph. Prove the eval saw that graph. Then ship.