The Model Swap Gate: A One-Page SOP for Changing the Model Behind Your Agent

A single-line diff changes the model string, the PR says "no code changes", CI stays green, and the agent ships on a Friday afternoon. By Monday the triage agent is answering support tickets with confident, well-formatted, subtly wrong routing decisions nobody can trace to a commit. Nothing in the pipeline was broken, because nothing in the pipeline ever asked whether the behavior behind that string was still the behavior you tested.

This failure mode keeps showing up in agent teams, and it is not really a model problem. It is a missing gate: a decision point that says a model change is a behavior change, and behavior changes need evidence before merge. Below is a one-page SOP you can paste into a wiki, plus a small harness you can run today.

Why a swap deserves its own gate

Your unit tests verify code paths, and your evals probably verify prompt templates, but neither one owns the model identifier. When the identifier moves, temperature handling, instruction-following, output shape, and refusal behavior can all drift at once, and each drift looks like a normal successful API response. A swap gate is deliberately narrow: it does not try to prove the new model is better, it only proves the change is not silently worse on cases you already care about.

Keep the gate cheap enough that nobody routes around it. A dozen to forty labeled cases, one script, one reviewer, and a written verdict is usually enough to catch the embarrassing class of regressions.

Three roles, one handoff

• Swap author — makes the change, assembles the candidate run, and records the exact model identifiers used.
• Gate reviewer — reruns the harness from the author's command transcript and signs the verdict; cannot be the same person.
• Behavior owner — owns the golden set and decides whether a newly failing case is a real regression or a stale expectation.

The handoff is a single artifact: a gate report with a pass rate, a list of failing case IDs, and a one-line verdict. If the report cannot stand alone without a meeting, it is not a report yet.

The artifact: a labeled golden set and a diff harness

Start with JSONL, one case per line, containing the input, hard requirements, and a criticality flag. Hard requirements are the strings or facts that must appear and the ones that must never appear; keep them boring and mechanical.

{"id":"triage-001","critical":true,"prompt":"Customer says their invoice shows a charge from a plan they cancelled. Route to the right queue and state the reason in one sentence.","must_include":["billing"],"must_not_include":["refund issued"]}
{"id":"triage-002","critical":false,"prompt":"A user reports the export button returns a 500 error on CSV only. Route and name the failing surface.","must_include":["bug","csv"],"must_not_include":["outage"]}
{"id":"triage-003","critical":true,"prompt":"A message contains only the word 'help'. Respond that more detail is needed, and do not guess a queue.","must_include":["more detail"],"must_not_include":["billing","bug"]}

Then a script that runs the candidate, runs the baseline when one is configured, and exits non-zero when the verdict is block. This example targets a chat-completions style endpoint; if your provider's request body differs, adapt the request and keep the verdict logic intact.

# gate.py — candidate vs baseline, exit 1 on BLOCK
import json, os, sys, urllib.request

BASE_URL = os.environ["MODEL_BASE_URL"].rstrip("/")
API_KEY = os.environ["MODEL_API_KEY"]
CANDIDATE = os.environ["CANDIDATE_MODEL"]
BASELINE = os.environ.get("BASELINE_MODEL", "")
THRESHOLD = float(os.environ.get("PASS_THRESHOLD", "0.90"))
TIMEOUT = float(os.environ.get("TIMEOUT_S", "30"))


def ask(model, prompt):
    body = json.dumps({
        "model": model,
        "temperature": 0,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        BASE_URL + "/chat/completions",
        data=body,
        headers={"Content-Type": "application/json", "Authorization": "Bearer " + API_KEY},
    )
    with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
        payload = json.loads(resp.read())
    return payload["choices"][0]["message"]["content"].strip()


def check(case, output):
    low = output.lower()
    missing = [s for s in case.get("must_include", []) if s.lower() not in low]
    leaked = [s for s in case.get("must_not_include", []) if s.lower() in low]
    return (not missing and not leaked), missing, leaked


def main(path):
    cases = [json.loads(line) for line in open(path) if line.strip()]
    rows, passed = [], 0
    for case in cases:
        out = ask(CANDIDATE, case["prompt"])
        ok, missing, leaked = check(case, out)
        passed += ok
        base_out = ask(BASELINE, case["prompt"]) if BASELINE else None
        rows.append({
            "id": case["id"], "critical": case.get("critical", False),
            "pass": ok, "missing": missing, "leaked": leaked,
            "baseline_identical": (base_out == out) if base_out is not None else None,
            "output": out[:400],
        })
    rate = passed / len(cases)
    critical_fail = [r["id"] for r in rows if r["critical"] and not r["pass"]]
    verdict = "BLOCK" if critical_fail or rate < THRESHOLD else "PASS"
    print(json.dumps({
        "candidate": CANDIDATE, "baseline": BASELINE or None,
        "cases": len(cases), "pass_rate": round(rate, 3), "threshold": THRESHOLD,
        "critical_failures": critical_fail, "verdict": verdict, "results": rows,
    }, indent=2))
    sys.exit(1 if verdict == "BLOCK" else 0)


if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "cases.jsonl")

One design note: printing baseline_identical is often more useful than the pass rate alone. A candidate that passes but disagrees with the baseline on half the cases is telling you the gate is under-specified, not that the swap is safe.

Run it in six numbered steps

1. Freeze the golden set for this swap and record its commit hash in the report.
2. Export the endpoint, key, candidate identifier, and baseline identifier as environment variables in your shell.
3. Run the harness with a fixed seed-free temperature of zero and capture the JSON to a file.
4. Rerun the exact same command once more to catch obvious nondeterminism before you argue about the numbers.
5. Send the report plus the command transcript to the gate reviewer, not to a chat thread.
6. Merge only after the reviewer signs a verdict line; attach the report path to the PR description.

export MODEL_BASE_URL="$GATE_ENDPOINT"
export MODEL_API_KEY="$GATE_KEY"
export CANDIDATE_MODEL="$NEW_MODEL_ID"
export BASELINE_MODEL="$CURRENT_MODEL_ID"
export PASS_THRESHOLD=0.90

python gate.py cases.jsonl | tee gate-report.json
python gate.py cases.jsonl > gate-report-rerun.json
diff <(jq '.results[].output' gate-report.json) <(jq '.results[].output' gate-report-rerun.json) && echo "stable"

If you want to compare several candidates before committing budget, this is exactly where a free model grant and a free server box earn their keep: cheap endpoints are enough to run a twelve-case gate in a few minutes, and the harness does not care which model string sits behind the URL. MonkeyCode's free model access and free server option are one way to stand that harness up quickly; the project's published terms describe a free token grant, and you should confirm the current allowance and server conditions on the project page rather than trusting a screenshot. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The decision table reviewers actually use

Signal from the report	Verdict	Who signs
Any critical case fails	BLOCK, no exceptions	Gate reviewer
Pass rate below threshold	BLOCK until cases are re-triaged	Behavior owner
Pass rate at or above threshold, stable rerun	PASS	Gate reviewer
Pass rate fine but many baseline disagreements	HOLD, expand golden set first	Behavior owner
Timeouts on more than a tenth of cases	HOLD as capacity risk, not quality	Swap author

That last row matters more than it looks. Timeouts during a gate often predict timeouts in production, and shared or free capacity makes latency noisy, so treat p95 latency as a separate observation rather than folding it into the pass rate.

Paste this into the wiki

## Model Swap Gate — <date>
- Author / Reviewer / Behavior owner: <names>
- Candidate: <model id>  Baseline: <model id>
- Golden set: <repo path> @ <commit>
- Report: <artifact path>   Rerun: <artifact path>
- Pass rate: <n>/<n> = <rate>   Critical failures: <ids or none>
- Known gaps: <tool-calling not covered / multi-turn not covered / ...>
- Verdict: PASS | BLOCK | HOLD
- Reviewer sign-off: <name>, <date>

A wiki page like this takes four minutes to fill and saves the next person an afternoon of archaeology.

Limitations and who should skip this

String matching is brittle: a model that paraphrases correctly will fail a lazy must_include, so budget time to maintain cases and prefer normalized checks over exact phrases. A small golden set overfits quickly, and a gate that only ever sees twelve cases will happily bless a model that breaks on the thirteenth. The harness above covers single-turn text only; if your agent calls tools or runs multi-turn plans, this gate is a floor, not a ceiling.

Skip this approach if you cannot label a dozen realistic cases with a behavior owner who will actually maintain them, because an unmaintained gate becomes a rubber stamp within two sprints. Skip it as your only safeguard for irreversible actions such as payments or account deletion, where a human checkpoint belongs in the loop regardless of pass rate. And if you are still choosing a model rather than swapping one, spend the effort on a broader evaluation instead of a narrow gate.

The gate is unglamorous, and that is the point: it converts a one-line diff nobody reviewed into a decision somebody signed.