If your agent dies after a model swap, it was never finished. You shipped a vendor accent inside a prompt. A green eval on one stack is still a maybe.
Take the position
You should treat model disagreement as a merge blocker. Treat that disagreement as a schema bug. Do not treat that disagreement as model personality.
Two callers must emit the same tool names. They must fill the same argument keys. Divergence means the task text is underspecified.
Paid models hide that hole every day. They guess the missing enums without asking you. They complete file paths you never declared.
Cheap models stay ruder, and that helps. They follow the schema you actually wrote. That rudeness is the cheapest bug report available.
Single-model green is a rumor
A single endpoint will only agree with itself. That kind of agreement is easy and misleading. It cannot falsify vendor lock-in at all.
Teams then write longer judges after each miss. They add extra rubric bullets for comfort. They still query the same vendor stack anyway.
That pattern is how measurement quietly dies. The model drifts and the judge drifts with it. Your repository never held a truly fixed contract.
You need a second caller on purpose. Make that caller cheaper than the primary path. Run it on a machine that is not your laptop.
Compare the contract, not the essay
Diff the tool shapes and skip the essay. Ignore the assistant's closing paragraph every time. Remember that prose is not your API.
Check only these fields:
- tool name
- argument key set
- required keys that arrived
- enum membership
- call order when order is part of the contract
Drop these fields every time:
- chatty summaries
- markdown flavor
- hedge words
- restated code with new comments
The essay is costume for the user. The tool list is the actual product. You should merge only on that product.
Proposed swap harness
The following harness is only a proposal. It is not a published benchmark. Do not cite it as production evidence.
1. Freeze a task fixture
{
"id": "pin-formatter",
"goal": "Pin the repo formatter to an exact version and open a PR.",
"must_include_tools": ["read_file", "edit_file", "run_tests"]
}
Keep the goals short and operational. Keep the allowed tools as a closed set. Never hide control flow inside style notes.
2. Extract a stable shape
# Proposed harness. Unexecuted example. Pin your own clients.
import json, os, sys, urllib.request
def chat(url, key, model, messages, tools):
payload = {
"model": model,
"messages": messages,
"tools": tools,
"temperature": 0,
}
req = urllib.request.Request(
url,
data=json.dumps(payload).encode(),
headers={
"Content-Type": "application/json",
"Authorization": "Bearer " + key,
},
method="POST",
)
with urllib.request.urlopen(req, timeout=60) as resp:
return json.load(resp)
def tool_shape(payload):
shapes = []
for choice in payload.get("choices", []):
message = choice.get("message", {})
for call in message.get("tool_calls") or []:
fn = call.get("function", {})
raw = fn.get("arguments") or "{}"
args = json.loads(raw) if isinstance(raw, str) else raw
keys = sorted(args) if isinstance(args, dict) else []
shapes.append({"name": fn.get("name"), "keys": keys})
return shapes
def main():
task = json.load(open(sys.argv[1]))
tools = json.load(open(sys.argv[2]))
messages = [{"role": "user", "content": task["goal"]}]
primary = chat(
os.environ["PRIMARY_URL"],
os.environ["PRIMARY_KEY"],
os.environ["PRIMARY_MODEL"],
messages,
tools,
)
shadow = chat(
os.environ["SHADOW_URL"],
os.environ["SHADOW_KEY"],
os.environ["SHADOW_MODEL"],
messages,
tools,
)
left = tool_shape(primary)
right = tool_shape(shadow)
json.dump(
{"id": task["id"], "primary": left, "shadow": right},
sys.stdout,
indent=2,
)
if left != right:
sys.stderr.write("tool shape disagreement\n")
sys.exit(2)
if __name__ == "__main__":
main()
Keep temperature at zero for this gate. Keep argument keys sorted before compare. Keep tool names literal, never normalized by hand.
3. Fail the job in CI
# Proposed GitHub Actions job. Replace secrets with yours.
name: model-swap
on: [pull_request]
jobs:
shadow:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: python swap_test.py tasks/pin-formatter.json tools.json
env:
PRIMARY_URL: ${{ secrets.PRIMARY_URL }}
PRIMARY_KEY: ${{ secrets.PRIMARY_KEY }}
PRIMARY_MODEL: ${{ secrets.PRIMARY_MODEL }}
SHADOW_URL: ${{ secrets.SHADOW_URL }}
SHADOW_KEY: ${{ secrets.SHADOW_KEY }}
SHADOW_MODEL: ${{ secrets.SHADOW_MODEL }}
Exit code 2 means you stop the merge. Do not bump the paid model to silence it. Fix the schema that leaked a vendor habit.
Decision table for review
Do not adjudicate disagreement inside chat threads. Put the observation on this table. Then follow the merge column without debate.
| Observation | Merge? | Why |
|---|---|---|
| Same names, same keys, same order | Yes | Contract survived the swap |
| Extra optional keys only on paid | Warn | Paid model is padding arguments |
| Cheap model skips a required tool | No | Prompt relies on vendor inference |
| Different tool names, same goal | No | Task text is ambiguous |
| Cheap model loops until timeout | No | Stop condition is unspecified |
| Both refuse a disallowed tool | Yes | Safety instruction survived |
| Only paid model hits a private plugin | No | You shipped lock-in |
Every "No" row is your bug. It is not the cheap model's personality. It is not a reason to hide the shadow run.
Why the shadow run must be remote and cheap
You need volume for this disagreement check. Isolation still matters more than raw volume. You cannot skip either constraint here.
A shadow model you never invoke is theater. A shadow model on your laptop is contaminated. Local files, caches, and tokens all leak.
You want a remote job you can rerun. You want a cost that does not train skipping. Cheap and remote is the point of the gate.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode is an open-source project for this workflow. It offers free model access and a free server option. Park the shadow caller there when you need a remote box. You avoid standing up a second invoice. Pin the model id in the job. Pin the base URL beside that id. Store the tool-shape JSON next to the trace. If that free path is down, fail the job closed.
The merge rule still stands without that project. Any pinned remote shadow can play this role.
Debug disagreement in a fixed order
Do not start by rewriting the system prompt. Start with the tool schema instead. Change only one variable per rerun.
Walk this list in order every time:
- Are tool names short verbs a second model can parse?
- Are enums closed lists instead of free strings?
- Are file paths constraints, not decorative examples?
- Did you bury branching rules in prose?
- Did you assume one vendor's function-call quirks?
Most so-called quality gaps die at step three. You wrote a suggestion, not a contract. The paid model treated the suggestion like law.
Tighten that one field and stop there. Rerun the pair on the same fixture. Keep both traces next to the schema change.
Limitations
This gate is blunt on purpose. It will punish agents that should improvise. It will also punish research spikes and sketch work.
It assumes both endpoints speak the same tool protocol. It assumes temperature remains zero during the compare. It assumes the task is tool-shaped and pinnable.
Agreement does not mean the edit was right. Two models can share one wrong refactor. This test catches lock-in, not business correctness.
You still need unit tests around the tools. You still need replay for the session. You still need a human on the patch.
Unpinned shadows are noise in CI. Noise does not belong on the merge gate.
Who should not use this
Skip this if you sell a single-vendor assistant on purpose. Skip it for story generation and marketing copy. Skip it for one-off notebooks with no tools.
Skip it if your tools are free text blobs. Diffing essays across models returns vibe scores. Those scores do not travel between vendors at all.
Skip it if you cannot name both models in the job. A floating alias is not a real fixture. Fixtures need identifiers that you can grep.
What to run this week
Pick five tasks from your real tickets. Reject demo prompts that have no tools. Reject poems and other toy summary prompts.
Run the proposed harness on those five. Read only the shape JSON output file. Apply the table without any extra debate.
Look for a hidden vendor guess in the shapes. Put that guess into the tool schema. Rerun the pair until the shapes match.
A model you can swap is an agent. A model you cannot swap is a demo. Keep the invoice off the critical path.
Top comments (0)