A Free Model Swap Is Not a Config Change: A Contract Probe You Can Run

You have seen this comment in a pull request. "Just point the base URL at the free model. Same API, right?"

It sounds reasonable. Most providers speak a chat-completions dialect now. So what could actually break?

More than you think. The request shape is shared. The response contract is not. And your parser only ever sees the response.

I keep one rule for model swaps. Never trust a swap you cannot diff. So here is a probe, four beliefs it tends to break, and the corrected model behind it.

"Same API" compares the wrong thing

Same API usually means same route and same field names. That is the easy part. It is also the part that fails loudest and earliest. You see a 404 in thirty seconds.

The quiet failures sit one layer down. Field presence, field shape, and error bodies. Those break retry logic at 3 a.m., not your curl command.

So compare contracts. Not endpoints.

Belief 1: Same route means same response contract

A route is a promise about a path. It says nothing about the body. Here is what tends to vary, judging by provider docs and issue threads:

• finish_reason values, and when they appear.
• Whether usage is present at all, and which counters it carries.
• Whether tool calls come back as a list, a string, or nothing.
• Error body keys (error.message vs detail vs a bare string).
• Streaming chunk boundaries, and where the last chunk lands.

None of that is exotic. All of it is contract.

Corrected model: the contract is what your parser accepts. Write that list down before you swap anything.

Belief 2: If it parses as JSON, the swap worked

This one collapses three different checks into one word. Split them:

1. Parse: is the body valid JSON?
2. Schema: does it have the fields your code indexes?
3. Meaning: does the content satisfy the task?

A free endpoint can pass level 1 and fail level 2. Plenty of responses return 200 with a body your code reads as None. Then you get an AttributeError three frames deep.

Level 3 is the one nobody tests. Parsing tells you nothing about whether the answer is right.

Corrected model: track parse rate, schema rate, and task rate separately. One number hides the bug.

Belief 3: Usage numbers are a fact you can bill on

The usage block is optional in most dialects. Some endpoints return it. Some return it only sometimes. Some count differently.

Ask yourself a blunt question. Would your budget guard crash if usage disappeared tomorrow?

If yes, you have a bug, not a provider problem. Count tokens on your side with a local tokenizer, or count characters. Store both.

Corrected model: treat usage as a hint. Treat your own counter as the ledger.

Belief 4: Five prompts I wrote is an eval

Everyone does this. Five happy-path prompts, written by the person who wrote the integration. All pass. Ship it.

That is a smoke test, and a weak one. It proves the wire works. It proves nothing about the cases that page you.

Build the golden set from failure, not imagination. Pull five real prompts from your logs. Include the ones that already broke a parser.

Corrected model: small N is fine if the cases are hostile. Small N of friendly cases is theater.

The probe

Here is the artifact. It is stdlib Python, and it records shape, not quality. Adapt build_payload and the auth header to whatever request shape your provider documents.

# probe_contract.py - stdlib only. Adapt build_payload and the auth header.
import json, os, time, urllib.request, urllib.error

BASE = os.environ["PROBE_BASE_URL"].rstrip("/")
KEY  = os.environ["PROBE_API_KEY"]
PATH = os.environ.get("PROBE_PATH", "/chat/completions")

CASES = [
    {"id": "json_only",  "prompt": 'Return only this JSON: {"ok": true}'},
    {"id": "tool_shape", "prompt": 'Call the tool "ping" with {"host": "a"}.'},
    {"id": "stop_early", "prompt": "Write 400 words, but stop after one sentence."},
    {"id": "empty_in",   "prompt": "Summarize this document: ''"},
]

def build_payload(case):
    return {"messages": [{"role": "user", "content": case["prompt"]}]}

def call(case):
    body = json.dumps(build_payload(case)).encode()
    req = urllib.request.Request(BASE + PATH, data=body, method="POST")
    req.add_header("content-type", "application/json")
    req.add_header("authorization", f"Bearer {KEY}")
    t0 = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=60) as r:
            status, raw = r.status, r.read()
    except urllib.error.HTTPError as e:
        status, raw = e.code, e.read()
    except Exception as e:
        status, raw = -1, str(e).encode()
    return status, raw, int((time.monotonic() - t0) * 1000)

def shape(status, raw, ms):
    row = {"status": status, "ms": ms, "json": "no", "finish": "-",
           "usage": "absent", "tools": "absent", "err_keys": "-", "head": "-"}
    try:
        doc = json.loads(raw)
    except Exception:
        row["head"] = raw[:60].decode("utf-8", "replace")
        return row
    row["json"] = "yes"
    if status >= 400 or status < 0:
        row["err_keys"] = ",".join(sorted(doc)[:4]) or "-"
        return row
    choice = (doc.get("choices") or [{}])[0]
    msg = choice.get("message") or {}
    row["finish"] = choice.get("finish_reason") or "-"
    row["usage"] = "present" if doc.get("usage") else "absent"
    row["tools"] = "present" if msg.get("tool_calls") else "absent"
    row["head"] = (msg.get("content") or "")[:60].replace("\n", " ")
    return row

def main():
    rows = []
    for case in CASES:
        for run in (1, 2):
            s, raw, ms = call(case)
            r = shape(s, raw, ms)
            r.update({"case": case["id"], "run": run})
            rows.append(r)
    cols = ["case", "run", "status", "json", "finish", "usage", "tools", "ms"]
    print("| " + " | ".join(cols) + " |")
    print("|" + "|".join(["---"] * len(cols)) + "|")
    for r in rows:
        print("| " + " | ".join(str(r[c]) for c in cols) + " |")
    with open("probe-receipt.json", "w") as f:
        json.dump(rows, f, indent=2)

if __name__ == "__main__":
    main()

Run it twice against each candidate. Then diff the two receipts.

PROBE_BASE_URL=https://a.example/v1 PROBE_API_KEY=$KEY_A python probe_contract.py > a.md
PROBE_BASE_URL=https://b.example/v1 PROBE_API_KEY=$KEY_B python probe_contract.py > b.md
diff -u a.md b.md

Commit both receipts. Now the swap has evidence attached to it.

How to read the receipt

Illustrative shape only. These are not measurements from any real run.

case	run	status	json	finish	usage	tools	ms
json_only	1	200	yes	stop	present	absent	-
tool_shape	1	200	yes	stop	absent	absent	-
stop_early	2	200	yes	length	absent	absent	-
empty_in	1	400	yes	-	absent	absent	-

Read it like a checklist, not a score:

• json=no on a case that must return JSON. Stop. Your parser dies here.
• tools=absent when you need tool calls. Stop, or accept a workaround and retest.
• finish=length where you asked it to stop. Your stop handling differs. Test it.
• usage=absent. Drop any logic that assumes it exists.
• status=400 with unfamiliar err_keys. Your error classifier needs a branch.

Do not read the ms column as a benchmark. It is one request, twice. That is a smoke signal, not a latency claim.

Decision table

Receipt signal	Swap as-is	Add adapter	Reject
All cases: json=yes, expected fields present	yes	-	-
json=yes, one field missing but derivable	-	yes	-
Any parse fail on a JSON-only case	-	maybe	yes
Tools absent and tool calls are core to your flow	-	-	yes
4xx bodies in an unknown shape	-	yes	-
Usage absent and you bill from it	-	yes	-

The middle column is where most swaps actually land. That is fine. Just make the adapter explicit, not accidental.

Where a free endpoint fits

This probe is provider-agnostic. Any endpoint you can reach works. MonkeyCode's free model access is one place to get an endpoint for the run. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two practical notes if you go that route. Keep the key in an env var, never in the script. And run the probe from the same network location as your app. If you deploy on the free server option, run it there. The path affects ms, and path is part of what you are diffing.

Limitations, and who should skip this

Be honest about what a probe is not:

• It does not grade answer quality. No rubric, no judge.
• It does not test long context, streaming edge cases, or concurrency.
• It is not a load test. Four cases twice will not find your saturation point.
• It will not catch semantic drift on prompts you never ran.

Skip this approach if you need graded evals with human review. Also skip it if you have no fixed output schema. Without a contract, there is nothing to diff.

And if your product bills on token counts, this probe is a starting point only. Reconcile against real invoices before you trust any counter.

The question to ask first

Before the next "it's the same API" comment lands, ask one thing. Which fields does my code actually index?

That list is your contract. The probe measures it in a minute. Then the swap stops being a vibe and becomes a diff.