A 200 Response Is Not a Contract: Canary Tests for Third-Party Model Endpoints

A provider can rename a field, change what finish_reason means, or quietly swap the model behind your endpoint. Your client keeps getting HTTP 200 the whole time. Logs look green, and a user is staring at a half-finished sentence with no error attached.

This is a testing article, not a model-selection article. It assumes you already picked an endpoint — paid, self-hosted, or free — and asks a narrower question: how do you find out that its behaviour moved before your users do?

The answer is a contract test suite plus a nightly canary that runs somewhere that isn't your product's host. Both are small. Both are boring. That is the point.

The failure classes worth testing

Not everything a provider does is a bug. Some of it is a documented version change that still breaks your call site. Group the risk into five buckets and test each one cheaply:

1. Shape drift — a key disappears, gets renamed, or changes type (finish_reason: "length" becomes "max_tokens").
2. Error semantics — a malformed request returns 200 with an empty string instead of 400 with a JSON error body.
3. Truncation honesty — you asked for 8 tokens on a long prompt; does the response tell you it was cut short?
4. Accounting drift — reported token usage stops matching the payload it describes, and your cost math silently rots.
5. Transport behaviour — new status codes, changed retry hints, or rate-limit headers that vanish.

Quality is a sixth category, and it does not belong here. More on that below.

Write the contract as assertions

Prose contracts get skimmed. Assertions fail the build. Below is a proposal you can adapt — it is not a record of a run against any specific provider, and the paths are placeholders.

# tests/test_model_contract.py
import os
import httpx
import pytest

BASE = os.environ["MODEL_BASE_URL"]
KEY = os.environ.get("MODEL_API_KEY", "")
HEADERS = {"Content-Type": "application/json"}
if KEY:
    HEADERS["Authorization"] = f"Bearer {KEY}"


def complete(prompt, max_tokens=64, **params):
    body = {"prompt": prompt, "max_tokens": max_tokens, **params}
    return httpx.post(f"{BASE}/v1/complete", json=body, headers=HEADERS, timeout=45)


def test_success_shape_is_stable():
    r = complete("Reply with the word ready.")
    assert r.status_code == 200
    data = r.json()
    assert isinstance(data.get("text"), str)
    assert data.get("finish_reason") in {"stop", "length"}


def test_truncation_is_flagged():
    r = complete("Describe a river in detail.", max_tokens=8)
    data = r.json()
    assert data["finish_reason"] == "length", "silent truncation breaks every summariser"
    assert len(data["text"]) > 0


def test_bad_input_is_a_client_error():
    r = complete("")
    assert r.status_code in (400, 422), "a 200 with empty text is the worst possible answer"
    assert "application/json" in r.headers.get("content-type", "")


def test_usage_is_present_and_plausible():
    r = complete("Count to three.", max_tokens=32)
    usage = r.json().get("usage") or {}
    assert isinstance(usage.get("total_tokens"), int)
    assert usage["total_tokens"] >= usage.get("completion_tokens", 0)

Four tests, maybe sixty lines, and they catch the majority of the shape and error-semantics breaks that reach production.

Separate blocking checks from advisory ones

Mark the weak assertions. Determinism at temperature=0 is a good example: many providers are nearly deterministic, and a flaky test that fails two nights a week trains your team to ignore the whole suite.

@pytest.mark.advisory
def test_same_seed_same_output():
    a = complete("Say exactly: ok", temperature=0).json()["text"]
    b = complete("Say exactly: ok", temperature=0).json()["text"]
    assert a == b

Run blocking checks on every deploy. Run advisory ones nightly and collect them as data rather than exit codes.

Put the canary off the product path

A canary that shares a box with your API tells you two things at once and separates neither. If the canary gets slow, is that the provider or your own traffic spike?

Run it elsewhere. A small spare host is enough — this is one of the few legitimate uses for a free server tier, since the job is short, bursty, and never user-facing. A cron entry is all the orchestration it needs.

# canary/run.sh
set -euo pipefail
STAMP="$(date -u +%F)"
mkdir -p canary/reports
python canary/probe.py --out "canary/reports/${STAMP}.json"

# canary/probe.py — one JSON report per day, committed or uploaded.
import argparse, hashlib, json, os, time, urllib.error, urllib.request

BASE = os.environ["MODEL_BASE_URL"]
KEY = os.environ.get("MODEL_API_KEY")
CASES = json.load(open("canary/cases.json"))


def run(case):
    body = json.dumps({"prompt": case["prompt"], **case.get("params", {})}).encode()
    headers = {"Content-Type": "application/json"}
    if KEY:
        headers["Authorization"] = f"Bearer {KEY}"
    req = urllib.request.Request(f"{BASE}/v1/complete", body, headers)
    started = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=45) as resp:
            status, payload = resp.status, json.loads(resp.read())
    except urllib.error.HTTPError as err:
        status, payload = err.code, json.loads(err.read() or b"{}")
    return {
        "name": case["name"],
        "status": status,
        "ms": round((time.perf_counter() - started) * 1000),
        "keys": sorted(payload) if isinstance(payload, dict) else [],
        "finish_reason": (payload or {}).get("finish_reason"),
        "text_len": len(str((payload or {}).get("text", ""))),
        "sha": hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()[:12],
    }


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--out", required=True)
    args = ap.parse_args()
    report = {"cases": [run(c) for c in CASES]}
    json.dump(report, open(args.out, "w"), indent=2)

Eight to twelve cases are enough. Pick cases that map one-to-one onto the failure classes above, not prompts you find interesting.

Diff yesterday, don't read today

A report nobody compares is a log file. Diff consecutive reports and print one line per case.

# canary/diff.py
import json, sys

load = lambda p: {c["name"]: c for c in json.load(open(p))["cases"]}
old, new = load(sys.argv[1]), load(sys.argv[2])

for name in sorted(new):
    prev, cur = old.get(name, {}), new[name]
    flags = []
    if prev.get("keys") != cur["keys"]:
        flags.append("shape")
    if prev.get("status") != cur["status"]:
        flags.append(f"status {prev.get('status')}->{cur['status']}")
    if prev.get("finish_reason") != cur["finish_reason"]:
        flags.append("finish_reason")
    if prev.get("sha") != cur["sha"]:
        flags.append("hash")
    if (prev.get("text_len") or 0) and cur["text_len"] < 0.5 * prev["text_len"]:
        flags.append("shorter")
    print(f"{name:24} {'OK' if not flags else ' '.join(flags)}")

Then triage with a table instead of a meeting:

Signal in the diff	What it usually means	First move
shape (missing key)	breaking version bump	patch the adapter, pin the client, ship same day
status 200->4xx	auth or request-format change	read the error body, fix headers before retrying
finish_reason changed	new vocabulary for truncation	update your client mapping, re-test the UI path
shorter with same finish reason	output cap changed	check max-token handling and cost math
hash only, everything else OK	sampling noise or model swap	do not fail the build; run the quality eval
ms up 3x, payload identical	capacity or routing change	add queue slack, do not rewrite the client

That last-but-one row matters. Output hash drift is the weakest signal and the loudest alarm. Keep it out of the blocking set.

Where free tiers and a spare server fit

Contract tests catch breaks in your endpoint. They get stronger when you have a second endpoint to diff against, because you can tell a client bug from a provider change.

MonkeyCode is an open-source project that offers free model access and a free server option. In this workflow it plays two narrow roles. The free models give the canary suite a second column to compare against without opening a procurement ticket, so a finish_reason change on one endpoint can be checked against the other before you rewrite anything. The free server option hosts the nightly probe itself, which keeps a short bursty job off your product's capacity. The project also advertises a free token allotment; treat the size of that allotment as a term to re-verify, not a constant to hardcode into your plans.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Neither role involves user traffic. If your canary host goes down for a night, you lose one row of a diff, not an outage.

What these tests cannot tell you

Be blunt about the boundary, because the boundary is where teams get hurt:

• Quality. A schema-perfect response can still be wrong. Contract tests never replace an eval set with graded answers.
• Cost. Usage fields tell you what the provider counted, not what your budget tolerates. That is a separate gate.
• Secrets. Nothing here checks whether a prompt should have left your network at all.
• Permanence. A canary proves behaviour today. It cannot promise the free tier exists next quarter.
• Rate limits. A canary sends a handful of requests. It will never surface the throttling your real traffic hits.

Who should skip this

Skip it if you control the model binary and deploy it yourself — you have deploys, not upstream drift, and your integration tests already cover the shape.

Skip it if you have no users yet. Twelve canary cases will cost you more attention than the bug they prevent.

Skip the second-endpoint column if you have no easy way to store daily reports. A diff needs history, and history needs somewhere to live.

If none of those apply, start with the four blocking tests and one cron job. The first time a renamed field shows up in a diff instead of a support ticket, the sixty lines pay for themselves.

If you want a second endpoint to diff against without a budget conversation, pointing the canary at the project's free models is the cheapest way to get one — just keep it in the canary column, never on the user path.