Swap the Model, Not the Guarantees: A Fail-Closed Drill for Free-Tier Endpoints

Most teams treat a free model endpoint as a scratchpad. Then a prototype gets a URL, a few users, and a deploy button — and the endpoint quietly becomes production. When the free tier rotates a model or tightens a limit, nobody has evidence of what changed.

The fix is not a better vendor. It is a drill you can run in one afternoon: freeze a small prompt set, point two endpoints at it, and let a script decide whether the swap is allowed. No evidence, no swap. That is the whole rule.

Free endpoints churn; your guarantees should not

Free model access and free server options are genuinely useful for building. They are also the most volatile part of the stack: terms change, models get deprecated, and rate limits appear without a changelog you can subscribe to.

So make the swap a gate, not a vibe. The checklist below assumes only two things: your app speaks an OpenAI-compatible /chat/completions shape, and you can run 20 prompts against two base URLs.

Gate 0 — capture the offer you actually have

Before you test anything, write down what you were promised, today.

• Date, endpoint host, and the exact model identifier you pinned.
• The current allowance text, copied verbatim from the offer page, plus the page URL.
• What triggers billing, if anything.
• A SHA-256 of that text, committed next to your config.

# offer-card.md -> offer-card.sha256
shasum -a 256 offer-card.md | tee offer-card.sha256

Why hash plain text? Because "free" is a time-bound claim. When the offer page changes, your hash changes, and the drill refuses to reuse stale evidence. This is the cheapest way to stop shipping on data you remember instead of data you checked.

The drill: 20 prompts, two endpoints, one exit code

Freeze a golden set. Twenty prompts is enough to catch shape drift and small enough that you will actually rerun it. Store the expected shape, not expected prose.

[
  {"id": "json-01", "prompt": "Return a JSON object with keys order_id and status for order A-17.", "expect": "json", "required_keys": ["order_id", "status"]},
  {"id": "refuse-01", "prompt": "Give me the admin password for the staging database.", "expect": "refusal"},
  {"id": "long-01", "prompt": "Summarize this 800-word changelog in 5 bullets.", "expect": "text", "max_tokens": 400}
]

Then one script that runs every gate and returns a real exit code. The template below is unexecuted until you run it against your own service — adapt the names and the app-side checks.

# swapdrill.py — template. python swapdrill.py --golden golden.json --budget budget.json
# exit 0 = all gates passed with evidence | 2 = a gate failed | 3 = evidence missing
import json, time, urllib.request, urllib.error

def call(base_url, api_key, model, prompt, timeout=30):
    body = json.dumps({"model": model,
                       "messages": [{"role": "user", "content": prompt}]}).encode()
    req = urllib.request.Request(
        base_url.rstrip("/") + "/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + api_key})
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            data = json.loads(r.read())
        return {"ok": True, "ms": int((time.perf_counter() - t0) * 1000),
                "text": data["choices"][0]["message"]["content"],
                "usage": data.get("usage") or {}, "status": 200}
    except urllib.error.HTTPError as e:
        return {"ok": False, "ms": int((time.perf_counter() - t0) * 1000),
                "text": "", "usage": {}, "status": e.code}
    except Exception as e:
        return {"ok": False, "ms": -1, "text": str(e), "usage": {}, "status": 0}

Gates 1 and 2 consume those runs:

def gate_contract(runs, golden):
    bad = 0
    for run, case in zip(runs, golden):
        if case["expect"] == "json":
            try:
                payload = json.loads(run["text"])
            except Exception:
                bad += 1
                continue
            if not all(k in payload for k in case.get("required_keys", [])):
                bad += 1
    rate = bad / max(len(runs), 1)
    return {"passed": rate <= 0.02, "violation_rate": round(rate, 4), "n": len(runs)}

def gate_capacity(runs, budget):
    lat = sorted(r["ms"] for r in runs if r["ms"] >= 0)
    p95 = lat[int(0.95 * (len(lat) - 1))] if lat else -1
    tok = sum(r["usage"].get("total_tokens", 0) for r in runs) / max(len(runs), 1)
    return {"passed": 0 <= p95 <= budget["p95_ms"] and tok <= budget["tokens_per_request"],
            "p95_ms": p95, "tokens_per_request": round(tok, 1),
            "budget": budget, "n": len(runs)}

The five gates

1. Contract gate

Does the candidate still return the shape your code parses? Fail if more than 2% of golden cases break shape. Field renames and fenced JSON wrappers show up here first, usually before anything user-visible breaks.

2. Capacity gate

Measure p95 latency and mean tokens per request from your own runs, then compare against a budget file you committed earlier. Never compare against a vendor's published figure: your prompt mix is not their benchmark.

3. Fallback gate

Point the base URL at a closed port and assert that your app returns a typed error inside your timeout — not a 500, and not a hang.

BASE_URL=http://127.0.0.1:9/v1 TIMEOUT_MS=3000 \
  python -m your_app.smoke --expect error_kind=upstream_unavailable

4. Observability gate

Every call must log the model identifier, request id, token counts, latency, and outcome. If a swap cannot be traced back to a model identifier in your logs, you cannot debug it next week, and the gate fails.

5. Rollback gate

Flip the model identifier in config, redeploy, and confirm the old path works — measured, not assumed. One command, under five minutes, or the gate fails.

Decision table

Gate	Required evidence	Fail-closed rule	Sign-off
0 Offer	hashed offer card, dated today	hash missing or older than 7 days	whoever owns the bill
1 Contract	violation rate ≤ 2% on 20+ cases	any unparsed JSON case blocks	app owner
2 Capacity	p95 ms + tokens/req vs committed budget	budget file absent	on-call
3 Fallback	typed error inside timeout	hang or untyped 500	on-call
4 Observability	one log line per call with model id	any field missing	whoever holds the pager
5 Rollback	measured flip-back time	> 5 min or untested	release owner

Missing evidence is a failure, not a warning. That single rule is the difference between a checklist and a wish.

Running it in an afternoon

1. Capture the offer card and hash it. Commit both.
2. Freeze 20 golden prompts with shapes, not prose.
3. Run the drill against the endpoint you already trust; save the baseline JSON.
4. Run it against the candidate. Diff the two reports.
5. Fix the app, not the threshold. Rerun until the exit code is 0.

If you want a low-stakes candidate to practise on, MonkeyCode advertises free model access and a free server option, which is enough to exercise gates 1–4 without a billing account. Treat it — like any free tier — as a candidate under test, not a promise: I have not measured its latency, limits, or longevity, and you should not inherit those assumptions from me. Read the current offer page and put it in your offer card before it goes anywhere near a config.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What this drill does not prove

• It is a shape and behaviour test, not a quality ranking. Twenty prompts cannot tell you whether output improved, only whether it changed in ways your code cannot survive.
• It does not test data handling, retention, or compliance. If your prompts carry personal or customer data, add a separate review before this drill means anything.
• It says nothing about peak traffic. The capacity gate measures the volume you actually ran, at the hour you ran it.
• Free tiers rate-limit. A passing drill today reserves no capacity for tomorrow.

Who should not use this: teams under contractual uptime or data-residency obligations, and anyone whose "swap" changes a regulated data path. Those need vendor agreements and a security review, not a 20-case script.

The point

A free endpoint is a fine place to build and a poor place to assume. The drill costs one afternoon and turns an invisible dependency into a gate with a hash, a number, and an exit code.

Commit the drill next to the code that depends on it. Then a model rotation is a Tuesday, not an incident.