Weekend Build Log: A Model-Swap Smoke Test You Can Run Before Monday

Friday night, your side project starts returning different answers. Same prompt, same input, new model version behind the endpoint. You find out on Sunday, when a friend tries the demo.

You do not need an eval platform to catch this. You need twelve fixtures, one script, and a red/green gate. Here is the build I would run this weekend, including the parts I would skip.

The scope cut

Three rules keep this to one evening.

1. Ship a CLI. No web UI, no database, no auth.
2. Score with string rules, not a second model.
3. Store every run as one JSON file you can diff in git.

Everything else is a later problem. The only output you need is a pass rate and an exit code.

Step 1: Write the fixtures you actually care about

Create cases.jsonl. Each line is one promise your project makes to a user.

{"id": "refund-policy", "prompt": "A customer asks for a refund 40 days after purchase. Answer in under 300 characters.", "must_contain": ["30 days"], "must_not_contain": ["guarantee", "always"], "max_chars": 300}
{"id": "json-shape", "prompt": "Return only JSON with keys title and tags for: 'weekly digest bot'. No prose.", "must_contain": ["title", "tags"], "must_not_contain": ["```

"], "max_chars": 400}
{"id": "empty-input", "prompt": "Summarize the following text: '' (empty). Reply with the single word EMPTY.", "must_contain": ["EMPTY"], "must_not_contain": [], "max_chars": 20}

Twelve is enough to start. Pick fixtures from three buckets.

• Output shape: JSON keys, no code fences, length limits.
• Business rules: numbers, dates, forbidden promises.
• Degenerate input: empty strings, very long strings, non-English text.

If a fixture cannot fail, delete it. It is decoration.

Step 2: The runner, standard library only

No SDK. No dependency drift. This file runs anywhere Python 3.9+ exists.

python
# smoke.py
import json, os, sys, urllib.request

BASE = os.environ["MODEL_BASE_URL"].rstrip("/")
KEY = os.environ["MODEL_API_KEY"]
MODEL = os.environ["MODEL_NAME"]

def call(prompt: str) -> str:
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.load(r)["choices"][0]["message"]["content"]

The endpoint shape is the generic chat-completions one. Take the base URL, key, and model name from whichever provider you are testing. Never hardcode them in the repo.

Step 3: Score with rules, not vibes

Each case returns pass or fail. Nothing in between.

python
def check(case: dict, out: str) -> list[str]:
    fails = []
    for needle in case.get("must_contain", []):
        if needle not in out:
            fails.append(f"missing:{needle}")
    for needle in case.get("must_not_contain", []):
        if needle in out:
            fails.append(f"forbidden:{needle}")
    limit = case.get("max_chars")
    if limit and len(out) > limit:
        fails.append(f"too_long:{len(out)}>{limit}")
    return fails

This looks crude. It is crude. It also catches every silent regression I have hit: a broken JSON contract, a dropped disclaimer, a model that suddenly writes essays.

Step 4: Run it against two models and diff

The main loop writes a scorecard, then sets the exit code.

python
cases = [json.loads(line) for line in open("cases.jsonl", encoding="utf-8") if line.strip()]
rows = []
for case in cases:
    try:
        out = call(case["prompt"])
    except Exception as exc:
        rows.append({"id": case["id"], "ok": False, "fails": [f"error:{exc}"]})
        continue
    fails = check(case, out)
    rows.append({"id": case["id"], "ok": not fails, "fails": fails, "chars": len(out)})

passed = sum(r["ok"] for r in rows)
scorecard = {"model": MODEL, "passed": passed, "total": len(rows), "rows": rows}
json.dump(scorecard, open("scorecard.json", "w"), indent=2)
print(f"{passed}/{len(rows)} passed on {MODEL}")
sys.exit(0 if passed == len(rows) else 1)

Run it twice into separate files, then diff:

bash
MODEL_NAME=model-a python smoke.py && mv scorecard.json scorecard.a.json
MODEL_NAME=model-b python smoke.py && mv scorecard.json scorecard.b.json
diff <(jq -S .rows scorecard.a.json) <(jq -S .rows scorecard.b.json)

The diff tells you which behaviors are model-sensitive. That list is your real risk surface, not the leaderboard.

Step 5: Gate the deploy

Wire the script into CI as one job. Fail the build on a non-zero exit. Now a provider swap cannot silently reach users.

Keep the scorecard as a build artifact. Two weeks later, when someone asks "did quality drop", you open a file instead of arguing.

Where free access fits, honestly

Iterating on a harness is a high-call-count activity. You run the suite dozens of times while tuning fixtures, and most of those calls produce nothing you keep. That is exactly the workload free model access helps with, and a free server option helps with the scheduled run that repeats the suite nightly.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two caveats you should hold onto. Free availability is a moving target, so read the project's current offer page before you plan a schedule around it. Never put the free tier on the critical path of a user-facing request either. Use it for iteration and for the nightly check, then keep a paid or local fallback ready.

A decision table for your own project

Your situation	Use this harness	Skip it
Pipeline parses model output into structured data	Yes, rules are enough
Output has legal or safety language	Yes, plus human sign-off
One-shot creative writing, no contract		Yes, rules will lie to you
Per-call cost already trivial, no swap planned		Yes, low value
You change model or prompt weekly	Yes, this pays for itself fast

What I skipped, and who should not do this

I skipped a web dashboard. JSON in git beats a chart nobody opens.
I skipped a model-as-judge. It adds latency and its own drift.
I skipped latency and cost capture. Add those only when they become the problem.

Three groups should not use this approach. Teams whose outputs need human review on every call get little from string rules. Long-form creative products will drown in false failures. And anyone who cannot pin a model name has nothing stable to test.

Also be honest about determinism. Temperature zero still is not bit-identical across runs. Run the suite twice before you trust a green light.

The Sunday demo

Your demo should not be the first time you learn the endpoint changed. Twelve fixtures and forty lines of Python buy you a Monday morning without surprises.

If you want to try the harness against a free endpoint first, check the current offer page for what is actually available this month, then point MODEL_BASE_URL at it.