Stop Shopping for Models: Your Harness Is the Only Asset You Keep

The opinion, up front

Model choice is a commodity decision. The harness that proves a model is good enough is the asset. Most teams have this exactly backwards.

They spend three weeks reading leaderboards. They spend one afternoon writing the test that decides the switch. Then they argue about models again next quarter.

Free tiers make this worse, not better. Cheap access invites endless shopping. My position is blunt: pick a model in an hour, then spend a month hardening the swap path.

Everything below is that swap path.

Portability beats preference

You will change models. Prices move, endpoints deprecate, defaults shift under you. That is not a prediction. That is the normal lifecycle of a dependency.

If your prompts live inside SDK-specific calls, every change is a migration. If they live behind one HTTP shape, every change is an environment variable.

Three things survive a swap:

• Your test cases and their expected outputs.
• Your output contract: schema, length budget, refusal rules.
• Your measurement: pass rate, latency, retry cost.

Three things do not survive:

• Prompt strings tuned to one model's quirks.
• Provider SDK imports scattered across the repo.
• Telemetry you can only read in one vendor's dashboard.

The artifact: a swap test you can run today

This is a single-file harness. It speaks the OpenAI-compatible chat shape. Point it at any endpoint that speaks the same shape. No SDK, no framework, no lock-in.

This is a runnable sketch, not a mature tool. Treat the checkers as stubs you replace with your own rules.

# swap_test.py — same cases, any OpenAI-compatible endpoint.
import json, os, sys, time, urllib.request

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]
MODEL = os.environ["LLM_MODEL"]

def call(prompt, temperature=0.0):
    payload = json.dumps({
        "model": MODEL,
        "temperature": temperature,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=payload,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=90) as r:
        body = json.loads(r.read())
    return body["choices"][0]["message"]["content"], round(time.time() - t0, 2)

# --- checkers: replace these three with your real rules ---
def exact_keys(out, keys):
    try:
        return set(json.loads(out)) == set(keys)
    except Exception:
        return False

def says_unknown(out):
    return "unknown" in out.lower() or "null" in out.lower()

def within_budget(out, max_lines):
    return len([l for l in out.splitlines() if l.strip()]) <= max_lines

CASES = [
    {"id": "schema", "prompt": "Return only JSON with keys id, owner, due.",
     "check": lambda o: exact_keys(o, ["id", "owner", "due"])},
    {"id": "no_guess", "prompt": "Ticket T-19 has no owner field. Report the owner or say unknown.",
     "check": says_unknown},
    {"id": "diff_budget", "prompt": "Rename getUser to fetchUser in 2 lines of diff. No commentary.",
     "check": lambda o: within_budget(o, 4)},
]

results = []
for c in CASES:
    out, secs = call(c["prompt"])
    ok = False
    try:
        ok = bool(c["check"](out))
    except Exception:
        ok = False
    results.append({"id": c["id"], "pass": ok, "secs": secs,
                    "chars": len(out), "output": out[:400]})

report = {"model": MODEL, "base": BASE, "results": results}
json.dump(report, sys.stdout, indent=2)

Run it once per candidate. Keep the JSON.

LLM_BASE_URL=https://your-endpoint/v1 \
LLM_API_KEY=... \
LLM_MODEL=your-candidate \
  python swap_test.py > run-a.json

Then diff two runs by case id, not by prose.

python - <<'PY'
import json
for name in ("run-a.json", "run-b.json"):
    r = json.load(open(name))
    row = {x["id"]: x["pass"] for x in r["results"]}
    print(name, row)
PY

That diff is your decision document. Prose reviews are not.

The failure signal you actually want

A useful candidate fails in boring ways. Watch for these three shapes:

1. Confident guessing. It fills a missing field instead of reporting unknown. This is the failure that reaches production.
2. Diff inflation. It passes correctness but writes 40 lines where you budgeted 4. Your review time pays that bill.
3. Silent narrowing. It drops a schema key you never asserted. Assert the key set, not the vibes.

Pass rate alone hides all three. That is why every case carries its own checker.

Where the free tier fits, and where it does not

MonkeyCode offers free model access and a free server option. Both are supplied by the project, not by me.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

My honest read: two uses are defensible. Three are not.

• Good: the free model access is a candidate pool for swap runs. Cheap candidates mean you test more of them.
• Good: the free server option is a disposable runner. Boot it, run the harness, throw the machine away.
• Good: both together lower the cost of a first, honest evaluation.
• Bad: production state on a free box. Treat it as ephemeral by default.
• Bad: customer data on any free tier before you read the terms.
• Bad: assuming quotas, limits, or availability are permanent. Check the project's current docs, not this post.

I am not claiming the allowance is unlimited or the server is production-grade. I am claiming the cost of the first evaluation should not be the thing that stops you.

Decision table

Signal from your run	Reading	Move
All cases pass, twice in a row	Stable enough to trial	Canary on low-risk tasks
Fails only the ambiguous case	Contract is underspecified	Fix your prompt, not the model
Passes but exceeds diff budget	Review tax incoming	Keep the current model
Passes at temperature 0 only	Fragile	Add a sampling run before trusting
Latency tail exceeds your wait budget	Workflow problem	Queue it, or drop it

Set the thresholds before you run. Post-hoc thresholds are negotiation, not measurement.

Who should not do this

• Solo developers with one model and one workflow. The harness costs more than the swap saves.
• Teams handling regulated data. Free tiers are the wrong container. Full stop.
• Anyone who needs an SLA. Free access is a convenience, not a contract.
• Teams that will not maintain the cases. A stale harness lies politely.

If you are in the first group, skip the code and keep the decision table.

The thirty-day version

1. Week one: write five cases with hard checkers. No prose judging.
2. Week two: run your current model as the baseline. Save the JSON.
3. Week three: run two cheaper candidates. Compare case by case.
4. Week four: add a weekly canary on the winner. Alert on any flipped case.

That canary is the real deliverable. It tells you when a model changes under you, quietly, on a Tuesday.

Close

Shopping for models feels like progress. Building the swap test is progress.

Do the boring thing. Write the checkers, keep the JSON, and let the diff decide.

If you want to try the workflow without touching a cloud bill, start with the free model access and run the harness on the free server option. Then throw the box away, because that was the point.