Switching to a Free Model? Run the Migration Harness First

A free model looks like a pure win. No invoice, no procurement, no approval. Then a required field disappears from the JSON. Your parser silently writes None into the database. The dashboard stays green. The data quality decays. Nobody notices until the weekly report looks wrong.

What does a free model actually cost when it drops a field for 2% of your traffic? That failure is not about model quality. It is about unverified equivalence. A free model can be excellent on happy paths and still break your contract on edge cases. The fix is not more manual testing. The fix is a migration harness that checks contract, quality, and cost before you route real traffic.

This article gives you that harness. It is a single Python script, under 150 lines, that compares your current endpoint against a candidate free model on your own test set. Run it once. Read the decision table. Then you will know whether the switch is safe.

Why a free model is a dependency

You already treat your database as a dependency. You version its schema. You test migrations. You monitor its latency. A model endpoint deserves the same treatment, because it has a schema too — the shape of its output.

The difference is that a database fails loudly. A model fails quietly. It returns valid JSON with a missing field. It uses a slightly different date format. It adds a key your serializer ignores. Every one of those failures is a contract violation, and none of them raise an exception.

That is why the harness checks the contract first. Quality matters, but a model that produces beautiful prose with the wrong shape is worse than a model that produces ugly prose with the right shape.

The migration harness

The script takes a JSONL test set, calls both endpoints, and compares three signals: contract pass rate, output similarity, and latency. It is deliberately small so you can read it, modify it, and trust it.

#!/usr/bin/env python3
"""migration_harness.py — prove a candidate model before you switch."""

import argparse
import difflib
import json
import statistics
import time
import urllib.request


def call_endpoint(url, api_key, payload, timeout=30.0):
    headers = {"Content-Type": "application/json",
               "Authorization": f"Bearer {api_key}"}
    req = urllib.request.Request(url, data=json.dumps(payload).encode(),
                                 headers=headers)
    start = time.perf_counter()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        body = json.loads(resp.read().decode())
    latency_ms = (time.perf_counter() - start) * 1000
    return body, latency_ms


def extract_text(response, field="content"):
    try:
        return response["choices"][0]["message"][field]
    except (KeyError, IndexError, TypeError):
        return response.get(field, "")


def contract_problems(text, required, forbidden):
    problems = []
    for field in required:
        if field not in text:
            problems.append(f"missing required field: {field}")
    for field in forbidden:
        if field in text:
            problems.append(f"forbidden field present: {field}")
    return problems


def similarity(a, b):
    return difflib.SequenceMatcher(None, a, b).ratio()


def summarize(results):
    n = len(results)
    contract_ok = sum(1 for r in results if not r["problems"])
    exact = sum(1 for r in results if r["actual"].strip() == r["expected"].strip())
    sims = [similarity(r["actual"], r["expected"]) for r in results]
    lats = sorted(r["latency_ms"] for r in results)
    p95 = lats[min(len(lats) - 1, int(len(lats) * 0.95))]
    return {
        "contract_pass_rate": contract_ok / n,
        "exact_match_rate": exact / n,
        "mean_similarity": statistics.mean(sims),
        "p95_latency_ms": p95,
    }


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--test-set", required=True)
    parser.add_argument("--current-url", required=True)
    parser.add_argument("--candidate-url", required=True)
    parser.add_argument("--current-key", default="")
    parser.add_argument("--candidate-key", default="")
    parser.add_argument("--required-fields", nargs="*", default=[])
    parser.add_argument("--forbidden-fields", nargs="*", default=[])
    parser.add_argument("--min-pass-rate", type=float, default=0.95)
    args = parser.parse_args()

    rows = [json.loads(line) for line in open(args.test_set) if line.strip()]
    current, candidate = [], []

    for row in rows:
        payload = {"messages": [{"role": "user", "content": row["input"]}]}
        cur_body, cur_ms = call_endpoint(args.current_url, args.current_key, payload)
        cand_body, cand_ms = call_endpoint(args.candidate_url, args.candidate_key, payload)
        cur_text = extract_text(cur_body)
        cand_text = extract_text(cand_body)
        current.append({
            "expected": row["expected"],
            "actual": cur_text,
            "latency_ms": cur_ms,
            "problems": contract_problems(cur_text, args.required_fields, args.forbidden_fields),
        })
        candidate.append({
            "expected": row["expected"],
            "actual": cand_text,
            "latency_ms": cand_ms,
            "problems": contract_problems(cand_text, args.required_fields, args.forbidden_fields),
        })

    cur_stats = summarize(current)
    cand_stats = summarize(candidate)

    print(f"{'metric':<22}{'current':>14}{'candidate':>14}")
    for key in cur_stats:
        print(f"{key:<22}{cur_stats[key]:>14.3f}{cand_stats[key]:>14.3f}")

    decision = "SWITCH" if (
        cand_stats["contract_pass_rate"] >= args.min_pass_rate
        and cand_stats["mean_similarity"] >= cur_stats["mean_similarity"] - 0.02
    ) else "STAY"
    print(f"decision: {decision}")


if __name__ == "__main__":
    main()

Run it like this:

python migration_harness.py \
  --test-set extraction_tasks.jsonl \
  --current-url https://api.current-provider.example/v1/chat/completions \
  --candidate-url https://your-free-endpoint.example/v1/chat/completions \
  --required-fields name address city \
  --forbidden-fields internal_id

Each line in the test set has two fields: input and expected. Thirty rows is enough for a first pass. Fifty is better. Use real production inputs, not hand-written examples, because real inputs contain the nulls and edge cases that break contracts.

How to read the output

The harness prints a table and one word: SWITCH or STAY. Do not trust the word. Trust the rows.

Signal	Threshold	Action
Contract pass rate	below 95%	STAY — shape bugs are silent killers
Mean similarity	more than 2% below current	STAY or hybrid
Exact match rate	drops sharply	Investigate before switching
p95 latency	above your budget	Keep paid for interactive paths

A SWITCH means the candidate passed the bar on your data. It does not mean the candidate is better. It means the risk is now measurable, which is the whole point.

Pointing the harness at free models

Free models are the usual suspects for this migration. The invoice is zero, so the finance approval disappears. But zero cost does not remove the contract risk. It just moves it.

MonkeyCode's open-source project offers free models, a 10-million-token allowance, and a free server option, which makes it a practical candidate URL for this harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script does not care which provider you test. Point it at any OpenAI-compatible endpoint and let the decision table speak.

The free server matters for a specific reason. You can run the harness without standing up your own infrastructure. That removes the setup excuse. If the numbers say STAY, you lost ten minutes. If they say SWITCH, you saved a monthly invoice.

A five-step workflow

1. Export 30 to 50 real inputs from your logs, with expected outputs written by a human.
2. Run the harness against your current endpoint to establish a baseline.
3. Point the candidate URL at the free model endpoint and run it again.
4. Read the decision table. If the contract pass rate is below 95%, fix your prompt or stay.
5. Re-run after every model update. Free models change without a changelog.

That last step is the one everyone skips. A model that passed last month can fail today. The harness is cheap enough to run on a schedule.

Limitations and who should skip this

The harness checks shape, not meaning. A model can pass every contract check and still produce confident nonsense. Add a human review loop for the first week after switching.

The test set is a snapshot. If your input distribution drifts, the results decay. Refresh the set monthly.

Free shared servers are not for regulated data. Do not send secrets, PII, or anything under a data-residency mandate to a free endpoint, no matter what the harness says.

Skip this workflow entirely if you need a guaranteed SLA, sub-second latency on every call, or private fine-tuning. A free model is a trial environment, not a production contract.

The switch is a measurement

A free model is a dependency with a different invoice. Test it like one. Run the harness, read the table, and let the numbers decide.

If you run this against a free endpoint, share the decision table. The interesting part is not whether it passes. It is which check fails first.