Free Models Belong in Your CI, Not Your Demo

Most teams use cheap models backwards. They wire them into a chat demo and shrug at the quality. I think the cheapest model you can call should gate every commit instead.

It should not be the thing you show customers. It should be the thing that tells you something moved.

Here is the argument, plus a runner you can copy today.

The real bottleneck is not model quality

You will never learn agent engineering from ten careful calls. You learn it from thousands of disposable runs.

Paid inference punishes that habit. Every run costs money, so you write fewer tests. You trim the suite and stop probing edge cases.

That is the failure mode. Your agent gets fragile because your test budget got tight.

Cheap inference removes the excuse. Not because cheap models are smart, but because they let you run the boring suite every single time.

The front page keeps arguing about whether agents are real. That debate misses the operational question. The question is which tier should run which check.

Treat the cheap model as a canary tier

Three tiers, three jobs

1. Canary tier: the cheapest model you can reach. Runs the full regression suite on every push.
2. Workhorse tier: a mid-cost model. Runs only cases the canary failed or never solved.
3. Review tier: an expensive model. Runs a small, hand-picked adversarial set and anything you plan to merge.

The canary does not judge product quality. It detects change cheaply.

A case flipping from pass to fail is a signal, even on a weak model. A stable pass on a weak model is weak evidence, so label it that way.

A minimal escalation runner

The code below is unexecuted here. Treat it as a sketch to adapt, not a benchmark.

# tier_runner.py — sketch, not a benchmark
import json, os, time, urllib.request, uuid

ENDPOINT = os.environ["MC_BASE_URL"]      # operator-supplied
API_KEY  = os.environ["MC_API_KEY"]
CANARY   = os.environ["MC_MODEL_CANARY"]
WORKER   = os.environ["MC_MODEL_WORKER"]
LOG      = "runs.jsonl"

def call(model, prompt, timeout=60):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(ENDPOINT, data=body, headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {API_KEY}",
    })
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=timeout) as r:
        payload = json.loads(r.read())
    return payload, round(time.time() - t0, 2)

The loop stays boring on purpose.

def run_case(case, model):
    row = {"run_id": str(uuid.uuid4()), "case": case["id"], "model": model}
    try:
        payload, secs = call(model, case["prompt"])
        row.update(ok=check(payload, case),
                   seconds=secs,
                   usage=payload.get("usage", {}))
    except Exception as exc:
        row.update(ok=False, error=type(exc).__name__)
    with open(LOG, "a") as fh:
        fh.write(json.dumps(row) + "\n")
    return row["ok"]

def main(cases):
    escalated = [c for c in cases if not run_case(c, CANARY)]
    for case in escalated:
        run_case(case, WORKER)
    print(f"{len(escalated)}/{len(cases)} escalated")

Two rules that keep this honest

1. Never compare pass rates across tiers and call it a quality score. You are measuring change, not intelligence.
2. Never let the canary gate a release. It gates a re-run.

Run it off your laptop

This is where server stability matters more than model choice. A canary tier only helps if it runs in the same place every time.

Three properties you want:

• Same image, same env vars, same seed data on every run.
• Logs written to a volume you can diff between runs.
• A wall-clock budget, so one stuck call cannot eat the night.

Commit the runner and the log schema. Then git diff on runs.jsonl tells you when behavior moved.

I run this pattern on MonkeyCode's free server option, using its free model access as the canary tier. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The project states a free token allowance for that model access. Check the current project page for the exact figure, because quotas and model lists change. Do not paste a number into your README and forget it.

Nothing above depends on that product. Any box with a pinned image works. The hard requirement is that the box is not your daily driver.

What the canary tier actually catches

Cheap models are decent detectors for a specific class of breakage.

• Prompt or template drift after a refactor.
• Tool schema changes that break argument parsing.
• Output regressions, like dropped JSON fields.
• Timeouts and retry storms under load.
• Latency shifts inside your own wrapper code.

It also produces noise. Weak models fail tasks for reasons unrelated to your change. Budget for that before you trust the log.

Decision table

Question	Canary tier	Workhorse tier	Review tier
Runs on every push	Yes	Only after canary failure	No
Gates a merge	No	No	Yes
Needs a pinned image	Yes	Yes	Yes
Logs diffed across runs	Yes	Yes	Yes
Good for cost tracking	Yes	Yes	Yes
Good for quality claims	No	Rarely	Sometimes

Read the last row twice. A pass on a cheap model is not evidence of correctness. It is evidence that nothing obvious broke.

Who should skip this

Do not use this pattern if any of these are true for you.

• You need reproducible quality claims for a customer or an auditor.
• Your task needs strict tool-calling fidelity or long multi-step planning.
• You cannot pin a model version, so the canary itself changes under you.
• Your data cannot leave your own infrastructure.

For those cases, run a fixed, versioned model on hardware you control. A free tier is the wrong tool for an audit trail, and pretending otherwise will burn you in review.

Start with one suite, one box

Pick your ten most annoying agent bugs. Turn each one into a case file. Run them on the cheapest model you can call, on a machine that is not your laptop.

If the suite is cheap, you will actually run it. If you run it, you will see the break on the commit that caused it. That is the whole argument.

If you would rather not provision anything yet, MonkeyCode's free model access and free server option are enough to stand up a canary tier and run the experiment.