Exit Criteria for a Free Model Tier

A free model tier is a lease with an unpublished end date. Signing it is not the mistake. Learning the terms during a production incident is.

The useful artifact before adoption is not a benchmark. It is an exit ticket: a short, executable set of conditions that says this tier has stopped fitting my workload, written while things still look fine, with the numbers filled in by the tier itself. Everything below is that artifact — the harness, the decision table, and the red flags that justify skipping the experiment entirely.

Averages hide the actual failure mode

Free endpoints rarely degrade along a smooth curve. They fail by category: the response envelope changes, output stops conforming to your schema, server errors replace client errors, a quota wall appears without a reset header, a long input comes back truncated.

None of that surfaces in a p50 latency chart. That is why the usual "is it fast enough" review reads green right up to the day it isn't. Probe categories instead. Every one of them is observable from outside with a plain HTTP client and no vendor SDK.

The harness

The script below uses only the standard library, because the moment a probe needs a vendor SDK it also needs a maintenance budget and a version pin. It assumes an OpenAI-compatible POST /chat/completions; confirm the path, auth header, and payload against your provider's documentation before trusting a single number it prints. Treat it as a template rather than a validated tool.

# probe.py - stdlib only. Run it from the host you plan to call the model from.
import json, os, time, urllib.error, urllib.request
from datetime import datetime, timezone

BASE  = os.environ["FREE_TIER_BASE_URL"].rstrip("/")
KEY   = os.environ["FREE_TIER_API_KEY"]
MODEL = os.environ["FREE_TIER_MODEL"]

CASES = [
    {"id": "json-shape", "kind": "json", "keys": ["ok", "n"],
     "prompt": 'Reply with JSON only: {"ok": true, "n": 7}'},
    {"id": "long-input", "kind": "text", "min_len": 1200,
     "prompt": "Repeat the word 'token' 400 times, comma separated."},
    {"id": "empty-input", "kind": "any",
     "prompt": "Summarise this text: ''"},
]

def call(prompt, timeout=60):
    body = json.dumps({"model": MODEL,
                       "messages": [{"role": "user", "content": prompt}]}).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions", data=body, method="POST",
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"})
    t0 = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            return {"status": r.status,
                    "body": r.read().decode("utf-8", "replace"),
                    "ms": int((time.monotonic() - t0) * 1000), "err": None,
                    "rl": {k.lower(): v for k, v in r.headers.items()
                           if k.lower().startswith("x-ratelimit")}}
    except urllib.error.HTTPError as e:
        return {"status": e.code,
                "body": e.read().decode("utf-8", "replace"),
                "ms": int((time.monotonic() - t0) * 1000),
                "err": f"http_{e.code}", "rl": {}}
    except Exception as e:                      # timeouts, DNS, TLS
        return {"status": None, "body": str(e),
                "ms": int((time.monotonic() - t0) * 1000),
                "err": type(e).__name__, "rl": {}}

def score(case, resp):
    if resp["err"]:
        return {"id": case["id"], "conform": 0, "reason": resp["err"]}
    try:
        text = json.loads(resp["body"])["choices"][0]["message"]["content"]
    except Exception:
        return {"id": case["id"], "conform": 0, "reason": "envelope_mismatch"}
    if case["kind"] == "json":
        try:
            ok = all(k in json.loads(text) for k in case["keys"])
        except Exception:
            ok = False
        return {"id": case["id"], "conform": int(ok),
                "reason": None if ok else "bad_json"}
    if case["kind"] == "text":
        ok = len(text) >= case["min_len"]
        return {"id": case["id"], "conform": int(ok),
                "reason": None if ok else "truncated_or_short"}
    return {"id": case["id"], "conform": 1, "reason": None}

for _ in range(5):                              # five passes, worst case counts
    run = {"ts": datetime.now(timezone.utc).isoformat(timespec="seconds"),
           "model": MODEL, "rows": []}
    for case in CASES:
        resp = call(case["prompt"])
        run["rows"].append({**score(case, resp), "status": resp["status"],
                            "ms": resp["ms"], "rl": resp["rl"]})
    with open("probe.jsonl", "a") as f:
        f.write(json.dumps(run) + "\n")
    print(run["rows"])

Five passes per case, append-only JSONL, one row per case per pass. The value is not the absolute conformance score; it is the diff between this run and the run after the next upstream change, which nobody will announce. Header capture matters more than latency here — an x-ratelimit-remaining that moves is a budget you can plan against, and its absence is itself a finding.

Turning rows into a verdict

The second file is where the exit ticket actually lives. The thresholds are policy you choose, not measurements anyone published; pick numbers that match your own tolerance and write them down where a reviewer can argue with them.

# verdict.py - reads probe.jsonl, exits 0 (keep), 1 (warn), 2 (exit)
import json, sys

STAY, LEAVE = 0.95, 0.80          # policy, not measured fact

runs = [json.loads(l) for l in open("probe.jsonl")]
if len(runs) < 2:
    sys.exit(0)

cur, prev = runs[-1], runs[-2]
conf  = lambda r: sum(x["conform"] for x in r["rows"]) / len(r["rows"])
fivex = lambda r: sum(1 for x in r["rows"] if (x["status"] or 0) >= 500)

print(f"conform cur={conf(cur):.2f} prev={conf(prev):.2f} 5xx={fivex(cur)}")
if conf(cur) < LEAVE or fivex(cur) >= 2:
    sys.exit(2)
if conf(cur) < STAY and conf(prev) < STAY:
    sys.exit(1)
sys.exit(0)

Exit code 2 is the ticket firing, not a suggestion. Wire it to whatever your team actually watches, otherwise the script becomes a file nobody reads and the whole exercise collapses into documentation theatre.

The decision table

Signal	Keep	Open the exit ticket	First move
Output shape across repeated identical prompts	one stable envelope	two shapes in five prompts	pin parsing to a schema, log failures
Error class mix over 100 calls	4xx only when your request is wrong	any 5xx cluster, or a 429 with no reset header	record retry budget, cap it in code
Fixture conformance between adjacent runs	flat	drop larger than your team's tolerance	snapshot last-good responses before retrying
Time to a first useful token	inside your UX budget	outside it three runs in a row	move the call off the request path
Cost of one extra retry	rounding error	free tier plus your retries outweighs a paid call	stop retrying, price the paid path

The last row is the one teams skip. Retries are free only if someone else pays for them; when the free tier is slow, your own compute absorbs the difference, and that is a real invoice even when the model line item reads zero.

Where MonkeyCode fits, and where it does not

MonkeyCode participates in this loop in two deliberately unexciting places. It advertises free model access and a free server option, and it is described as an open-source project; a free allowance on the order of ten million tokens has been mentioned in outreach material. That combination makes it a reasonable subject for the probe and a reasonable host for running the probe, since the measurement host stops being a budget line of its own. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Verify the allowance and the server terms against the project's own pages before planning any capacity around them. Quotas move, free endpoints get re-rated, and a number quoted in a post is not a number you can page someone about. The honest use of a free tier is as a candidate under measurement, not as a line item you have already spent.

Red flags that end the experiment early

A credential on the free host that can spend money elsewhere is the first one. If a deploy key on a free box can push to main or mint a paid API key, the saved cost is an illusion and the blast radius has quietly grown; rotate the key or move the host.

Second, the free endpoint must never be the only implementation of an interface in your codebase. One adapter module, one function, one configuration seam — if swapping the backend touches more than a single file, the tier owns you and the exit ticket cannot fire on schedule.

Third, scheduled work with no owner. A cron job on a host nobody is on call for will fail silently for a week and then fail loudly on the day it matters. Fourth, data you cannot re-derive: fixtures, prompt logs, and eval rows should be reproducible from the repository, not resident on the free disk.

Who should not use this approach

Teams with retention or residency obligations that a free host cannot contractually meet should not start. Neither should anyone whose product promises a pinned model version, a hard p95, or deterministic output, because free tiers re-rate and re-route without notice to you. And a team with nobody to own the probe should skip straight to a paid tier; an unowned exit ticket decays into folklore within a quarter.

An exit ticket is cheap to write and embarrassing to write late. Keep the JSONL, run the harness for a week, and the decision stops being a debate about vibes — the rows make the argument for you, and they travel with you to whatever tier comes next.