I Treated the Free Server as a Confound.

I will not publish a pass rate from a free server until the log can tell a wrong answer from a tired plan. That is the conclusion. A clean percentage that mixes those two is not an evaluation. It is weather with a legend.

You know the costume, because I keep wearing it. I point a tiny coding harness at whatever model is cheap enough for a Tuesday, a number twitches, and I start narrating as if the model twitched it. Did the function miss the assertion, or did the box sit there, cough, and hand back half a diff? Same red cell. Different animal. Would you fail a unit test because the laptop slept? I hope not. I have done the eval version of that, and the chart looked rigorous while it was just the room temperature of a shared machine.

Demos keep dragging the agent closer to the browser, the desktop, the repo. Cute. The score still depends on which box answered. A free box answers like a shared kitchen, not like a lab bench you booked. If you do not write that down, you will explain a capacity hiccup as a reasoning failure, and you will "fix" a prompt that was never the leak.

So I started treating "free" as an environment variable. Not a personality. Not a quality tier with a halo. An environment, the way CI=true is an environment. It changes timing, error strings, and how often the door is locked. Score the plan and the model in one bucket, and you can no longer say which one moved.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The brief I am willing to repeat is narrow. There is a free model access path, and there is a free server option. I am not naming a model, printing a token quota, or promising that either offer still looks like a screenshot from last month. Allowances move, and a blog is a bad cache. If a number matters to your budget, read the project docs the morning you run this. I would rather omit a figure than ship a stale one.

Why touch a free path during evaluation at all? Harness bugs are cheaper when a clumsy call does not invoice you. I want a dummy that will sometimes be slow, sometimes be full, and sometimes return JSON my parser did not rehearse. A free model is a decent dummy for "can my scorer survive a weird completion." A free server is a decent dummy for "can my client survive a shared machine." Neither dummy is a leaderboard. The minute you want a number other people might quote, you have left the dummy, and you owe them a denominator.

What follows is a protocol, and I am marking it unexecuted against any live host. I have not filled this scorecard with a private run and then sanded it into confidence. The artifact you can reproduce today is the classifier and its tests. They run offline, on synthetic rows, on your laptop, with no key. The live hook is a sketch. Its payload is a placeholder you replace after you read the current API, because I will not invent a request shape and call it integration.

Four tasks. The fixture is small on purpose. One is a pure function with a hidden assertion, so a wrong return has somewhere to land. One demands a unified diff and rejects a paragraph of advice, because prose is how a model dodges a spec. One is fatter than the context you think you have, so truncation can show its face. One is a no-op probe, a ping, so a cold start is not blamed on your clever prompt. This is not an IQ test. It is a stethoscope. If you need a hundred tasks before you trust the plumbing, you are decorating a leak.

Every call becomes one row. Latency. Status. Whether the body parsed. Whether the assertion passed. And a kind, the column that ruins a pretty bar chart, which is why people drop it. Quota, timeout, cold, capacity, truncation, task, auth. Those words are not synonyms, even though they all feel like "the model failed" at 1 a.m. Only task may enter the pass rate. Everything else blocked the run. A blocked run is data about the environment. It is not a vote on the model.

I picked the cutoffs. They are knobs, not laws of physics. No HTTP status, or a non-200 that already burned twenty seconds, is a timeout. HTTP 200, eight seconds or more, and a body I cannot parse is filed as cold, so I remember to look at first-byte time later instead of cursing the spec. Anything else unparsed, on a fast 200, is truncation. 429 or quota language is the plan talking. 500 and the 502 family are capacity. 401 and 403 abort the meaning of the run, because an auth miss is not a model opinion. Keyword sniffing is brittle. You will edit it the first time a real error string laughs at you. That edit is the method, not a footnote you skip.

# tier_split.py — offline protocol. Thresholds are proposals, not measurements.
import json

TIMEOUT_S = 20.0
COLD_S = 8.0

def classify(status, body, latency_s, parsed_ok):
    low = (body or "").lower()
    if status in (401, 403):
        return "auth"
    if status == 429 or "quota" in low or "rate limit" in low:
        return "quota"
    if status in (500, 502, 503, 504) or "overloaded" in low or "at capacity" in low:
        return "capacity"
    if status == 0 or (status != 200 and latency_s >= TIMEOUT_S):
        return "timeout"
    if status != 200:
        return "capacity"
    if (not parsed_ok) and latency_s >= COLD_S:
        return "cold"
    if not parsed_ok:
        return "truncation"
    return "task"

def row(task_id, status, body, latency_s, parsed_ok, assertion_ok):
    kind = classify(status, body, latency_s, parsed_ok)
    scorable = kind == "task"
    return {
        "task_id": task_id,
        "latency_s": round(float(latency_s), 3),
        "status": status,
        "kind": kind,
        "scorable": scorable,
        "passed": bool(assertion_ok) if scorable else None,
    }

def summarize(rows):
    scorable = [r for r in rows if r["scorable"]]
    blocked = [r for r in rows if not r["scorable"]]
    hits = sum(1 for r in scorable if r["passed"])
    return {
        "n": len(rows),
        "scorable": len(scorable),
        "blocked": len(blocked),
        "task_pass_rate": (hits / len(scorable)) if scorable else None,
        "publishable": bool(scorable) and not blocked and len(scorable) >= 4,
        "blocked_kinds": sorted({r["kind"] for r in blocked}),
    }

def test_quota_is_not_a_model_miss():
    r = row("diff_only", 429, "quota exceeded", 0.4, False, False)
    assert r["kind"] == "quota" and r["passed"] is None and not r["scorable"]

def test_assertion_miss_stays_on_the_model():
    r = row("assert_add", 200, "{}", 2.1, True, False)
    assert r["kind"] == "task" and r["passed"] is False

def test_cold_is_not_a_fast_truncation():
    slow = row("probe", 200, "", 9.5, False, False)
    fast = row("long_ctx", 200, "truncated", 3.0, False, False)
    assert slow["kind"] == "cold" and fast["kind"] == "truncation"

def test_mixed_run_is_not_publishable():
    rows = [
        row("probe", 200, "{}", 0.3, True, True),
        row("assert_add", 200, "{}", 1.1, True, False),
        row("diff_only", 429, "quota exceeded", 0.2, False, False),
        row("long_ctx", 200, "truncated", 3.0, False, False),
    ]
    summary = summarize(rows)
    assert summary["publishable"] is False
    assert summary["blocked_kinds"] == ["quota", "truncation"]
    # 1 of 2 scorable rows, not 1 of 4 calls. The denominator is the whole trick.
    assert summary["task_pass_rate"] == 0.5

if __name__ == "__main__":
    demo = [
        row("probe", 200, "{}", 0.3, True, True),
        row("assert_add", 200, "{}", 1.1, True, False),
        row("diff_only", 429, "quota exceeded", 0.2, False, False),
        row("long_ctx", 200, "truncated", 3.0, False, False),
    ]
    print(json.dumps(summarize(demo), indent=2))

Run the synthetic path before you even think about a host. The tests are the contract. A comment rots. A failing assert nags you the next time you are tempted to average the reds.

python tier_split.py
python -m pytest tier_split.py -q

The script prints a synthetic summary, not a product measurement. I am pasting it so the denominator is visible, and so nobody can pretend the fixture secretly timed a server.

{
  "n": 4,
  "scorable": 2,
  "blocked": 2,
  "task_pass_rate": 0.5,
  "publishable": false,
  "blocked_kinds": ["quota", "truncation"]
}

Look at that 0.5 and tell me what it means. It does not mean "the model scored fifty percent." It means one of the two calls the plan did not interrupt held its assertion. The other two rows never entered the denominator. Fold them in and you can manufacture 25 percent, or 50, or a story about truncation being a logic bug. Which story did you mean? If your dashboard still demands a single rate, it wants theater. I have published theater. The publishable flag is me trying to stop. Four scorable rows, zero blocked, or the run stays in the lab notebook.

Where does a free server earn its keep, if I refuse to invent a timing table for it? It earns its keep when the thing under test is your client. Retries that should be bounded. Parsers that should fail closed. Logs that should keep the status code instead of a smile. It breaks the moment the thing under test is the model under a stable load. Shared capacity wobbles. A cold start impersonates stupidity. A quota wall impersonates a refusal. Hold the model id constant and swap only the server, and you are still not holding the environment constant, because a free server is a crowd, not a box with your name on the door.

Who should walk away from this approach. If you are choosing a model for production codegen, this is a preflight, not the decision. If the repository is private, do not paste it into a server you have not reviewed just because the invoice is zero. Free is not a privacy policy. If you have a latency SLO, a sundial will not time your marathon, and a contended free box is that sundial. If you are about to post a leaderboard, do not. Not from four tasks, and not from an unstratified rate. And if you needed me to name a winner between hosts, this piece will disappoint you on purpose.

A live call, if you insist, stays outside the tests. I am not embedding a vendor path, and I am not claiming the JSON below is accepted by anyone. Set the URL from the docs you read today. Then edit the body until it matches. Until that edit exists, you do not have a client. You have a wish.

def live_call(url, model, prompt, timeout_s=30):
    import time, urllib.request
    payload = json.dumps({"model": model, "prompt": prompt}).encode()
    req = urllib.request.Request(
        url, data=payload, headers={"Content-Type": "application/json"}
    )
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout_s) as resp:
            raw = resp.read().decode("utf-8", "replace")
            return resp.status, raw, time.perf_counter() - t0
    except Exception as exc:
        return 0, str(exc), time.perf_counter() - t0

export EVAL_BASE_URL="https://example.invalid/v1/complete"
export EVAL_MODEL_ID="read-this-from-current-docs"
# Wire live_call yourself after the payload matches. Do not point pytest at the network.

Map the return straight into row. Parse first, then classify, then summarize. If kind is auth, stop the batch. More calls will not become more science. If kind is quota or capacity, you learned something about the plan's afternoon, and you learned almost nothing about the diff task. Write that sentence in the log. Future you, skimming at speed, will otherwise promote the red cell into a model opinion.

Limitations, before anyone quotes me. The fixture cannot see whether a model can hold a real repository in its head. Token accounting stays out until a response actually carries usage fields, and I will not invent those fields to look thorough. The keyword list will miss a new error string and misfile a row, which is why raw bodies belong next to the kind. Free access can shrink or vanish without this article updating. A green pytest run means the classifier obeyed the synthetic rows. It does not mean a host agreed with me. I am not reporting a measured win, or a measured break, of any free tier here, because I did not run that measurement for this piece. If I had, you would get the raw log, the date, the model id, and the docs URL. Not a vibe. Not a crown.

There is still a Monday use. Point the same client at a free model access path and a free server option, collect kinds instead of a single rate, and decide whether the harness is even ready for a paid, pinned run. That is the job. Shake the pipes. Do not grade the water.

If your harness already writes a row per call, point it at MonkeyCode's free model access and free server option after you confirm both still exist in the docs, and keep the blocked kinds. I would rather read those than another bare pass rate.