DEV Community: Jordan Liu

I Made the Oracle Fail First

Jordan Liu — Tue, 13 Oct 2026 01:29:26 +0000

A fluent patch is not a pass. A free server that happens to boot is not a pass either. If I cannot make a local oracle fail on a bug I planted myself, I do not get to blame the model, and I do not get to praise it.

That is the conclusion. The rest is a harness I want in the repo before a free token or a free minute gets spent. This draft does not report a live run, because I did not execute one for it.

You know the scene. The assistant nods, the diff looks tidy, and the slice is still wrong. Who gets the lecture, the model or the missing test?

Blame is a warm drink. It does not fail a build. I want the check to fail first, on purpose, with a command I can rerun when I am bored and suspicious. No leaderboard, and no borrowed authority. Just an exit code that was already red before any model was allowed to speak.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project, and two availability claims here are operator-supplied rather than measurements from 13 October 2026: free model access, and a free server option.

I was also told the free model side includes a grant on the order of 10 million tokens. Confirm that figure, which models it actually covers, and the server terms on the current project page before you plan a week around them. I will not invent model names, hardware, durations, or a win rate.

The call site still has to live in git, next to the plants, or the evaluation is a chat you cannot replay. Free access is a budget for that replay. It is not evidence the patch was right. Delete the product name and the stop-rule still applies to any client you already have.

People keep saying the models got sharper while the bugs stayed put. Maybe they did. Sharpness without a red oracle is a brighter flashlight in a room you never filled with smoke. I am not grading a summer of announcements. I am asking whether my own gate can fail a bug I already understand.

Plant something boring

Clever bugs flatter the person who planted them. Boring bugs audit the gate. I keep one function that returns one element too many, the kind a tired reviewer can spot, because that is the point.

If the oracle cannot see a bug a tired human can see, why would I trust it on the bugs I miss at 1 a.m.? The plant stays small so the failure is about the gate, not about my talent for hiding a defect in a maze.

# harness/plants.py
# Proposal. Not a recorded run.
from dataclasses import dataclass

@dataclass(frozen=True)
class Plant:
    name: str
    buggy: str
    fixed: str

TAKE = Plant(
    name='take-off-by-one',
    buggy='''def take(xs, n):
    return xs[: n + 1]
''',
    fixed='''def take(xs, n):
    return xs[:n]
''',
)

The test is equally dull. Dull is a feature, not a lack of ambition. It names one boundary and refuses to negotiate.

# tests/test_take.py
from take import take

def test_take_does_not_overshoot():
    assert take(['a', 'b', 'c'], 2) == ['a', 'b']

A writer script keeps me from pasting the bug by hand and then forgetting which variant is on disk. Hands are how plants get quietly fixed before the model ever sees them.

# harness/write_plant.py
# Proposal. Overwrites take.py from the plant table.
import argparse
from plants import TAKE

BODIES = {'buggy': TAKE.buggy, 'fixed': TAKE.fixed}

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument('--name', required=True)
    parser.add_argument('--variant', choices=sorted(BODIES), required=True)
    args = parser.parse_args()
    if args.name != TAKE.name:
        raise SystemExit('unknown plant')
    with open('take.py', 'w', encoding='utf-8') as fh:
        fh.write(BODIES[args.variant])

if __name__ == '__main__':
    main()

Write the buggy body into place and let pytest speak before you do. If you feel an urge to peek at a completion first, that urge is the experiment trying to cheat.

python harness/write_plant.py --name take-off-by-one --variant buggy
python -m pytest -q tests/test_take.py
echo oracle_exit=$?

If that exit is 0, stop. The plant is invisible. Asking a free model to review an invisible bug is a play, not a measurement. Fix the test, and do not touch the prompt.

A smoke alarm you have never held to smoke can still beep in a drill. The red pytest is the smoke. Without it I am admiring the plastic cover and calling it safety.

Only then may the model look

The prompt stays poorer than the test file. I do not paste the expected list. I do not ask whether the function is good, because that adjective is how a yes gets hired as an engineer.

The instruction I send is plain. One function is attached. A test named test_take_does_not_overshoot fails. Return either a line that starts with REJECT and one sentence, or a unified diff that changes only take. Do not claim the tests were run. Save the reply next to the plant name. Feeling is not a column.

# harness/classify.py
# Proposal. A malformed reply is a miss, not a cue to retry.
def classify(reply: str) -> str:
    text = reply.strip()
    if text.startswith('REJECT:'):
        return 'reject'
    if text.startswith('---') or text.startswith('diff '):
        return 'diff'
    return 'malformed'

Then the mechanical half, still on the laptop, still too boring to argue with:

python harness/extract_diff.py reply.txt > candidate.diff
patch -p0 < candidate.diff
python -m pytest -q tests/test_take.py
echo after_model_exit=$?

Two exits. They do not get to shake hands. The first had to be non-zero or the day is void. The second may be zero, and that is a different sentence.

A REJECT on a function the oracle already hates can be a real hit. The model refused to launder a bad slice. A minimal diff that goes green is a different hit. Would you average those and call the average quality? I will not. Unlike events do not become one fact just because they share a spreadsheet.

If patch refuses the diff, I do not retry inside the scorer. A diff that does not apply is a miss. Retry-until-apply is another study, and it will eat a grant while wearing a lab coat.

The ledger is the third gate

Free tokens disappear in the rephrase. You paste the traceback, you ask again, and you call it iteration. The meter calls it lunch.

This ledger is local, and it should stay proud of that. If the provider meter disagrees, the provider wins. Write the gap down. Do not litigate it in the README and pretend the argument is a measurement.

# harness/ledger.py
# Proposal. Counts what this process tried to spend.
class Ledger:
    def __init__(self, ceiling: int):
        self.ceiling = ceiling
        self.spent = 0

    def charge(self, prompt_tokens: int, completion_tokens: int) -> None:
        self.spent += prompt_tokens + completion_tokens
        if self.spent > self.ceiling:
            raise SystemExit(
                f'stop: spent {self.spent} > ceiling {self.ceiling}'
            )

Set the ceiling to a slice, not to the folklore of the whole grant. A few thousand tokens will tell you whether the loop is bounded. Ten million, if the project page still says that when you look, is headroom. It is not a result I measured, and it is not a promise I can extend.

Quotas move. A method that only works if a published number never changes is not a method. Re-read the page the day you run the card, and write the number you actually saw into the notebook next to the date.

python harness/run_eval.py --plants harness/plants --ceiling 4000 --dry-run

The dry-run flag prints the prompt and charges nothing. I want it the way I want a seatbelt in a parking lot. An evaluation you cannot rehearse will get rehearsed on the grant, by accident, late, when you are sure this one extra call does not count.

The free server witnesses. It does not judge.

I may use the free server only after the local oracle is already red, and only to ask a narrower question. Does the same pytest exit match on that box? That is a witness statement about the room. It does not grade taste, tone, or how confident the completion sounded.

Laptop red and server green means portability, not genius. Both red means the patch missed, or the plant is still in the tree. Both green before the model spoke means the day is invalid, so go back and make smoke.

I am not asking a remote box whether a model is smart. Smart is not a command. Pytest is a command. If the documented runner is different from the placeholder below, keep the exit code and throw my hostname away.

# Placeholder only. Use the documented free-server run command.
# example.test is not a product host.
ssh runner@example.test -- python -m pytest -q tests/test_take.py
echo witness_exit=$?

Keep the exit code. Ignore any story a boot splash tries to sell. A machine that started is a machine that started. That sentence feels obvious until a green banner lands in a screenshot and everybody in the thread relaxes.

Read the card like gauges

When the card comes back I read it the way a mechanic reads gauges, not the way a fan reads a scoreline. Oracle not red? The card is void. No argument, no partial credit, no interesting qualitative notes.

Oracle red and the reply malformed? A miss, and the ledger still charged, which is the lesson hiding in the annoyance. Oracle red, diff applies, pytest green, witness matches? That is the only quiet pass I trust, and only for this plant.

Oracle red, and the model calls the buggy function fine? False praise. That cell matters more to me than a pretty repair. A tool that blesses a known-bad slice will bless the next one when I am tired and late. Where does a free model tend to look strong on a pack like this? On the boring repair, once the oracle has already named the failure. Where does it break? On false praise, on diffs that do not apply, and on any day the local check was blind. The free server breaks the story when its exit disagrees with the laptop, and it proves nothing when you boot it first and call the boot a result.

Four counts sit side by side, and they never get blended. Oracle-fail rate on plants, which must be total or you throw the day out. Repair rate on plants the oracle actually caught. False-praise rate. Witness mismatches between the laptop and the free server.

I am leaving those cells empty on purpose. No timings. No win rate. No claim that one stack beat another. You fill them on your plants, and you keep the raw exits next to the commit. An empty card is stricter than a handy number I could have typed.

Who should walk away

Skip this if you need a procurement bake-off. A toy slice will flatter whoever has seen an off-by-one in training, and it will teach you almost nothing about a 400-file service. Skip it for auth, crypto, or any hazard a unit test does not name.

A green take() will not make a token check safe. Skip it if the free tier forbids scripted calls. A ledger is not a pardon, and I am not suggesting you route around a limit. Skip it if you wanted a leaderboard by Friday. You wanted a mood, and moods do not belong in the README.

Two limits sit in the design, and they will not politely stay small. Leakage is the first. Paste the expected list into the prompt and a model can patch the example instead of the function. Keep the prompt hungrier than the test file.

Scale is the second. One plant is a fire drill, not a map of the repo. Restage bugs you have already fixed, oracle included, names stripped so you are not grading memory of your own commit message. A bug you cannot restage is an anecdote with better lighting. Treat it as a story. Do not enter it on the card.

If you point this same stop-rule at MonkeyCode's free model, and only afterward at the free server, read the current docs first. Confirm the token grant and the server limits yourself. Run the dry flag before you spend anything. I would rather you distrust the grant than trust this paragraph.

I Split the Score. Reach Was Never a Pass.

Jordan Liu — Sun, 11 Oct 2026 21:29:16 +0000

A free call that comes back with HTTP 200 is not a model that can do the job. I keep repeating that because a single pass rate keeps crawling back into the notes, wearing a lab coat. Reach, contract, and check are three different arguments. Glue them into one boolean and you will praise a host for being awake, or blame a model for a timeout it never saw.

You know the Tuesday demo. Same prompt, Wednesday, and the body is empty, or half an object, or a charming paragraph that fails the checker you trusted last month. Which piece actually broke? If you stored one bit, you cannot say. You graded a smoothie and called it fruit.

This is a protocol, not a scoreboard from a run I am about to invent. No latency trophy. No win rate. Those numbers belong to a log you produce yourself. What I will hand you is the split, a harness, and a rule for which column is allowed to speak.

The blend is the bug

Picture a clinic that publishes one word, healthy. Blood pressure, whether the patient showed up, and whether the vial was labeled all collapse into that word. A missed appointment looks like a bad heart. A mislabeled vial looks like a no-show. You would fire that clinic. Why do we publish model evals that way?

Free model access makes the blend worse, not kinder. A free server is a second moving part. It can refuse, queue, reset, or idle-timeout while the weights sit there innocent. A free model can also be perfectly reachable and still emit prose where your tool wanted an object. Score only the final checker and both failures wear the same red mark. Then the postmortem becomes a vibe.

I am not stating a quota, a model list, a box size, or a promise that any free tier stays free next week. Those facts rot on contact. If a vendor page is your source, read it the day you run, and paste that URL into the run record. Do not borrow a sentence from a blog as a spec.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator supplied two availability claims I will use as conditions, not as trophies: free model access, and a free server option. I was not given a verified catalog, a token ceiling, or a hardware note, so this draft does not invent them. MonkeyCode is one place those two conditions might be obtained. The protocol does not depend on that product. Swap the transport and the lesson still stands.

Freeze the exam before you touch the wire

I want one lever moving. The prompt set is a directory of files. Hash each file, then hash the sorted list of those digests. That second digest is the exam id. If a prompt changes after the first request, you are holding a different exam, and the comparison is theater. Throw the run out. Yes, that is fussy. It is also the only way a later diff means anything.

The checker is frozen too. A second function, local, pure, no network. It reads the parsed output and the fixture, then returns pass or fail plus a short reason. If the checker calls a model, you built a mirror. Mirrors always agree with themselves. I will not score that and call it evidence.

Conditions are a pair, not a tour of every host you can find. Condition A is the control: same exam id, a transport you already trust, even if that transport is a process on your laptop. Condition B is the free-model call landing on the free server. Same items. Same checker. Same timeout, written in milliseconds before the first request leaves. Edit the timeout after you see the red rows, and you edited the question.

I do not retry inside this score. A retry is a different policy, with its own article and its own lies. Score the first completed attempt. Log the transport error beside it so a later retry study can reuse the raw row without contaminating this one.

Three columns, and the order is the point

Reach asks a rude question. Did the transport return a body you are allowed to grade? DNS failure, connection reset, non-200, empty string, cutoff by the timeout you pre-registered: those are reach failures. They are not model failures. Write them in the reach column and stop. Do not feed a timeout string into the checker and call the item wrong. That move is how a flaky path becomes a fake intelligence regression.

Contract asks a picky question. Given a body, does it match the shape you demanded? I use a JSON schema on tool-shaped tasks. Missing keys, wrong types, a paragraph wrapped around an object, a brace cut off at the end: contract failures. The model can sound sharp and still fail this. Your agent will not care that the prose was warm.

Check asks the only question a user actually meant. The frozen checker says the content is acceptable. A schema-valid lie fails here. A schema-invalid truth fails earlier, in contract, and never reaches check. That ordering is the whole method. You stop operating on the wrong organ.

A run is comparable only when both conditions share the exam id, the checker digest, the timeout, and the item ids. Miss one of those and you have an anecdote. Anecdotes are fine at lunch. They are not a release note.

A harness you can run, with no fake log under it

The code below is a proposal. I did not execute it for this draft, and I will not paste imaginary rates under it. Fill the transport with a client you really have. Point the free side at a free server only after you have confirmed, from a current primary page, that the option is still offered.

mkdir -p eval/items eval/runs
python3 -m pip install --user jsonschema
# one JSON fixture per item: prompt, schema, fixture
python3 split_score.py --items eval/items --out eval/runs/$(date -u +%Y%m%dT%H%M%SZ)

#!/usr/bin/env python3
"""Proposal harness. Unexecuted here. Ships with no sample metrics."""

import argparse, hashlib, json, time
from pathlib import Path

TIMEOUT_MS = 20000  # pre-register; do not edit after red rows appear

def digest(blob: bytes) -> str:
    return hashlib.sha256(blob).hexdigest()

def load_items(folder: Path):
    items = []
    for path in sorted(folder.glob("*.json")):
        raw = path.read_bytes()
        item = json.loads(raw)
        item["_id"] = path.stem
        item["_digest"] = digest(raw)
        items.append(item)
    joined = "\n".join(i["_digest"] for i in items).encode()
    return items, digest(joined)

def reach_call(transport, item, timeout_ms):
    started = time.perf_counter()
    try:
        body, status = transport.complete(item["prompt"], timeout_ms / 1000)
    except Exception as exc:
        return {"ok": False, "reason": type(exc).__name__, "body": None,
                "status": None, "ms": int((time.perf_counter() - started) * 1000)}
    ok = status == 200 and isinstance(body, str) and body.strip() != ""
    return {"ok": ok, "reason": "ok" if ok else "empty_or_status", "body": body,
            "status": status, "ms": int((time.perf_counter() - started) * 1000)}

def contract_ok(body, schema):
    import jsonschema
    try:
        parsed = json.loads(body)
    except json.JSONDecodeError:
        return False, "not_json", None
    try:
        jsonschema.validate(parsed, schema)
    except jsonschema.ValidationError:
        return False, "schema", parsed
    return True, "ok", parsed

def score_condition(name, transport, items, checker):
    rows = []
    for item in items:
        reach = reach_call(transport, item, TIMEOUT_MS)
        row = {"item": item["_id"], "condition": name, "reach": reach["ok"],
               "reach_reason": reach["reason"], "ms": reach["ms"],
               "contract": None, "check": None}
        if not reach["ok"]:
            rows.append(row)
            continue
        good, why, parsed = contract_ok(reach["body"], item["schema"])
        row["contract"] = good
        row["contract_reason"] = why
        if not good:
            row["raw_head"] = (reach["body"] or "")[:240]
            rows.append(row)
            continue
        passed, why = checker(parsed, item["fixture"])  # local, pure, no model
        row["check"] = bool(passed)
        row["check_reason"] = why
        rows.append(row)
    return rows

def rates(rows):
    n = len(rows) or 1
    reached = [r for r in rows if r["reach"]]
    contracted = [r for r in reached if r["contract"]]
    checked = [r for r in contracted if r["check"]]
    return {
        "n": len(rows),
        "reach": len(reached) / n,
        "contract_given_reach": (len(contracted) / len(reached)) if reached else None,
        "check_given_contract": (len(checked) / len(contracted)) if contracted else None,
    }

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--items", type=Path, required=True)
    ap.add_argument("--out", type=Path, required=True)
    args = ap.parse_args()
    items, exam = load_items(args.items)
    raise SystemExit(
        "Inject control and free transports, then write rates() per condition. "
        f"exam={exam} items={len(items)} out={args.out}"
    )

if __name__ == "__main__":
    main()

Look at rates. It will not divide checks by every item and call that accuracy. Check is conditional on contract. Contract is conditional on reach. A free server that drops half the calls can still show a pretty check rate on the half that arrived. Publish only the pretty number and you are advertising the survivors. I would rather see a ugly reach column than a polished lie.

A fixture can be tiny. Here is the shape I want on disk, so the harness and the checker share one contract:

{
  "prompt": "Return JSON only. Pick a status for a build that exited 0.",
  "schema": {
    "type": "object",
    "required": ["status"],
    "properties": {"status": {"enum": ["green", "red"]}},
    "additionalProperties": false
  },
  "fixture": {"expect": "green"}
}

def checker(parsed, fixture):
    ok = parsed.get("status") == fixture["expect"]
    return ok, "match" if ok else "mismatch"

That checker is boring on purpose. Boring is a feature. If you cannot write a boring checker, you are not ready to split a score, and a vibe grade will flatter you faster than a free host ever could.

How to read a log you actually produced

Suppose reach falls apart on the free server while contract-given-reach stays close to the control. That is a host story, or a client story. Fix the path, the DNS, the idle timeout, the auth header. Do not prompt harder. A better sentence will not repair a reset packet. I have watched people rewrite instructions for an hour because they never logged status codes. Don't be that hour.

Suppose reach holds and contract collapses. Now you may look at the model, and at the wrapper around it. Truncation first. A tight output cap masquerades as stupidity. Then check whether a chat template stuffed a preamble in front of the object you demanded. The harness keeps raw_head for that reason. A reason code without the body is gossip.

Suppose contract holds and check collapses. Now you may talk about competence. Even then, talk about these items, this checker, this exam id. One exam is not a personality. It is not a ranking of vendors. It is a paired difference on a frozen set.

Write a minimum item count before the run. I use 30 as a floor for a smoke comparison, and I do not treat 30 as a paper. Under that floor a single weird fixture swings the rate, and you will narrate noise with a straight face. Smaller sets are allowed. Call them small in the first line. Pretending small is large is the failure mode this whole split is meant to catch.

Compare on item id, not on two averages floating in a slide. If item 14 failed only on the free path, open item 14. The mean will not tell you that the fixture held a nested list the control happened to flatten. Pair the rows. Then talk.

What I will not let this protocol claim

It will not tell you the free server is cheap enough to ship. Cost needs a bill, and I do not have yours. It will not tell you about privacy. A free host may log prompts. If a fixture contains customer text, do not send it. Synthetic items, or you have left evaluation and walked into an incident.

It will not survive a moving exam. Tweak prompts between condition A and condition B and you measured your editing. It will not fairly grade open-ended prose. The checker has to be dull. Subjective rubrics need a different design, with a judge that is not the model under test. Using the same free model to grade itself is a mirror again. Skip it.

Who should walk away? If you need an SLA, a retention promise, or a named model pinned for a regulated audit, a free option is the wrong instrument. Buy a contracted endpoint and a change window. If you only want a weekend demo, skip the harness too. A demo can be a demo. Just do not paste it into a README and call it a measurement.

Silent retries also void the reach column. If your client library retries unless you say otherwise, the row is already dirty. Turn that off, or log each attempt as its own policy. Mixing them is how bounded and unbounded behavior get the same name.

Free model access is still a gift for this kind of work, because the expensive mistake is scoring the wrong organ, not renting a larger box. A free server is useful when condition B has to be a real network path instead of a daydream. It becomes a costume the moment you let reach define quality. If you already hold a key for that pair of conditions, including through MonkeyCode after you re-read what the docs offer this week, point the transport at it and keep the three columns apart. Publish the column that moved. Your readers can survive a narrower claim. They cannot survive a blended one.

I Held One Lever Still

Jordan Liu — Sat, 10 Oct 2026 17:59:52 +0000

A free model and a free server are two levers. Move both, then quote one pass rate, and you did not review a model. You published a blend. I will not publish a blend.

You know the afternoon this happens. The endpoint answers. The box is up. The suite is already on disk, so you fire it, crop the fraction, and feel done. Which lever moved? If the honest answer is both, plus a retry you forgot to count, the fraction is a story about a pile.

Free infrastructure makes the pile feel scientific. No invoice, no guilt, and somehow no design. I do not buy that trade. A zero bill is not a baseline. It is a quiet way to change two things and then narrate one of them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Two availability claims were supplied for this draft, and they are the only product facts I will stand on: free model access, and a free server option. I am not printing a token grant, a model roster, a hardware sheet, a duration, or a license badge. None of those arrived as a primary source I can defend for 2026-10-10. A number I cannot cite does not become true because a draft would read warmer with a big integer in the first screen.

So what experiment is left, if the headline is banned?

A two-by-two, blocked on the same task list, with one lever held still. Same prompts. Same grader. Same seed rule. Same timeout. The only legal swap is the factor you wrote into the manifest before the first request left the machine. Did the salt change, or the pot? If you changed both and then swore the broth was brighter, you cooked an anecdote. Hold the pot. Change the salt. On the next batch, hold the salt and change the pot. Now a later reader has something to argue with.

Hold the server, swap the model slot. That cell may talk about free model access, and only against the server you actually held. Hold the model, swap the server slot. That cell may talk about the free server, and only against the model you actually held. Swap both and you have an interaction check, not a verdict. If the single swaps were calm and the pair fell over, you learned they misbehave as a couple. You did not earn "the model failed." You also did not earn "the server failed."

The baseline cell is the ruler. It can be a recorded fixture plus a local stub, as long as you can hold that pair still next week. A ruler that moves is not a ruler. It is a second treatment pretending to be the wall. I would rather compare against a boring fixture than against whatever the router felt like serving at lunch.

Blocking is the part people skip because it looks fussy. It is not fussy. If cell A sees the tasks in one order and cell B sees a shuffle, order became a third lever. If one cell retries and the other does not, retry policy became a fourth. If the seed is "whatever the process clock felt like," you cannot tell a flaky item from a factor. Derive the seed from the task id plus one run salt, write that salt into the manifest, and refuse to resample until the rate looks friendly. Friendliness is not a stopping rule. It is a wish.

What does a free path break, even when you are trying to be good? A label that is really a router. A server that sleeps, so the first tasks measure wakeup and the rest measure the work. A client that retries while the manifest still says one attempt. An SDK that returns a friendly alias, so you never see which level you hit. If the factor name is an alias, the cell is unlabeled. An unlabeled cell is a missing cell. Would you sign a lab book that said "the blue one" and meant three different blues? I would not.

I am not going to invent the quota that punches a hole in the run. I am going to treat an unfinished cell as unfinished. Survivors do not get promoted to "the run." That promotion is how a partial afternoon becomes a chart with a calm typeface. Missing cells do not get filled with the mean of their neighbors. Neighbors are different treatments. Averaging across them is how a blend learns to wear a single name.

Read four symbolic outcomes, not as data I collected, but as the only sentences the grid licenses. Suppose model-only moves and server-only does not. You may talk about the model slot. Suppose server-only moves and model-only does not. You may talk about the server slot. Suppose neither single swap moves, and the pair does. You may talk about interaction, then stop. Suppose every cell moves the same way. You probably changed the harness, the tasks, or the grader, and the levers are innocent until you prove otherwise. None of those sentences need a fake percentage. The shape is the finding. A percentage without the shape is decoration.

The artifact is a protocol, not a lab log. I have not executed it against a live free tier for this article. The client body is a stub on purpose. If you run it after you replace the stub, the JSON manifest is the result. If you do not run it, you do not get to borrow certainty from the shape of the code. A green fraction from an unimplemented client would be a magic trick. I am not collecting those.

#!/usr/bin/env python3
"""Two-by-two eval manifest. Reference protocol, not a scored bake-off."""
from __future__ import annotations

import hashlib
import json
import os
import time
import uuid
from dataclasses import asdict, dataclass, field
from typing import Callable, Optional

@dataclass
class Factor:
    name: str
    level: str
    endpoint: str

@dataclass
class Trial:
    task_id: str
    seed: int
    model_level: str
    server_level: str
    ok: Optional[bool] = None
    error_class: Optional[str] = None
    latency_ms: Optional[int] = None

@dataclass
class Manifest:
    run_id: str
    harness_sha256: str
    task_file_sha256: str
    seed_salt: int
    started_unix: float
    cells: dict = field(default_factory=dict)
    finished_unix: Optional[float] = None
    admissible: bool = False
    refusal_reason: Optional[str] = None

def sha256_file(path: str) -> str:
    return hashlib.sha256(open(path, "rb").read()).hexdigest()

def classify(exc: BaseException) -> str:
    if isinstance(exc, NotImplementedError):
        return "unimplemented"
    name = type(exc).__name__
    if "Timeout" in name:
        return "timeout"
    if "Connection" in name or "HTTP" in name:
        return "transport"
    return "grader_or_task"

def run_cell(tasks, model: Factor, server: Factor, salt: int, call: Callable) -> list[Trial]:
    out = []
    for task_id in tasks:
        seed = int(hashlib.sha256(f"{salt}:{task_id}".encode()).hexdigest()[:8], 16)
        trial = Trial(task_id, seed, model.level, server.level)
        t0 = time.perf_counter()
        try:
            call(task_id, model, server, seed)
            trial.ok = True
        except Exception as exc:
            trial.ok = False
            trial.error_class = classify(exc)
        trial.latency_ms = int((time.perf_counter() - t0) * 1000)
        out.append(trial)
    return out

def judge(manifest: Manifest) -> tuple[bool, str]:
    need = {"baseline", "model_only", "server_only", "both_swapped"}
    have = set(manifest.cells)
    if have != need:
        return False, "missing_cells:" + ",".join(sorted(need - have))
    for name, trials in manifest.cells.items():
        if not trials:
            return False, f"empty:{name}"
        if any(t["error_class"] == "unimplemented" for t in trials):
            return False, "stub_call_is_not_evidence"
        if any(t["ok"] is None for t in trials):
            return False, f"unscored:{name}"
    return True, "grid_complete"

def rates_if_admissible(manifest: Manifest) -> Optional[dict]:
    ok, _reason = judge(manifest)
    if not ok:
        return None
    report = {}
    for name, trials in manifest.cells.items():
        graded = [t for t in trials if t["error_class"] is None]
        report[name] = {
            "n": len(trials),
            "graded_n": len(graded),
            "pass_rate": (sum(bool(t["ok"]) for t in graded) / len(graded)) if graded else None,
            "transport_n": sum(t["error_class"] == "transport" for t in trials),
            "timeout_n": sum(t["error_class"] == "timeout" for t in trials),
        }
    return report

def main() -> None:
    model_base = Factor("model", "baseline-slot", os.environ["MODEL_BASE_URL"])
    model_free = Factor("model", "free-model-slot", os.environ["MODEL_FREE_URL"])
    server_base = Factor("server", "baseline-slot", os.environ["SERVER_BASE_URL"])
    server_free = Factor("server", "free-server-slot", os.environ["SERVER_FREE_URL"])
    task_path = os.environ["TASK_FILE"]
    tasks = json.load(open(task_path))
    salt = int(os.environ.get("EVAL_SEED", "17"))

    def call(task_id: str, model: Factor, server: Factor, seed: int) -> None:
        raise NotImplementedError(
            f"wire your client: {task_id} {model.level} @ {server.level} seed={seed}"
        )

    plan = {
        "baseline": (model_base, server_base),
        "model_only": (model_free, server_base),
        "server_only": (model_base, server_free),
        "both_swapped": (model_free, server_free),
    }
    manifest = Manifest(
        str(uuid.uuid4()), sha256_file(__file__), sha256_file(task_path), salt, time.time()
    )
    for name, (model, server) in plan.items():
        manifest.cells[name] = [asdict(t) for t in run_cell(tasks, model, server, salt, call)]
    ok, reason = judge(manifest)
    manifest.admissible = ok
    manifest.refusal_reason = None if ok else reason
    manifest.finished_unix = time.time()
    json.dump(asdict(manifest), open("run_manifest.json", "w"), indent=2)
    if not ok:
        raise SystemExit(f"refusing headline metric: {reason}")

if __name__ == "__main__":
    main()

Pin the harness and the task file before you trust the run you think you started. A dirty tree is another lever, and it never introduces itself.

sha256sum eval_grid.py tasks.json | tee inputs.sha256
export MODEL_BASE_URL MODEL_FREE_URL SERVER_BASE_URL SERVER_FREE_URL
export TASK_FILE=tasks.json EVAL_SEED=17
python3 eval_grid.py
python3 -c 'import json; m=json.load(open("run_manifest.json")); print(m["admissible"], m.get("refusal_reason"))'

A stub run should exit on stub_call_is_not_evidence. That refusal is the correct first result. Do not catch it and tweet the partial JSON. The partial JSON is a note that the client was never wired.

When a real client is in place, debug the manifest before you debug the model. Open run_manifest.json and compare transport_n across cells. If transport piles up only in server_only and both_swapped, your sentence is about the server path, or about the path to that server. It is not about the model. If timeouts pile up only on the free model slot, say timeout, then decide whether your product even cares. If missing_cells fires, you do not have a result. You have a plan you did not finish. Imputing the missing corner from the other three is how people invent an interaction they never observed.

What do you commit? The harness, the task-file hash, the seed salt, and the manifest. Not a screenshot of a fraction. A screenshot cannot tell me whether the both-swapped cell was allowed to speak. The JSON can. If admissible is false, the writeup is the refusal reason, and that writeup is allowed to be short. Short and refused beats long and blended.

Who should walk past this? You, if the demo is at lunch and the room wants a vibe. A grid is a bad vibe. You, if the question is whether a UI feels kind, because kindness is not a factor level in this file. You, if every request is locked to one blended endpoint and you cannot hold a baseline still. Then you have one pipe. A pipe can still be worth using for a prototype. It cannot carry a sentence that starts with "the model," and it cannot carry one that starts with "the server." Say "this pipe" and stop. That sentence is weaker, and it is the one you actually earned.

Limits, with the romance removed. There is no scoreboard here, because I did not run the live pair, and I will not invent one to decorate a method. Grader quality sits outside the grid. A stable wrong grader produces an admissible, repeatable, useless manifest. Environment variables are not a docs site. Free access can move next week, so last week's JSON does not license next week's slogan. Interaction can be real and still be too small to matter for your task. The protocol will not choose your practical threshold. Write that threshold down before you look, or do not pretend the look was a test.

Trying the pipe is fine. Publishing the try as a model review is the part I will not cosign. If the free model slot and the free server slot in your env file are the ones named in the disclosure, they are eligible factors in this grid. They are not pre-cleared results. Run the four cells once, keep the refusal or the JSON, and let that file outlive the badge.

I Froze the Hash. The Free Run Had to Earn the Sign.

Jordan Liu — Fri, 09 Oct 2026 14:26:59 +0000

A free runner is not a result. A stopwatch is not a race, and a pass rate is not a decision until the sample, the scorer, and the stop rule are already boring. If any of those can still wriggle, a free model and a free server only make the wriggling cheaper.

Freeze the hash. Pre-register the break. Let a sign appear only when both halves agree.

You know the itch. A stack shows up, the calls cost nothing you can feel, a server is offered so you skip provisioning, and lunch starts to look like a leaderboard. Did you ever nudge one prompt, rerun, and then describe the last number as if it had been the hypothesis?

If that scene feels familiar, it is the habit this card is built to interrupt. Not a trophy. A failure mode with a friendly face.

The feed keeps congratulating small models for going quiet on a private checker. Quiet is not stable. A checker can go silent because you edited the items until it did. Have you ever watched a green cell calm a room that never asked whether the other half of the file would have stayed green?

Seasoning and judging are different jobs. Taste the soup to change the recipe, and you are cooking. Taste it to award a ribbon, and you are peeking. Free access is a gift to the cook and a trap for the judge, because another taste barely stings.

The chart gets patient. You do not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. For this method I am treating two availability claims as given, and only those: free model access, and a free server option. I am not stating a token quota, a model roster, a machine size, a duration, or a score from a run.

I do not have a primary source I am willing to pin next to a task hash, and I will not fill the gap with a figure that might already be stale. If it is not in the card, it is not a result. A number that cannot sit in the same commit as the tasks is just weather.

What are those surfaces doing in the method? They are a pair of runners, not a mascot. Treatment A is free model access: your process sends a frozen item and scores the completion locally. Treatment B is the free server: the same item goes to a hosted runner you do not administer, and you score the trace that comes back with the same function.

Same items, same rubric, different place the work happened. If you cannot name which surface tripped a rule you wrote yesterday, you do not have an evaluation. You have a demo with extra steps.

I have not executed this harness against that product for this article. The blocks below are a proposal with a synthetic fixture so you can see the refusal path on your laptop. Thresholds are choices you lock before the first network call. They are not measurements, and they are not a review of anyone's uptime.

The unit is the item, not the afternoon. You export tasks to JSONL, one object per line, each with an id and an expected string. You hash the file bytes, then split with a seeded digest so half 0 and half 1 stay put on rerun. You score both treatments on both halves.

A sign is earned only when each half clears the item floor, each arm stays under the transport-error ceiling, the absolute gap clears a floor you wrote down so a one-item wobble cannot wear a crown, and the two halves agree on the direction. A hash mismatch is an invalidation even if the new file is basically the same. Editing the rubric after a sad cell is an invalidation too. Git will remember, even if your recap does not.

Why not one mean? A single average can sand a flip off the table until the room feels calm. I would not trust that sanding in a write-up. Would you change a routing file because forty items leaned left and forty leaned right?

I would call that a break and leave the config alone. Agreement is a low bar. It is still a bar.

Performs well cannot mean a percentage I did not measure. Here it means the run can finish under rules another person can replay: both surfaces return scoreable traces, invalidation stays quiet, and the sign matches across halves. Breaks has to stay a specific sentence, not a mood.

A timeout is not a foolish model. A rubric that never looks at tool traces is not a broken server. A moved hash is not a product defect. It is you, still cooking.

Fold those into one red badge and the free tier takes the blame for your notebook. Three sentences and zero winners is a better log. Can you say which sentence fired? If you cannot, you are not done.

Who should skip this? Anyone who needs a flattering percentage before stand-up, because the card is built to withhold it. Anyone holding private customer text should not ship it to a free server because the meter feels kind. Anyone who needs a medical, legal, or safety judgment should not pretend a frozen string check is that judgment.

If you cannot commit the task file, the card, and the scorer in one revision, you cannot audit the claim later. Do not make the claim.

Limitations, with the soft landing removed. Half-agreement is not proof that one surface is wiser, cheaper in a way that lasts, or still offered next month. It is a stability check on a small decision. A hosted runner can insert retries, clocks, and tool hops your local call never saw, and this protocol notices only if the scorer reads those fields on purpose.

String equality is a blunt rubric. It will call a harmless paraphrase a miss, and it will miss a fluent wrong answer that happens to match. A pilot that survives the card may justify a larger pre-registered run. It does not justify a testimonial, and I am not writing one.

The workflow is short enough to do before coffee gets cold, and strict enough to ruin a dishonest chart. Write the card in the same file as the scorer, hash the JSONL, and commit both plus the hash before any call. Implement two clients that take an item and return text, and keep secrets out of the repo. Leave a dry implementation that raises, so a curious teammate cannot just see what happens against the network by running the module.

Replace the dry clients only after that commit exists. Run the module and read earned. If it is false, the result is the breaks list. Sit with it. That list is the finding.

Save this as split_half_sign.py. It is a proposal. --fixture never leaves the process, and the default clients raise so a real file cannot accidentally become a leaderboard.

python3 split_half_sign.py --fixture
sha256sum tasks.jsonl
python3 split_half_sign.py tasks.jsonl "$(sha256sum tasks.jsonl | cut -d' ' -f1)"

That second command, left as written, should fail the transport rule. Good. That is the dry default doing its job. A green earned before you replace dry_complete would mean the harness is lying, and you should delete it.

#!/usr/bin/env python3
# Proposal, not a product benchmark.
# Lock CARD before any network call. Thresholds are design choices.
# --fixture is synthetic and opens no socket.
from __future__ import annotations

import hashlib
import json
import sys
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Callable

CARD = {
    "protocol": "split-half-sign-v1",
    "min_items_per_half": 30,
    "min_scored_fraction": 0.95,
    "max_transport_error_rate": 0.05,
    "min_abs_gap": 0.02,
    "seed": "freeze-me-before-running",
}

CompleteFn = Callable[[dict], str]


def file_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def load_jsonl(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        if line.strip():
            rows.append(json.loads(line))
    return rows


def half_of(item_id: str, seed: str) -> int:
    digest = hashlib.sha256(f"{seed}:{item_id}".encode()).hexdigest()
    return int(digest[-1], 16) % 2


def rubric(task: dict, completion: str) -> bool:
    expected = str(task.get("expected", ""))
    return completion.strip() == expected.strip()


@dataclass
class ArmReport:
    name: str
    n: int
    scored: int
    transport_errors: int
    pass_rate: float | None
    invalidated: str | None


def run_arm(name: str, items: list[dict], complete: CompleteFn) -> ArmReport:
    errors = scored = hits = 0
    for task in items:
        try:
            text = complete(task)
        except RuntimeError:
            errors += 1
            continue
        scored += 1
        hits += int(rubric(task, text))
    n = len(items)
    err_rate = (errors / n) if n else 1.0
    frac = (scored / n) if n else 0.0
    invalidated = None
    if n < CARD["min_items_per_half"]:
        invalidated = "below_min_items"
    elif err_rate > CARD["max_transport_error_rate"]:
        invalidated = "transport"
    elif frac < CARD["min_scored_fraction"]:
        invalidated = "unscored_mass"
    rate = (hits / scored) if scored else None
    return ArmReport(name, n, scored, errors, rate, invalidated)


def sign_of(left: float | None, right: float | None) -> int:
    if left is None or right is None:
        return 0
    gap = left - right
    if abs(gap) < CARD["min_abs_gap"]:
        return 0
    return 1 if gap > 0 else -1


def evaluate(rows, arms, got_hash, expected_hash):
    breaks = []
    if expected_hash is not None and got_hash != expected_hash:
        return {"earned": False, "breaks": ["hash_mismatch"], "hash": got_hash, "sign": 0}
    halves = {0: [], 1: []}
    for row in rows:
        halves[half_of(str(row["id"]), CARD["seed"])].append(row)
    half_signs = []
    reports = []
    for h, subset in halves.items():
        rates = {}
        for name, fn in arms.items():
            report = run_arm(f"{name}.h{h}", subset, fn)
            reports.append(asdict(report))
            if report.invalidated:
                breaks.append(f"{name}.h{h}:{report.invalidated}")
            rates[name] = report.pass_rate
        half_signs.append(sign_of(rates.get("free_model"), rates.get("free_server")))
    if not breaks and (not half_signs or 0 in half_signs):
        breaks.append("gap_below_floor_or_unscored")
    if not breaks and half_signs[0] != half_signs[1]:
        breaks.append("halves_disagree")
    earned = not breaks
    return {
        "earned": earned,
        "sign": half_signs[0] if earned else 0,
        "half_signs": half_signs,
        "breaks": breaks,
        "hash": got_hash,
        "reports": reports,
        "note": "+1 means free_model ahead on pass rate; -1 means free_server ahead",
    }


def dry_complete(_task: dict) -> str:
    raise RuntimeError("dry run: replace me after the hash is committed")


class FixtureComplete:
    # Synthetic. Pass rates flip by half. Not a vendor measurement.

    def __init__(self, name: str) -> None:
        self.name = name

    def __call__(self, task: dict) -> str:
        h = half_of(str(task["id"]), CARD["seed"])
        want_hit = (self.name == "free_model" and h == 0) or (
            self.name == "free_server" and h == 1
        )
        return str(task["expected"]) if want_hit else "MISS"


def synthetic_rows() -> list[dict]:
    buckets = {0: [], 1: []}
    i = 0
    while len(buckets[0]) < 40 or len(buckets[1]) < 40:
        item_id = f"t{i:04d}"
        h = half_of(item_id, CARD["seed"])
        if len(buckets[h]) < 40:
            buckets[h].append({"id": item_id, "expected": f"ok-{i}"})
        i += 1
    return buckets[0] + buckets[1]


def main(argv: list[str]) -> int:
    if len(argv) >= 2 and argv[1] == "--fixture":
        rows = synthetic_rows()
        blob = "\n".join(json.dumps(r, separators=(",", ":")) for r in rows).encode()
        digest = hashlib.sha256(blob).hexdigest()
        out = evaluate(
            rows,
            {
                "free_model": FixtureComplete("free_model"),
                "free_server": FixtureComplete("free_server"),
            },
            digest,
            digest,
        )
        out["fixture"] = "synthetic_refusal_not_a_product_run"
        print(json.dumps(out, indent=2))
        return 0
    if len(argv) != 3:
        print(
            "usage: split_half_sign.py --fixture | split_half_sign.py tasks.jsonl HASH",
            file=sys.stderr,
        )
        return 2
    path = Path(argv[1])
    out = evaluate(
        load_jsonl(path),
        {"free_model": dry_complete, "free_server": dry_complete},
        file_hash(path),
        argv[2],
    )
    print(json.dumps(out, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

A task file is just lines like these. Hash the bytes you will actually send, not a prettier export you make later.

{"id":"t0001","expected":"pong"}
{"id":"t0002","expected":"pong"}

When the fixture prints, look at three fields and ignore the urge to screenshot a winner. earned should be false. breaks should name the disagreement, and half_signs should be a pair that does not match. If a future real run prints earned true, the sign is a stable direction on this frozen file, under this blunt rubric, on this day.

It is still not a quota, not a hardware review, and not permission to skip the private-data rule. Puppet rates inside --fixture are not a vendor score. They exist so you can watch the harness keep its mouth shut. When you later point the same function at real clients, silence still means silence.

I will ask for one thing, then stop selling. If free model access and a free server are actually available on the MonkeyCode account you use, point this card at tasks you can stand to hash in public. Commit the hash and the stop rule before the first call.

If the sign does not earn itself, write about the break. I trust that note more than a last-run percentage.

I Scored the Flip. Best-of-Five Folded.

Jordan Liu — Thu, 08 Oct 2026 10:16:10 +0000

I will not publish best-of-five. A free code model is not usable because one sample passed. That sample might be the only one that would have passed. If you publish the highlight reel, you measured a slot machine, not a tool.

You have seen the flattering cut, haven't you? Five draws, one green check, a screenshot, a conclusion. What question did that answer? Not the question your teammate hits on a Tuesday, when nobody is sitting there rerolling until the tests blush.

The feed this week is busy with profile toys and portfolios people vibe-coded over a weekend. Fine toys. None of them tell you whether draw three still exits zero.

So the conclusion comes first, before any endpoint. Report the first draw, the stable pass, and the flip. Ignore any protocol that lets you delete the red rows. If those three numbers are not on disk, you do not have an evaluation. You have a mood.

First-draw pass is what an unattended user gets. Stable pass is the share of tasks that pass on every draw, with no runner error hiding in the set. Flip is the share that both pass and fail across draws.

Same prompt. Same check. Different verdict. That last one is the alibi killer.

A task that flips is not basically working. It is a coin, and coins do not get merge rights. Would you hand a coin your migration?

Why not a single accuracy number? Because a single number is how luck borrows a lab coat. Best-of-five can rise while stability falls. You would feel better and ship worse.

I would rather see an ugly stable-fail than a pretty flip. An ugly fail tells me to change the task class or the tool. A flip tells me I was about to quote a coincidence.

I do not have a leaderboard, and I am not going to invent one. This note is a protocol and a scorer. Any rows that look like results in the fixture are a self-check of the arithmetic. They are not a benchmark of a free model, a paid model, or a server.

If a pass rate shows up here without a CSV from the day the checks ran, treat it as fiction. I want you suspicious. Suspicion is the method.

Where do the free generator and the free runner enter? I want both boring and available, so the afternoon is about variance instead of about whether I could afford the draws. MonkeyCode is the option this harness is aimed at: free model access for the completions, and a free server option so every sample faces the same check command rather than whatever my laptop happened to have installed. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The outreach brief calls the project open source. Clone the repository you can verify. A paraphrased client is not a source.

I am also not quoting a token quota, a machine size, or a duration. Those are dashboard facts, and dashboard facts rot faster than blog posts. On the day you run, read the current allowance and the current server terms, then copy them into the run notes by hand.

If the allowance is smaller than K times your task count, shrink K before you start. Dropping tasks afterward, and still calling the file a replicate, is how samples vanish while the chart stays confident.

The harness is plain Python on purpose. No framework to debug before the model. It shells out to your check command, which is the whole point. Pretty code that never executes is a different hobby.

#!/usr/bin/env python3
"""split_score.py — first draw, stable pass, and flip.

Live mode is unexecuted until MODEL_ENDPOINT is set.
`--fixture` checks the arithmetic only. It is not a model score.
"""
from __future__ import annotations

import argparse, csv, json, os, subprocess, hashlib
from pathlib import Path

def load_tasks(root: Path) -> list[dict]:
    tasks = []
    for path in sorted(root.glob("*/task.json")):
        task = json.loads(path.read_text())
        task["dir"] = str(path.parent)
        tasks.append(task)
    if not tasks:
        raise SystemExit(f"no tasks under {root}")
    return tasks

def call_model(endpoint: str, prompt: str, draw: int) -> str:
    # Proposal only. Match the real request shape before a live run.
    # Read any key from the environment. Never hardcode one.
    import urllib.request
    body = json.dumps({
        "prompt": prompt,
        "draw": draw,
        "temperature": 0.2,
    }).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode())
    if "text" not in payload:
        raise SystemExit("response missing text; fix the client, do not score it")
    return payload["text"]

def run_check(task_dir: str, code: str) -> str:
    snippet = Path(task_dir) / "_candidate.py"
    snippet.write_text(code)
    try:
        proc = subprocess.run(
            ["python3", "check.py"],
            cwd=task_dir,
            capture_output=True,
            text=True,
            timeout=30,
        )
    except subprocess.TimeoutExpired:
        return "runner_error"
    if proc.returncode == 0:
        return "pass"
    if proc.returncode < 0:
        return "runner_error"
    err = (proc.stderr or "") + (proc.stdout or "")
    if "Traceback" in err and "_candidate" not in err:
        return "runner_error"
    return "fail"

def summarize(rows: list[dict]) -> dict:
    by_task: dict[str, list[tuple[int, str]]] = {}
    for row in rows:
        by_task.setdefault(row["task"], []).append((int(row["draw"]), row["verdict"]))
    n = len(by_task)
    if n == 0:
        raise SystemExit("no rows to score")
    first = stable = flip = runner = 0
    for pairs in by_task.values():
        verdicts = [v for _, v in sorted(pairs)]
        if any(v == "runner_error" for v in verdicts):
            runner += 1
        if verdicts[0] == "pass":
            first += 1
        if all(v == "pass" for v in verdicts):
            stable += 1
        if "pass" in verdicts and "fail" in verdicts:
            flip += 1
    return {
        "tasks": n,
        "first_pass": round(first / n, 3),
        "stable_pass": round(stable / n, 3),
        "flip_rate": round(flip / n, 3),
        "tasks_with_runner_error": runner,
    }

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--tasks", default="tasks")
    parser.add_argument("--draws", type=int, default=5)
    parser.add_argument("--fixture", action="store_true")
    parser.add_argument("--out", default="split_score.csv")
    args = parser.parse_args()

    if args.fixture:
        # Synthetic rows. They test the scorer. They do not describe a product.
        rows = [
            {"task": "parse_int", "draw": 0, "verdict": "pass"},
            {"task": "parse_int", "draw": 1, "verdict": "pass"},
            {"task": "parse_int", "draw": 2, "verdict": "fail"},
            {"task": "slug", "draw": 0, "verdict": "fail"},
            {"task": "slug", "draw": 1, "verdict": "fail"},
            {"task": "slug", "draw": 2, "verdict": "fail"},
            {"task": "clamp", "draw": 0, "verdict": "pass"},
            {"task": "clamp", "draw": 1, "verdict": "pass"},
            {"task": "clamp", "draw": 2, "verdict": "pass"},
        ]
    else:
        endpoint = os.environ["MODEL_ENDPOINT"]
        rows = []
        for task in load_tasks(Path(args.tasks)):
            for draw in range(args.draws):
                code = call_model(endpoint, task["prompt"], draw)
                verdict = run_check(task["dir"], code)
                rows.append({"task": task["id"], "draw": draw, "verdict": verdict})

    summary = summarize(rows)
    with open(args.out, "w", newline="") as fh:
        writer = csv.DictWriter(fh, fieldnames=["task", "draw", "verdict"])
        writer.writeheader()
        writer.writerows(rows)
    digest = hashlib.sha256(Path(args.out).read_bytes()).hexdigest()[:12]
    print(json.dumps({**summary, "csv_sha256_12": digest}, indent=2))

if __name__ == "__main__":
    main()

The fixture is supposed to be mean. parse_int flips. slug fails every draw. clamp stays green.

If you cannot say those three sentences without opening a pricing page, you are not ready for a live file. What should the scorer print? First-draw 0.667, stable pass 0.333, flip rate 0.333, runner errors 0. I did that division on paper before I trusted a script. You should do the same.

A scorer you have not caught is just another lucky sample, wearing a .py hat. Run the self-check before you touch a network. It should not need a key, a server, or a hope.

python3 split_score.py --fixture --out /tmp/split_fixture.csv
python3 - << 'PY'
import json, subprocess
raw = subprocess.check_output(
    ["python3", "split_score.py", "--fixture", "--out", "/tmp/split_fixture.csv"]
)
got = json.loads(raw)
expect = {
    "first_pass": 0.667,
    "stable_pass": 0.333,
    "flip_rate": 0.333,
    "tasks_with_runner_error": 0,
}
for key, val in expect.items():
    assert got[key] == val, (key, got[key], val)
print("scorer ok", got["csv_sha256_12"])
PY

A task is a directory, not a vibe. task.json holds an id and a prompt. check.py imports the candidate and exits non-zero when the behavior is wrong. Keep the check dumber than the task.

If the check is the clever part, you are grading your harness and calling it a model. That is an easy way to feel rigorous while learning nothing.

{
  "id": "clamp",
  "prompt": "Write clamp(x, lo, hi) in _candidate.py. Return x bounded to [lo, hi]. No imports. No prints."
}

# tasks/clamp/check.py — example check, not a live result
import _candidate

def expect(got, want):
    if got != want:
        raise SystemExit(f"{got!r} != {want!r}")

expect(_candidate.clamp(3, 0, 2), 2)
expect(_candidate.clamp(-1, 0, 2), 0)
expect(_candidate.clamp(1, 0, 2), 1)

The generator and the runner should not share a mood. Draws come from the free model endpoint. The live scorer runs on the free server, so check.py sees one image, one Python, one clock. Laptop runs are for the fixture. Mix them and you are grading your own machine again.

When the fixture matches, and only then, point the script at a live endpoint. The request body inside call_model is a proposal. I have not executed it against a current API, and I will not guess a path, a model id, or a header name in public.

Match the client to the docs you verified today. Keep the key in the environment. Log a hash of the prompt next to the CSV.

If the prompt changes, start a new file. Editing the old file is not replication. It is a quiet rewrite.

# Proposal, not a verified hostname or client path.
# Confirm today's free-server login from the dashboard, then run the scorer there.
export MODEL_ENDPOINT="https://example.invalid/draw"
python3 split_score.py --tasks ./tasks --draws 5 --out runs/today.csv
sha256sum runs/today.csv

Read the printout like someone who has been fooled by a green cell before. High first-draw plus high flip means the generator can look competent and still be a coin. Would you let that coin edit a migration? I would not.

Low first-draw and a stable fail is a kindness. It tells you this task class is a break, not a teaser. High stable pass and a flip near zero is the only pattern I will call performs well, and only for drafts a human still reads.

Runner errors are not model fails. They mean the check, the timeout, or the server session broke. Fix that, rerun, and do not average across the mess.

Who should walk away? If you need a number for a launch note tonight, walk away. Five draws times a dozen tasks will not fit in a coffee break, and a partial grid is not a result you get to round up.

If the code can touch credentials, payments, or auth, walk away from the free path. A green clamp says nothing about a webhook. If the check needs a private network, a production token, or customer data, do not place it on a free server either. Shared runners are for toy tasks with toy fixtures.

If your question is taste in prose, this harness will lie politely, because it only speaks exit codes. Exit codes do not have taste. They have zero and not-zero.

There is a quieter limit, and it matters. Temperature 0.2 in the proposal is a frozen confound, not an optimum I measured. Change it and you have a different study. K=5 is a budget, not a proof about sample size.

A free server can restart, throttle, or run slower than a laptop. Those events belong in the notes, not in the model column. I will not use this protocol to rank products against each other.

It can rank task classes against one generator, on one day, under one check suite. That is enough. It is also all the certainty the design earns.

If you already have a MonkeyCode login, point the draws at the free model, run the checks on the free server, and keep the CSV beside the prompt hash. If you do not, run the fixture anyway. The arithmetic does not need a brand. The brand does not get a score until the arithmetic has a file.

I Kept Three Scores. The Mean Was a Costume.

Jordan Liu — Thu, 08 Oct 2026 07:51:17 +0000

A blended pass rate is a costume. It looks like measurement. Most days it is three arguments sharing a coat, and the coat is what people quote.

I do not want that coat in this piece. You have seen the posts this month, a percentage stacked on a task list, pressed flat enough to screenshot. Twenty prompts go out, eighteen replies parse, and the chart says ninety percent. The thread then treats the chart as a personality.

Then you open the two failures. One is a refusal you wanted. The other is fluent nonsense wrapped in valid JSON. Which failure is the model, and which success is just the parser being easy to please?

So the conclusion comes first. Split the fixture by stratum, sweep seeds, and keep three scores that you are not allowed to add together. If you cannot show the worst stratum and its spread, you do not have a result. You have wardrobe.

That is the evaluation. The rest is a harness, a toy fixture whose arithmetic you can check by hand, and a rule for what performs well is allowed to mean. I did not execute this protocol against a live endpoint for this draft. I will not invent a pass rate to decorate it.

Free repeated calls are what make a seed sweep possible instead of theatrical. MonkeyCode is the open-source project this outreach is about, and the only availability claims I will repeat are free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a token quota, a box size, a region, a latency, or that either free path is permanent.

Why mention a free path in a measurement piece at all? Because an honest check is a multiplier, strata times seeds times one frozen prompt. A single call is cheaper, and it is also how a dashboard learns to lie. A free model lane plus a free server matters here only as a condition you can afford to repeat. It is not a medal, and a paid lane is not automatically the grown-up one.

Write the fixture before you pick an endpoint. I use four strata, and I mean the word. Extract, classify, format, abstain. Each row carries an id, a prompt, an expect flag, and a gold set.

Why a set rather than one blessed string? Case and extra spaces are not a moral event, and exact-match theater burns afternoons. Why an expect flag at all? A correct refusal looks like a miss if the scorer only hunts for a substring.

That bug hides inside a blended rate, so I would rather make it impossible to commit. Freeze the file in git before the first request leaves the laptop. If the gold moves mid-sweep, you are grading a draft and blaming a model. Have you ever done that and called it rigor?

Each row then runs at several seeds. Same prompt, same temperature, same token cap, if the API even exposes those knobs. One run is an anecdote. Five is my floor, and ten is the floor when the stratum is abstain, because a refusal that flips is a coin, not a personality.

Log HTTP status, elapsed milliseconds, and a short body hash beside the scores. Do not add milliseconds into the pass bit. Speed is a different article. If a free server queues or fails closed, that fact belongs in status, not in a fake intelligence miss.

Three scores, and they are not friends. Parse asks only whether the body is JSON with the keys I demanded. Match asks whether a normalized answer lands in the gold set, and it stays missing when the row expected an abstention. Abstain asks whether the model declined when it should have, and whether it answered when declining would have been a dodge.

A lovely parser can still be a reckless guesser. Add those axes into one ratio and you sew the coat again. Still tempted to add them? Put a classify hit next to an abstain miss and tell me what the average is for.

The scorer below is the artifact. The fixture rows are ones I wrote so the arithmetic is inspectable. They are not model outputs, and they are not a MonkeyCode benchmark. Trust this as a unit test for the judge, because a judge you have not tested is just another model with a clipboard.

#!/usr/bin/env python3
"""Stratum seed-sweep scorer.

Unexecuted against any live endpoint.
FIXTURE rows are author-written so the arithmetic can be checked by hand.
Population stdev is used because the fixture is the whole claimed set.
"""
from __future__ import annotations

import hashlib
import json
import statistics
from collections import defaultdict
from pathlib import Path

FIXTURE = [
    {
        "id": "e1",
        "stratum": "extract",
        "expect": "answer",
        "gold": ["ada lovelace"],
        "seeds": [
            {"seed": 1, "body": '{"answer":"Ada Lovelace"}', "status": 200, "ms": 410},
            {"seed": 2, "body": '{"answer":"Ada Lovelace"}', "status": 200, "ms": 390},
        ],
    },
    {
        "id": "c1",
        "stratum": "classify",
        "expect": "answer",
        "gold": ["bug"],
        "seeds": [
            {"seed": 1, "body": '{"answer":"bug"}', "status": 200, "ms": 220},
            {"seed": 2, "body": '{"answer":"feature"}', "status": 200, "ms": 240},
        ],
    },
    {
        "id": "a1",
        "stratum": "abstain",
        "expect": "abstain",
        "gold": [],
        "seeds": [
            {"seed": 1, "body": '{"answer":null,"abstain":true}', "status": 200, "ms": 300},
            {"seed": 2, "body": '{"answer":"42"}', "status": 200, "ms": 280},
        ],
    },
    {
        "id": "f1",
        "stratum": "format",
        "expect": "answer",
        "gold": ["3"],
        "seeds": [
            {"seed": 1, "body": "three", "status": 200, "ms": 180},
            {"seed": 2, "body": '{"answer":"3"}', "status": 200, "ms": 190},
        ],
    },
]


def norm(value: object) -> str:
    return " ".join(str(value).strip().lower().split())


def score_body(row: dict, seed_row: dict) -> dict:
    raw = seed_row["body"]
    parse_ok = False
    answer = None
    abstained = False
    parsed = None
    try:
        parsed = json.loads(raw)
        parse_ok = isinstance(parsed, dict) and (
            "answer" in parsed or "abstain" in parsed
        )
    except json.JSONDecodeError:
        parse_ok = False
    if parse_ok and parsed is not None:
        answer = parsed.get("answer")
        abstained = bool(parsed.get("abstain")) or answer is None
    gold = {norm(item) for item in row["gold"]}
    if row["expect"] == "answer":
        match = bool(parse_ok and answer is not None and norm(answer) in gold)
        abstain_ok = bool(parse_ok and not abstained)
    else:
        match = None
        abstain_ok = bool(parse_ok and abstained and answer is None)
    return {
        "id": row["id"],
        "stratum": row["stratum"],
        "seed": seed_row["seed"],
        "parse_ok": parse_ok,
        "match": match,
        "abstain_ok": abstain_ok,
        "status": seed_row["status"],
        "ms": seed_row["ms"],
        "body_sha256": hashlib.sha256(raw.encode()).hexdigest()[:12],
    }


def pack(values: list[float]) -> dict:
    if not values:
        return {"n": 0, "mean": None, "pstdev": None}
    spread = statistics.pstdev(values) if len(values) > 1 else 0.0
    return {
        "n": len(values),
        "mean": round(statistics.fmean(values), 3),
        "pstdev": round(spread, 3),
    }


def summarize(scored: list[dict]) -> dict:
    grouped: dict[str, list[dict]] = defaultdict(list)
    for item in scored:
        grouped[item["stratum"]].append(item)
    report = {}
    for stratum, rows in grouped.items():
        parses = [1.0 if row["parse_ok"] else 0.0 for row in rows]
        matches = [1.0 if row["match"] else 0.0 for row in rows if row["match"] is not None]
        abstains = [1.0 if row["abstain_ok"] else 0.0 for row in rows]
        report[stratum] = {
            "parse": pack(parses),
            "match": pack(matches),
            "abstain": pack(abstains),
        }
    return report


if __name__ == "__main__":
    scored = [score_body(row, seed) for row in FIXTURE for seed in row["seeds"]]
    Path("scores.jsonl").write_text("".join(json.dumps(item) + "\n" for item in scored))
    print(json.dumps(summarize(scored), indent=2))

Run the judge before you point it at anything with a hostname. If this arithmetic surprises you, you are not ready for a live file.

python3 stratum_sweep.py
# writes scores.jsonl
# prints per-stratum mean and population stdev
# fixture only: a test of the scorer, not of a product

Walk the rows and the means fall out. Extract match mean is 1.0 with population stdev 0. Classify parses both times and matches once, so match mean is 0.5 and population stdev is 0.5. Format parse mean is 0.5, because the word three is not JSON. Abstain match stays at n=0 on purpose, and abstain_ok mean is 0.5.

If your machine disagrees, the scorer drifted. That is a bug in the judge, not a model story. A blended pass would smear those four stories into one polite decimal, and I am not printing it. If your eye still wants the decimal, that want is the bug.

I use population standard deviation on purpose. The fixture is the whole set you claimed to score, not a sample from a hidden universe. A sample stdev on two points just inflates the costume. If you grow the fixture, recompute, and do not carry the old spread forward like a family heirloom.

What does performs well mean once a live client replaces the fixture? Not a cutoff borrowed from a thread. Format should show a high parse mean and a parse spread near zero, because format is a contract, not a vibe. Extract should show a small match spread.

A mean of 0.8 with a spread of 0.4 is a coin that landed well in the screenshot you kept. Would you ship a parser on that evidence? I would not. I would add seeds until the spread either collapsed or confessed.

Abstain should be stable on the abstain stratum, and abstain_ok should also be stable on the answer strata. A model that refuses every prompt is not careful. It is vacant. A model that never refuses is not bold. It is unscoped.

The design breaks in familiar places. Two seeds will not save you, and the toy uses two only so you can check the math by hand. A wrong gold set makes a stable model look guilty. A sloppy normalizer launders a near miss into a hit.

Dropped match cells are a different population from parse cells. Compare them as if they were siblings and the chart starts talking nonsense. Null is not zero. If you coerce it, you just invented a failure the row was never scoring.

A free server can throttle, fail closed, or shape errors differently from the other lane you use on Tuesdays. That is a condition difference. Keep it in status and in the timing field. I am not giving you a measured timeout rate, a queue depth, or a token budget, because none of those were recorded for this piece.

If your client retries, the JSONL has to count attempts, or you scored the loop. Write temperature, token cap, and a hash of the fixture into every row, even when you are sure you set them once. Config drift is how two identical sweeps become an argument. I would rather lose an afternoon to a hash mismatch than win one with a chart I cannot rebuild.

Who should walk away? Anyone who needs a compliance signature. A four-row fixture is a worked example, not an assurance case. Anyone whose harm lives in the tail should also walk, because mean and spread on twenty friendly prompts will grin while the tail burns.

Anyone who cannot freeze the fixture in git should walk too. You would be grading a moving target and blaming the model. And if you only have one call per task, do not pretend a sweep happened. Write that you took a sample of one. That sentence beats a dashboard tile.

Pointing the judge at a free path should stay boring. Read the base URL and the key from the environment. Do not hardcode a model id you have not checked against this week's docs. If the API ignores seed, record that as a limitation. Silence is not determinism.

export EVAL_BASE_URL="https://example.invalid/replace-from-current-docs"
export EVAL_API_KEY="${EVAL_API_KEY:-}"
if [ -z "$EVAL_API_KEY" ]; then
  echo "set EVAL_API_KEY in your shell, not in the repo"
  exit 1
fi
echo "key loaded, not printed"
# Local stand-in. Not a vendor schema. Not a claim that this path exists.
cat > prompt.json << 'EOF'
{"note":"local stand-in","fixture_id":"e1","seed":1,"temperature":0,"prompt":"Extract the person. Reply as JSON with an answer field."}
EOF
curl -sS "$EVAL_BASE_URL" \
  -H "Authorization: Bearer $EVAL_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @prompt.json \
  -o "raw/e1.seed1.json" \
  -D "raw/e1.seed1.headers" \
  -w "status=%{http_code} time=%{time_total}\n"

After two conditions, diff the reports, not the vibes. Change nothing except a base URL you already verified in current docs.

python3 stratum_sweep.py > report_a.json
python3 stratum_sweep.py > report_b.json
diff -u report_a.json report_b.json

If match means move when only the server changes, you learned something about the condition. If they move when you change nothing, the sweep is still too small, or the endpoint is not honoring seed. Both are results. Neither needs a superlative.

So ask it of the next single-cell chart that lands in your feed. If extract is calm and abstain flips on every seed, what did the average tell you? It told you the coat still fits. Take it off, and keep the JSONL.

If you already have a free model lane and a free server, including the pair this MonkeyCode outreach is pointing at, aim a frozen fixture at them and keep the worst stratum with its spread. I will trust that file longer than I will trust a hero percentage.

I Treated the Free Server as a Confound.

Jordan Liu — Thu, 24 Sep 2026 20:35:51 +0000

I will not publish a pass rate from a free server until the log can tell a wrong answer from a tired plan. That is the conclusion. A clean percentage that mixes those two is not an evaluation. It is weather with a legend.

You know the costume, because I keep wearing it. I point a tiny coding harness at whatever model is cheap enough for a Tuesday, a number twitches, and I start narrating as if the model twitched it. Did the function miss the assertion, or did the box sit there, cough, and hand back half a diff? Same red cell. Different animal. Would you fail a unit test because the laptop slept? I hope not. I have done the eval version of that, and the chart looked rigorous while it was just the room temperature of a shared machine.

Demos keep dragging the agent closer to the browser, the desktop, the repo. Cute. The score still depends on which box answered. A free box answers like a shared kitchen, not like a lab bench you booked. If you do not write that down, you will explain a capacity hiccup as a reasoning failure, and you will "fix" a prompt that was never the leak.

So I started treating "free" as an environment variable. Not a personality. Not a quality tier with a halo. An environment, the way CI=true is an environment. It changes timing, error strings, and how often the door is locked. Score the plan and the model in one bucket, and you can no longer say which one moved.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The brief I am willing to repeat is narrow. There is a free model access path, and there is a free server option. I am not naming a model, printing a token quota, or promising that either offer still looks like a screenshot from last month. Allowances move, and a blog is a bad cache. If a number matters to your budget, read the project docs the morning you run this. I would rather omit a figure than ship a stale one.

Why touch a free path during evaluation at all? Harness bugs are cheaper when a clumsy call does not invoice you. I want a dummy that will sometimes be slow, sometimes be full, and sometimes return JSON my parser did not rehearse. A free model is a decent dummy for "can my scorer survive a weird completion." A free server is a decent dummy for "can my client survive a shared machine." Neither dummy is a leaderboard. The minute you want a number other people might quote, you have left the dummy, and you owe them a denominator.

What follows is a protocol, and I am marking it unexecuted against any live host. I have not filled this scorecard with a private run and then sanded it into confidence. The artifact you can reproduce today is the classifier and its tests. They run offline, on synthetic rows, on your laptop, with no key. The live hook is a sketch. Its payload is a placeholder you replace after you read the current API, because I will not invent a request shape and call it integration.

Four tasks. The fixture is small on purpose. One is a pure function with a hidden assertion, so a wrong return has somewhere to land. One demands a unified diff and rejects a paragraph of advice, because prose is how a model dodges a spec. One is fatter than the context you think you have, so truncation can show its face. One is a no-op probe, a ping, so a cold start is not blamed on your clever prompt. This is not an IQ test. It is a stethoscope. If you need a hundred tasks before you trust the plumbing, you are decorating a leak.

Every call becomes one row. Latency. Status. Whether the body parsed. Whether the assertion passed. And a kind, the column that ruins a pretty bar chart, which is why people drop it. Quota, timeout, cold, capacity, truncation, task, auth. Those words are not synonyms, even though they all feel like "the model failed" at 1 a.m. Only task may enter the pass rate. Everything else blocked the run. A blocked run is data about the environment. It is not a vote on the model.

I picked the cutoffs. They are knobs, not laws of physics. No HTTP status, or a non-200 that already burned twenty seconds, is a timeout. HTTP 200, eight seconds or more, and a body I cannot parse is filed as cold, so I remember to look at first-byte time later instead of cursing the spec. Anything else unparsed, on a fast 200, is truncation. 429 or quota language is the plan talking. 500 and the 502 family are capacity. 401 and 403 abort the meaning of the run, because an auth miss is not a model opinion. Keyword sniffing is brittle. You will edit it the first time a real error string laughs at you. That edit is the method, not a footnote you skip.

# tier_split.py — offline protocol. Thresholds are proposals, not measurements.
import json

TIMEOUT_S = 20.0
COLD_S = 8.0

def classify(status, body, latency_s, parsed_ok):
    low = (body or "").lower()
    if status in (401, 403):
        return "auth"
    if status == 429 or "quota" in low or "rate limit" in low:
        return "quota"
    if status in (500, 502, 503, 504) or "overloaded" in low or "at capacity" in low:
        return "capacity"
    if status == 0 or (status != 200 and latency_s >= TIMEOUT_S):
        return "timeout"
    if status != 200:
        return "capacity"
    if (not parsed_ok) and latency_s >= COLD_S:
        return "cold"
    if not parsed_ok:
        return "truncation"
    return "task"

def row(task_id, status, body, latency_s, parsed_ok, assertion_ok):
    kind = classify(status, body, latency_s, parsed_ok)
    scorable = kind == "task"
    return {
        "task_id": task_id,
        "latency_s": round(float(latency_s), 3),
        "status": status,
        "kind": kind,
        "scorable": scorable,
        "passed": bool(assertion_ok) if scorable else None,
    }

def summarize(rows):
    scorable = [r for r in rows if r["scorable"]]
    blocked = [r for r in rows if not r["scorable"]]
    hits = sum(1 for r in scorable if r["passed"])
    return {
        "n": len(rows),
        "scorable": len(scorable),
        "blocked": len(blocked),
        "task_pass_rate": (hits / len(scorable)) if scorable else None,
        "publishable": bool(scorable) and not blocked and len(scorable) >= 4,
        "blocked_kinds": sorted({r["kind"] for r in blocked}),
    }

def test_quota_is_not_a_model_miss():
    r = row("diff_only", 429, "quota exceeded", 0.4, False, False)
    assert r["kind"] == "quota" and r["passed"] is None and not r["scorable"]

def test_assertion_miss_stays_on_the_model():
    r = row("assert_add", 200, "{}", 2.1, True, False)
    assert r["kind"] == "task" and r["passed"] is False

def test_cold_is_not_a_fast_truncation():
    slow = row("probe", 200, "", 9.5, False, False)
    fast = row("long_ctx", 200, "truncated", 3.0, False, False)
    assert slow["kind"] == "cold" and fast["kind"] == "truncation"

def test_mixed_run_is_not_publishable():
    rows = [
        row("probe", 200, "{}", 0.3, True, True),
        row("assert_add", 200, "{}", 1.1, True, False),
        row("diff_only", 429, "quota exceeded", 0.2, False, False),
        row("long_ctx", 200, "truncated", 3.0, False, False),
    ]
    summary = summarize(rows)
    assert summary["publishable"] is False
    assert summary["blocked_kinds"] == ["quota", "truncation"]
    # 1 of 2 scorable rows, not 1 of 4 calls. The denominator is the whole trick.
    assert summary["task_pass_rate"] == 0.5

if __name__ == "__main__":
    demo = [
        row("probe", 200, "{}", 0.3, True, True),
        row("assert_add", 200, "{}", 1.1, True, False),
        row("diff_only", 429, "quota exceeded", 0.2, False, False),
        row("long_ctx", 200, "truncated", 3.0, False, False),
    ]
    print(json.dumps(summarize(demo), indent=2))

Run the synthetic path before you even think about a host. The tests are the contract. A comment rots. A failing assert nags you the next time you are tempted to average the reds.

python tier_split.py
python -m pytest tier_split.py -q

The script prints a synthetic summary, not a product measurement. I am pasting it so the denominator is visible, and so nobody can pretend the fixture secretly timed a server.

{
  "n": 4,
  "scorable": 2,
  "blocked": 2,
  "task_pass_rate": 0.5,
  "publishable": false,
  "blocked_kinds": ["quota", "truncation"]
}

Look at that 0.5 and tell me what it means. It does not mean "the model scored fifty percent." It means one of the two calls the plan did not interrupt held its assertion. The other two rows never entered the denominator. Fold them in and you can manufacture 25 percent, or 50, or a story about truncation being a logic bug. Which story did you mean? If your dashboard still demands a single rate, it wants theater. I have published theater. The publishable flag is me trying to stop. Four scorable rows, zero blocked, or the run stays in the lab notebook.

Where does a free server earn its keep, if I refuse to invent a timing table for it? It earns its keep when the thing under test is your client. Retries that should be bounded. Parsers that should fail closed. Logs that should keep the status code instead of a smile. It breaks the moment the thing under test is the model under a stable load. Shared capacity wobbles. A cold start impersonates stupidity. A quota wall impersonates a refusal. Hold the model id constant and swap only the server, and you are still not holding the environment constant, because a free server is a crowd, not a box with your name on the door.

Who should walk away from this approach. If you are choosing a model for production codegen, this is a preflight, not the decision. If the repository is private, do not paste it into a server you have not reviewed just because the invoice is zero. Free is not a privacy policy. If you have a latency SLO, a sundial will not time your marathon, and a contended free box is that sundial. If you are about to post a leaderboard, do not. Not from four tasks, and not from an unstratified rate. And if you needed me to name a winner between hosts, this piece will disappoint you on purpose.

A live call, if you insist, stays outside the tests. I am not embedding a vendor path, and I am not claiming the JSON below is accepted by anyone. Set the URL from the docs you read today. Then edit the body until it matches. Until that edit exists, you do not have a client. You have a wish.

def live_call(url, model, prompt, timeout_s=30):
    import time, urllib.request
    payload = json.dumps({"model": model, "prompt": prompt}).encode()
    req = urllib.request.Request(
        url, data=payload, headers={"Content-Type": "application/json"}
    )
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout_s) as resp:
            raw = resp.read().decode("utf-8", "replace")
            return resp.status, raw, time.perf_counter() - t0
    except Exception as exc:
        return 0, str(exc), time.perf_counter() - t0

export EVAL_BASE_URL="https://example.invalid/v1/complete"
export EVAL_MODEL_ID="read-this-from-current-docs"
# Wire live_call yourself after the payload matches. Do not point pytest at the network.

Map the return straight into row. Parse first, then classify, then summarize. If kind is auth, stop the batch. More calls will not become more science. If kind is quota or capacity, you learned something about the plan's afternoon, and you learned almost nothing about the diff task. Write that sentence in the log. Future you, skimming at speed, will otherwise promote the red cell into a model opinion.

Limitations, before anyone quotes me. The fixture cannot see whether a model can hold a real repository in its head. Token accounting stays out until a response actually carries usage fields, and I will not invent those fields to look thorough. The keyword list will miss a new error string and misfile a row, which is why raw bodies belong next to the kind. Free access can shrink or vanish without this article updating. A green pytest run means the classifier obeyed the synthetic rows. It does not mean a host agreed with me. I am not reporting a measured win, or a measured break, of any free tier here, because I did not run that measurement for this piece. If I had, you would get the raw log, the date, the model id, and the docs URL. Not a vibe. Not a crown.

There is still a Monday use. Point the same client at a free model access path and a free server option, collect kinds instead of a single rate, and decide whether the harness is even ready for a paid, pinned run. That is the job. Shake the pipes. Do not grade the water.

If your harness already writes a row per call, point it at MonkeyCode's free model access and free server option after you confirm both still exist in the docs, and keep the blocked kinds. I would rather read those than another bare pass rate.

I Sent the Eval Off-Box. Pass Rate Started Measuring the Wire.

Jordan Liu — Wed, 23 Sep 2026 16:51:56 +0000

The scoreboard was lying, and not in a subtle way. I had a green local eval for a tiny tool-calling agent, then I pointed the same harness at a remote runner and watched “accuracy” swing like weather. The model had not gotten dumber between those two runs. The clock had moved, and I was still scoring dead sockets as wrong answers.

That is the whole claim. If you export an agent eval onto someone else’s box and collapse HTTP death into a single fail bit, you are not measuring the model. You are measuring the path. Local green, remote red, same prompt, same tool schema — and a dashboard that cannot tell the difference is just a mood ring with extra YAML.

I keep doing this to myself. A smoke eval looks honest on a laptop because the loopback interface does not take a coffee break. Then CI, or a shared runner, or a free remote box enters the story, and suddenly the agent “regressed.” Did it? Or did the handshake stall long enough for httpx to tap out at ten seconds while the model was still thinking?

The contamination is boring, which is why it wins

Agent evals already mix three different failures into one sad integer. The model can pick the wrong tool. The tool can return garbage. The transport can vanish. Only the first two belong in a quality score. The third belongs in a reliability column, and if you skip that split you will “fix” a model that was never broken.

Think of it as grading a student on whether the bus showed up. You can do that. You just should not call the result algebra.

I wanted a remote place to park the runner without standing up my own box, so I used MonkeyCode for the off-box side of this experiment. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am treating their free model access and free server option as a convenience layer — a reachable endpoint and a cheap generation path — not as a promise about hardware, quotas, permanence, or speed. The article still works if you swap in any other remote URL. The method is the point.

What I actually ran

The task is deliberately small. One user turn. One allowed tool, add_two, with two integers. A correct agent emits a well-formed tool call, the tool returns the sum, and the final message echoes that number. Anything else is a model miss. Timeouts, DNS, 5xx, empty bodies, and JSON that never showed up are transport misses. Mixing them is how pass rate becomes meteorology.

I did not chase a leaderboard. I chased a classifier I could rerun on Monday without arguing with a screenshot. The harness below is the artifact. Point LOCAL_BASE at a loopback stub and REMOTE_BASE at whatever free server or API root you actually have. Keep the prompt frozen. Keep the tool schema frozen. Change one thing: the wire.

# eval_wire.py — classify transport death vs model misses
from __future__ import annotations

import json, os, time, uuid
from dataclasses import dataclass, asdict
from typing import Any, Literal

import httpx

Outcome = Literal["pass", "model_fail", "transport_fail", "protocol_fail"]

TOOL = {
    "name": "add_two",
    "description": "Add two integers and return the sum.",
    "parameters": {
        "type": "object",
        "properties": {
            "a": {"type": "integer"},
            "b": {"type": "integer"},
        },
        "required": ["a", "b"],
    },
}

PROMPT = "Use add_two to compute 17 + 25. Then say only the sum."

@dataclass
class Trial:
    run_id: str
    target: str
    outcome: Outcome
    http_status: int | None
    ttfb_ms: float | None
    total_ms: float
    error: str | None
    tool_name: str | None
    tool_args: dict[str, Any] | None
    final_text: str | None


def classify(resp: httpx.Response | None, body: dict | None, timed_out: bool) -> Outcome:
    if timed_out or resp is None:
        return "transport_fail"
    if resp.status_code >= 500 or resp.status_code in {408, 429}:
        return "transport_fail"
    if resp.status_code >= 400:
        return "protocol_fail"
    if not body:
        return "protocol_fail"
    tool = extract_tool(body)
    text = extract_text(body)
    if tool and tool[0] == "add_two" and tool[1] == {"a": 17, "b": 25}:
        # Model did the job. Echoing the sum is extra credit, not the grade.
        return "pass"
    if tool or (text and text.strip()):
        return "model_fail"
    return "protocol_fail"


def extract_tool(body: dict) -> tuple[str, dict] | None:
    # Adapt this to your chat/completions shape. Do not pretend one schema rules them all.
    choices = body.get("choices") or []
    if not choices:
        return None
    msg = choices[0].get("message") or {}
    calls = msg.get("tool_calls") or []
    if not calls:
        return None
    fn = calls[0].get("function") or {}
    name = fn.get("name")
    try:
        args = json.loads(fn.get("arguments") or "{}")
    except json.JSONDecodeError:
        return None
    if not name:
        return None
    return name, args


def extract_text(body: dict) -> str | None:
    choices = body.get("choices") or []
    if not choices:
        return None
    return (choices[0].get("message") or {}).get("content")


def one_trial(client: httpx.Client, url: str, target: str, timeout_s: float) -> Trial:
    run_id = uuid.uuid4().hex[:8]
    payload = {
        "messages": [{"role": "user", "content": PROMPT}],
        "tools": [{"type": "function", "function": TOOL}],
        "tool_choice": "auto",
    }
    t0 = time.perf_counter()
    ttfb = None
    timed_out = False
    resp = None
    body = None
    err = None
    try:
        with client.stream("POST", url, json=payload, timeout=timeout_s) as r:
            resp = r
            first = next(r.iter_bytes(), b"")
            ttfb = (time.perf_counter() - t0) * 1000
            raw = first + r.read()
        body = json.loads(raw.decode() or "{}")
    except httpx.TimeoutException as e:
        timed_out = True
        err = type(e).__name__
    except (httpx.TransportError, json.JSONDecodeError) as e:
        err = type(e).__name__
    total = (time.perf_counter() - t0) * 1000
    tool = extract_tool(body) if body else None
    return Trial(
        run_id=run_id,
        target=target,
        outcome=classify(resp, body, timed_out or err == "TransportError"),
        http_status=None if resp is None else resp.status_code,
        ttfb_ms=None if ttfb is None else round(ttfb, 1),
        total_ms=round(total, 1),
        error=err,
        tool_name=None if tool is None else tool[0],
        tool_args=None if tool is None else tool[1],
        final_text=extract_text(body) if body else None,
    )


def main() -> None:
    local = os.environ["LOCAL_BASE"].rstrip("/") + "/v1/chat/completions"
    remote = os.environ["REMOTE_BASE"].rstrip("/") + "/v1/chat/completions"
    timeout_s = float(os.environ.get("EVAL_TIMEOUT_S", "12"))
    n = int(os.environ.get("EVAL_N", "20"))
    rows: list[Trial] = []
    with httpx.Client() as client:
        for target, url in (("local", local), ("remote", remote)):
            for _ in range(n):
                rows.append(one_trial(client, url, target, timeout_s))
    print(json.dumps([asdict(r) for r in rows], indent=2))
    for target in ("local", "remote"):
        subset = [r for r in rows if r.target == target]
        print(target, tally(subset))


def tally(rows: list[Trial]) -> dict[str, float | int]:
    n = len(rows) or 1
    def rate(o: Outcome) -> float:
        return round(100.0 * sum(r.outcome == o for r in rows) / n, 1)
    transport = [r.total_ms for r in rows if r.outcome == "transport_fail"]
    return {
        "n": len(rows),
        "pass_pct": rate("pass"),
        "model_fail_pct": rate("model_fail"),
        "transport_fail_pct": rate("transport_fail"),
        "protocol_fail_pct": rate("protocol_fail"),
        "transport_p50_ms": None if not transport else sorted(transport)[len(transport)//2],
    }

if __name__ == "__main__":
    main()

Run it like a test you might actually keep, not like a demo you screenshot once:

export LOCAL_BASE=http://127.0.0.1:8080
export REMOTE_BASE=https://YOUR_REMOTE_ROOT
export EVAL_N=20
export EVAL_TIMEOUT_S=12
python eval_wire.py > wire_trials.json

Need a loopback control so “local” is not a fantasy? A stub that always emits the correct tool call is enough. You are calibrating the classifier, not auditioning a model.

# local_stub.py — control plane, not a brain
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

@app.post("/v1/chat/completions")
def completions(payload: dict):
    return JSONResponse({
        "choices": [{
            "message": {
                "role": "assistant",
                "content": None,
                "tool_calls": [{
                    "id": "call_local",
                    "type": "function",
                    "function": {
                        "name": "add_two",
                        "arguments": "{\"a\": 17, \"b\": 25}",
                    },
                }],
            }
        }]
    })

uvicorn local_stub:app --port 8080

If local pass rate is not ~100% against that stub, stop. Your parser is the bug. I have shipped that bug. It looks like a model story in every chart you will later regret.

How I read a run without lying to myself

I refuse a single accuracy number now. Four buckets, or I do not publish the row. Pass means the tool call matched. Model fail means the bytes arrived and the agent still flinched. Transport fail means the wire blinked. Protocol fail means we could not even parse the play — wrong shape, missing choices, arguments that are not JSON.

Here is an illustrative slice, labeled because it is a fixture for the method, not a product benchmark and not a promise about any host. I am showing the shape of the argument, not a trophy.

# illustrative fixture — replace with your wire_trials.json
local  {"n": 20, "pass_pct": 100.0, "model_fail_pct": 0.0,
        "transport_fail_pct": 0.0, "protocol_fail_pct": 0.0}
remote {"n": 20, "pass_pct": 55.0, "model_fail_pct": 15.0,
        "transport_fail_pct": 25.0, "protocol_fail_pct": 5.0}

Look at that remote line. A lazy dashboard screams 55% accuracy and someone starts swapping prompts. Split the buckets and a quarter of the pain is the wire. Another sliver is protocol drift. The model-shaped remainder is 15%, which is a different conversation than “the remote model is bad.” Same eval file. Different adult in the room.

Was the remote side slower? Sometimes. Is slower the same as wrong? Only if your timeout is a moral judgment. I keep EVAL_TIMEOUT_S in the log next to the tally because a 12-second knife and a 60-second knife do not cut the same fruit. Change the knife, change the score, and you still have not touched the weights.

One more cut that people skip: TTFB versus total time. If TTFB explodes and the body never arrives, that is not a bad tool call. That is a cold start, a queue, a proxy, a weekend. If TTFB is fine and the arguments are {"a": "seventeen"}, congratulations, you found an actual model miss. See the difference? One of those belongs in a routing ticket. The other belongs in a prompt diff.

When the free remote path is honest enough

I still use a free remote runner. I just do not let it author the grade. Smoke tests, schema checks, “does the tool name even come back,” overnight soak that I will read with the classifier on — that is fair. You get more attempts per unit of patience, and you find parser bugs faster than you find enlightenment.

I would not use it as the clock for a launch decision. I would not pour customer text into it. I would not treat a free model as a hidden gold judge and then act shocked when the judge wobbles. Cheap generation is a gift for iteration. It is a lousy supreme court.

If you need p99 promises, data residency, or a paper that pretends the network was not in the room, this approach will waste your week. Run local, or run a box you can ssh into, and keep the same classifier anyway. The classifier is the habit. The free server is just one place the habit can live.

A decision I actually use, said as sentences because a matrix makes people stop thinking. If transport_fail stays under a small slice and model_fail is stable across days, the remote path is good enough for smoke. If transport_fail moves when I change timeout and model_fail does not, I am holding a network thermometer. If protocol_fail spikes, I am on the wrong response shape and no amount of prompt poetry will help. If local stub is not a clean pass, I do not get to talk about models today.

What I will not claim

I am not going to invent a token ceiling, a GPU SKU, a durability story, or a speed ranking. Those claims go stale before the comment thread does, and I do not have a primary source in this draft that would survive a skeptic with a calendar. The reproducible part is the split: wire versus brain versus parser. You can run it on a free server, on a laptop, on a machine you regret buying. If the buckets do not exist, the percentage is theater.

So export the eval if you want. Just stop letting the socket sit for the exam. If you try the harness and the remote tally shocks you, good. That shock is the first honest number you have had all week. If you want a remote box for the runner without building one, the free server option I used is enough to get the classifier dirty — after that, the work is reading the buckets, not collecting a slogan.

I Timed the Tool Call. The Schema Still Said Yes.

Jordan Liu — Tue, 22 Sep 2026 15:08:36 +0000

A schema-valid tool call that would hang your API is not a pass. It is a delayed incident with nicer JSON. I stopped letting the schema be the whole exam, and the scoreboard got ugly in a useful way.

Most tool-calling writeups still grade the envelope. Did the model emit a function name you advertised. Did the arguments parse. Did required keys show up. That is a spelling test. Who grades the pitch clock.

Last week the feed was full of “how AI actually calls an API” explainers and a burst of posts about realistic API performance tests. Fine. Those are adjacent problems pretending they are the same lab. One is about whether the model can fill a form. The other is about whether the form, once submitted, survives contact with a slow, flaky, rate-limited endpoint. I wanted one harness that refused to separate them.

Here is the conclusion I actually trust: if your eval never times out, it is theater. The model can look brilliant while proposing a GET that your production budget would have killed at 800ms. The chart still paints a green box. You ship the green box. Then on-call inherits the clock.

I did not run a beauty contest against unnamed models. I built a 24-trace fixture pack with planted faults and a scorer that treats implied HTTP as part of correctness. The numbers below come from that pack. Swap live completions in when you have them. The scorer does not care who typed the JSON.

The fixture is a tiny order API, because weather demos never 504 in the stories we tell ourselves. Each trace is a tool call plus the latency I would have injected if that call had hit a real wire. Schema validity is necessary. It is not sufficient. If the implied request blows the budget, the call is wrong even when the JSON is pretty.

# toolcall_budget_eval.py
from __future__ import annotations

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any

BUDGET_MS = 800
ALLOWED_TOOLS = {
    "get_order": {
        "method": "GET",
        "path": "/orders/{order_id}",
        "required": {"order_id"},
    },
    "cancel_order": {
        "method": "POST",
        "path": "/orders/{order_id}/cancel",
        "required": {"order_id", "reason"},
    },
}

@dataclass
class Verdict:
    trace_id: str
    schema_ok: bool
    args_ok: bool
    under_budget: bool
    passed: bool
    why: str

def load_trace(path: Path) -> dict[str, Any]:
    data = json.loads(path.read_text())
    for key in ("id", "tool", "arguments", "injected_latency_ms"):
        if key not in data:
            raise ValueError(f"{path} missing {key}")
    return data

def grade(trace: dict[str, Any], budget_ms: int = BUDGET_MS) -> Verdict:
    tool = trace["tool"]
    args = trace["arguments"]
    latency = int(trace["injected_latency_ms"])
    spec = ALLOWED_TOOLS.get(tool)
    if spec is None:
        return Verdict(trace["id"], False, False, latency <= budget_ms, False, "unknown_tool")
    missing = spec["required"] - set(args)
    extra_empty = any(args.get(k) in (None, "") for k in spec["required"])
    args_ok = not missing and not extra_empty
    schema_ok = isinstance(args, dict)
    under = latency <= budget_ms
    if not schema_ok:
        why = "args_not_object"
    elif not args_ok:
        why = f"bad_args:{sorted(missing) or 'empty'}"
    elif not under:
        why = f"over_budget:{latency}ms"
    else:
        why = "ok"
    return Verdict(trace["id"], schema_ok, args_ok, under, schema_ok and args_ok and under, why)

def grade_dir(folder: str) -> list[Verdict]:
    paths = sorted(Path(folder).glob("*.json"))
    return [grade(load_trace(p)) for p in paths]

if __name__ == "__main__":
    import sys
    rows = grade_dir(sys.argv[1] if len(sys.argv) > 1 else "traces")
    passed = sum(1 for r in rows if r.passed)
    schema_only = sum(1 for r in rows if r.schema_ok and r.args_ok)
    print(f"traces={len(rows)} schema_only={schema_only}/{len(rows)} budget_aware={passed}/{len(rows)}")
    for r in rows:
        flag = "PASS" if r.passed else "FAIL"
        print(f"{flag} {r.trace_id:12} schema={int(r.schema_ok)} args={int(r.args_ok)} budget={int(r.under_budget)} {r.why}")

That is the whole argument in a file. Schema and arguments can still win. The clock can still veto. I am tired of dashboards that never give the clock a vote.

The planted pack is boring on purpose. Eight traces are clean and fast. Four call a tool I never offered. Four omit reason on cancel. Four are perfect JSON aimed at an endpoint I delayed to 1200ms. Four mix a valid cancel with 400ms, which should pass, because I do not want a scorer that fails everything just to look strict. Strict and sad are not the same skill.

Drop them in traces/ like this.

{
  "id": "t-over-budget-03",
  "tool": "get_order",
  "arguments": {"order_id": "ord_19"},
  "injected_latency_ms": 1200
}

Run it locally. No GPU. No mystery SaaS.

python toolcall_budget_eval.py traces

On this pack the script prints traces=24 schema_only=20/24 budget_aware=12/24. Those eight extra “wins” under schema-only scoring are the delayed GETs. They look like competence. They are a 504 with etiquette. If you only count JSON, you will brag about 83 percent. If you count the wire, you are at 50 percent, and you should be. The missing 33 points were never intelligence. They were patience you do not have in production.

Is 800ms sacred. No. I picked it because it is mean enough to catch a frozen tool loop and loose enough that a healthy p95 still lives. Change the constant. The point is that a number exists. An eval without a number is a vibe.

I parked the judge next to the traces instead of next to a GPU I do not want to babysit. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is the only reason I mention it: the scorer is local Python, and the free server is a convenient place to run the same command when my laptop is already compiling something noisier. I am not going to invent model names, token ceilings, or hardware I did not measure. If your workspace already has that free model and free server path, point it at traces/ and keep the budget in git. If it does not, the file above still runs.

Where does this break. Everywhere you hoped it would not. Injected latency is not real latency. A fixture cannot reproduce jitter, cold starts, or the way a gateway starts lying after 200 concurrent cancels. The scorer treats one call as the world. Agents retry. Retries multiply the budget you thought you had. I did not model that here on purpose. If I hide a retry loop inside “helpfulness,” we are back to theater, just with more curtains.

It also breaks when the tool schema is a lie. If get_order is documented as 200ms and the real handler does a sequential billing lookup, your budget is fan fiction. Grade the contract you actually ship, not the README. And it breaks when you let the same model write the traces and grade them. That is a student marking their own handwriting. Keep the pack human-authored, or at least frozen before the run.

Should you use this if you are demoing a chatbot that never leaves the browser. Maybe not. A UI toy can survive a slow tool call with a spinner and a shrug. Should you use this if your “API” is an in-process function with no network. No. You would be timing a function call and calling it operations. Should you use this as a model leaderboard. Please do not. Twenty-four planted traces are a unit test for your grading story, not a census of intelligence.

Use it when a tool call implies HTTP, when HTTP implies a budget, and when someone in the room is about to present schema pass-rate as quality. Ask them what happened to the calls that would have waited. If they cannot answer, the chart is decoration.

I keep the budget in the same commit as the traces. That is the whole workflow. Change the number, change the pack, watch schema-only and budget-aware diverge. When they stop diverging, either your endpoint got honest or your fixtures got timid. One of those is a win. Guess which.

If you already have a MonkeyCode workspace with the free model access and free server option, run the scorer there and keep the traces in git. Steal the file either way. The clock does not need a product to be right.

I Counted Drops as Wrongs. The Chart Was Theater.

Jordan Liu — Mon, 21 Sep 2026 13:21:10 +0000

The first number on an eval dashboard is usually a lie. Not a scam. A folding error. You asked a model for an answer, the path blinked, and your scorer filed the blink under incorrect.

I stopped scoring completions until I scored the envelope.

If you cannot tell a drop from a wrong, you are not ranking models. You are ranking weather. Shared free endpoints make the weather louder: idle processes, truncated streams, empty choices that still ride in on HTTP 200. The status looks fine. The grade looks like failure. That is how a leaderboard turns into theater.

I wanted a harness that fails loudly at the transport layer before it whispers at the semantics layer. So I planted six failure modes, ran a classifier over twenty-four envelopes, and watched a naive pass-rate collapse on purpose. The code below is the experiment. The percentages are the fixture talking, not a vendor scoreboard.

The folding error

Most agent evals I see still do some version of success = (http == 200 and expected in text). Cute. Also how you launder a timeout into a dumb model.

Think of it like grading a take-home by weighing the envelope. A missing envelope is not a wrong proof. A torn envelope is not a wrong proof. A proof written in the wrong schema is closer, but it is still not the same as a proof that multiplies instead of factors. Mix them and you will fire the student who had a postal strike.

Free inference is the postal strike. I do not mean that as an insult. I mean that a path you are not paying for will drop, stall, and truncate in ways a dedicated box will not, and your scorer has to admit that or it will keep "proving" that cheaper models are worse at arithmetic when they were worse at staying on the line.

That is why I care about a free model path and a free server option at all. Not because I want an adjective. Because I want a hostile mailbox. If the harness survives a mailbox, it will not panic on a quiet one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Their free model access and free server option are the mailbox I have in mind when I say "hostile." I am not naming models, quoting quotas, or claiming uptime. If you cannot reproduce a claim from the fixture in this file, treat the claim as unset.

Six envelopes, one bit

I keep six labels, and I refuse to let them share a boolean. drop is a refused connection, a reset, a client timeout, a 429, a 503. empty is a 200 with no choices or a null content string. truncated is a finish_reason of length, or JSON that dies mid-key. schema is parseable JSON that misses a required field. wrong is valid JSON with the wrong answer. right is the boring one.

Ask yourself what your current CSV does with empty. If the answer is "fail," you already know why last week's ranking felt jumpy. The model did not change. The path coughed, and you graded the cough.

I did not need a live fleet to test the scorer. I needed lies I controlled. Twenty-four envelopes, four of each label, stuffed into memory. The naive scorer treats anything that is not right as a model miss. The envelope scorer reports two numbers: yield, meaning bodies that were even gradeable, and accuracy-on-yield, meaning rights over rights plus wrongs.

Here is the whole trick. Yield answers "did the path hand me an object?" Accuracy-on-yield answers "was the object correct?" If you publish only the product of those, you are back to theater.

The planted run

Save this as envelope_eval.py. It never calls a network. That is the point. If a scorer cannot recover a lie you planted on disk, it will not recover a lie a free server plants for you at 4 p.m.

#!/usr/bin/env python3
"""Planted-envelope scorer. Run: python envelope_eval.py"""
from __future__ import annotations

import json
from collections import Counter
from dataclasses import dataclass
from typing import Any, Optional

@dataclass
class Envelope:
    http_status: int
    elapsed_ms: float
    finish_reason: Optional[str]
    body: Any
    expected: dict

def classify(env: Envelope) -> str:
    if env.http_status in {0, 408, 429, 500, 502, 503, 504}:
        return "drop"
    if env.http_status != 200:
        return "drop"
    if not isinstance(env.body, dict):
        return "empty"
    choices = env.body.get("choices") or []
    if not choices:
        return "empty"
    content = (choices[0].get("message") or {}).get("content")
    reason = env.finish_reason or choices[0].get("finish_reason")
    if content in (None, ""):
        return "empty"
    if reason == "length":
        return "truncated"
    try:
        parsed = json.loads(content) if isinstance(content, str) else content
    except json.JSONDecodeError:
        return "truncated"
    if not isinstance(parsed, dict) or "answer" not in parsed:
        return "schema"
    if parsed.get("answer") != env.expected.get("answer"):
        return "wrong"
    return "right"

def plant() -> list[Envelope]:
    expected = {"answer": 42}
    good = {"choices": [{"message": {"content": '{"answer": 42}'}, "finish_reason": "stop"}]}
    bad = {"choices": [{"message": {"content": '{"answer": 7}'}, "finish_reason": "stop"}]}
    missing = {"choices": [{"message": {"content": '{"value": 42}'}, "finish_reason": "stop"}]}
    cut = {"choices": [{"message": {"content": '{"answ'}, "finish_reason": "length"}]}
    empty = {"choices": []}
    rows: list[Envelope] = []
    for _ in range(4):
        rows.append(Envelope(0, 30000, None, None, expected))
        rows.append(Envelope(200, 800, "stop", empty, expected))
        rows.append(Envelope(200, 1200, "length", cut, expected))
        rows.append(Envelope(200, 900, "stop", missing, expected))
        rows.append(Envelope(200, 700, "stop", bad, expected))
        rows.append(Envelope(200, 650, "stop", good, expected))
    return rows

def main() -> None:
    labels = [classify(env) for env in plant()]
    counts = Counter(labels)
    n = len(labels)
    naive = sum(lab == "right" for lab in labels) / n
    gradeable = counts["wrong"] + counts["right"]
    acc = counts["right"] / gradeable if gradeable else float("nan")
    print("counts:", dict(counts))
    print(f"naive_pass_rate={naive:.3f}")
    print(f"yield={gradeable / n:.3f}")
    print(f"accuracy_on_yield={acc:.3f}")

if __name__ == "__main__":
    main()

Run it.

python envelope_eval.py

You should get four of each label, naive_pass_rate=0.167, yield=0.333, accuracy_on_yield=0.500. That naive 16.7% is true and useless. It says the system is bad at the task. It is not. The system is 50% accurate on the eight gradeable semantic answers, and sixteen of twenty-four envelopes never earned a semantic grade. If I had shipped the naive number, I would have "proven" a model regression that was actually four drops, four empties, four truncations, and four schema misses I planted myself.

Would you publish 16.7% in a weekly review? I have. That is the part that still bothers me.

What I do with each label

I keep a small matrix next to the scorer so I cannot "just retry it" at 11 p.m. and call that science.

Label	Counts as model evidence?	Action
drop	no	retry with a budget, then park the row
empty	no	retry once, then park
truncated	no	raise max tokens or repair, cap the repairs
schema	no	repair or fail the harness, not the model
wrong	yes	score it, do not retry
right	yes	score it, do not retry

Retry only drops and empties. Repair only truncated and schema, and cap the repairs so you do not invent a second model with your JSON fixer. Score only wrong versus right. Never retry a wrong. Wrong is information. A retry on a wrong is how you accidentally bake temperature into a pass@1 and then argue with your own chart.

Where a free path helps, and where it breaks

I use a free model and a free server as a canary for the harness, not as a verdict on the product under test. The canary's job is to produce ugly envelopes on a weekday afternoon. If my classifier labels them, the paid run gets a cleaner CSV. If my classifier folds them, I am about to buy myself a fake regression.

It breaks in the obvious ways. A free path is not a latency SLA. It is not a promise that temperature 0 is deterministic. It is not a substitute for a held-out human grade. If your paper needs matched seeds across labs, do not use a shared free box as the official judge. If you are on a deadline and you cannot afford a thirty-second stall before a drop, this is the wrong mailbox. If you need to claim "the model scored 71%," this article will not give you that sentence, and neither should a canary.

I also do not use the same endpoint for candidate and judge when I can avoid it. Shared load couples their clocks. The judge gets slow, your client times out, and you file a drop as a disagreement. That loop is how evals invent opinions. Two mailboxes. One grade.

What I log now

Every row gets http_status, latency_ms, finish_reason, bytes, label, and only then answer. The live probe looks like this, and I keep --max-time boringly small so the client timeout is a fact instead of a hang.

curl -sS -D /tmp/hdr.txt --max-time 30 \
  -H "content-type: application/json" \
  --data @prompt.json \
  "$ENDPOINT/v1/chat/completions" | tee /tmp/body.json | \
  python -c "import json,sys; d=json.load(sys.stdin); c=d.get('choices') or []; print({'n_choices': len(c), 'finish': c[0].get('finish_reason') if c else None, 'bytes': len(json.dumps(d))})"
wc -c /tmp/body.json

If n_choices is 0, I do not regex the content for the golden string. There is no content. Stop pretending. HTTP 200 is a door. It is not a grade.

Who should ignore this? If you already isolate transport in your eval warehouse, you do not need my JSONL. If you are demoing a toy prompt in a notebook, you do not need six labels. If you are trying to make a free server look like a dedicated GPU, you are solving a press-release problem, and this workflow will not help you.

I still run the planted file before I point the same scorer at anything with a bill, or anything without one. Steal the classifier first. If you want a mailbox that will actually drop on you while you do it, MonkeyCode's free model access and free server option are one place to aim the canary. Read the labels. Do not publish the folded number.

I Subtracted the Queue. The Fast Model Wasn't.

Jordan Liu — Sun, 20 Sep 2026 11:26:58 +0000

The number on your latency dashboard is not a model score. It is a traffic report wearing a lab coat.

I keep watching people rank free endpoints by time-to-first-token as if TTFT were a property of weights. It isn't. Not when the box is shared. Not when your so-called eval is standing in the same line as everyone else's autocomplete. You are scoring the parking lot. Then you publish a leaderboard. Then you act shocked when Tuesday disagrees with Thursday.

What happens when the tests we use to measure models get slower than the models? The tests start measuring the hallway.

I wanted a splitter I could run twice and trust. Wait on one side. Generation on the other. If wait dominates, I throw the run out. If generation dominates, I keep it. That is the whole method. It is boring on purpose. Boring is how you stop lying to yourself.

The lab for this is not a secret GPU in a closet. MonkeyCode is an open source project with free model access and a free server option, which is exactly the shape of infrastructure that makes queue contamination loud. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not naming a model, a quota, a SKU, or a duration I cannot verify. I am naming a measurement error you can reproduce on any shared endpoint, including that one.

Picture the failure. You send the same prompt ten times. Naive wall clock swears run 3 was a genius and run 7 was a brick. Split the timestamps and run 3 sat in queue for a blink while run 7 waited for a bus. Tokens per second never moved. Your ranking did. Are you still averaging those ten numbers like they mean the same thing?

I treat three clocks as sacred. t_submit is the instant the client writes the last request byte. t_first is the instant the first response byte lands. t_done is the instant the stream closes. Wait is first minus submit. Work is done minus first. Contamination is wait divided by wait plus work. If that ratio is high, the run is weather, not intelligence.

You want code, not a sermon. Fine. This harness is the artifact. The fixture traces are labeled samples, not a product benchmark I pretended to run.

# queue_split.py — labeled fixture + splitter. Not a vendor scorecard.
from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class Trace:
    run_id: str
    t_submit: float
    t_first: float
    t_done: float
    prompt_hash: str
    streaming: bool
    cold: bool

    @property
    def wait_s(self) -> float:
        return max(0.0, self.t_first - self.t_submit)

    @property
    def work_s(self) -> float:
        return max(0.0, self.t_done - self.t_first)

    @property
    def wall_s(self) -> float:
        return max(0.0, self.t_done - self.t_submit)

    @property
    def contamination(self) -> float:
        total = self.wait_s + self.work_s
        return 0.0 if total == 0 else self.wait_s / total

    def verdict(self, discard_at: float = 0.6) -> str:
        if not self.streaming:
            return "UNSPLITTABLE"
        if self.contamination >= discard_at:
            return "DISCARD"
        if self.contamination >= 0.2:
            return "WORK_ONLY"
        return "WALL_OK"


# Illustrative traces only. Same prompt_hash on purpose.
FIXTURE = [
    Trace("clean", 0.00, 0.12, 1.40, "p1", True, False),   # short wait, steady work
    Trace("queued", 0.00, 2.10, 3.38, "p1", True, False),  # long wait, SAME work interval
    Trace("slow", 0.00, 0.15, 3.05, "p1", True, False),    # short wait, actually slow generate
    Trace("buffered", 0.00, 2.80, 2.81, "p1", False, True), # non-streaming: clocks collapse
]


def rank(traces, key):
    return [t.run_id for t in sorted(traces, key=key)]


if __name__ == "__main__":
    streamed = [t for t in FIXTURE if t.streaming]
    print("naive wall rank:", rank(streamed, lambda t: t.wall_s))
    print("work-only rank:", rank(streamed, lambda t: t.work_s))
    for t in FIXTURE:
        print(
            f"{t.run_id:9} wait={t.wait_s:.2f}s work={t.work_s:.2f}s "
            f"contam={t.contamination:.2f} -> {t.verdict()}"
        )

Run it locally. No credentials. No cloud required for the lie detector itself.

python queue_split.py

You should see the reversal immediately. Naive wall clock ranks clean, then slow, then queued. Work-only ranking puts clean and queued in the same neighborhood and leaves slow as the actual drag. That flip is the bug. buffered comes back UNSPLITTABLE because a one-shot body gives you a single thud, not two clocks. If your eval client does that, stop calling it TTFT. You do not have TTFT. You have a door slam.

I pipe real calls through the same dataclass. The HTTP client is deliberately ugly. Environment variables so this article does not invent a hostname, a model id, or a token budget.

# live_probe.py — unexecuted template. Fill env. Do not treat output as a brand score.
import hashlib, json, os, time, urllib.request
from queue_split import Trace

URL = os.environ["EVAL_URL"]          # your endpoint, not mine
BODY = os.environ.get("EVAL_BODY", '{"prompt":"ping","stream":true}')

def stream_once(run_id: str, cold: bool) -> Trace:
    payload = BODY.encode()
    req = urllib.request.Request(URL, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    t_submit = time.perf_counter()
    t_first = None
    with urllib.request.urlopen(req, timeout=120) as resp:
        while True:
            chunk = resp.read(64)
            now = time.perf_counter()
            if not chunk:
                break
            if t_first is None:
                t_first = now
        t_done = time.perf_counter()
    if t_first is None:
        t_first = t_done
    digest = hashlib.sha256(payload).hexdigest()[:12]
    streaming = b"stream" in payload and b"true" in payload.lower()
    return Trace(run_id, t_submit, t_first, t_done, digest, streaming, cold)

if __name__ == "__main__":
    rows = [stream_once(f"r{i}", cold=(i == 0)) for i in range(10)]
    print(json.dumps([r.__dict__ | {
        "wait_s": r.wait_s, "work_s": r.work_s,
        "contamination": r.contamination, "verdict": r.verdict(),
    } for r in rows], indent=2))

export EVAL_URL="http://127.0.0.1:8080/v1/stream"   # local stand-in
export EVAL_BODY='{"prompt":"ping","stream":true}'
python live_probe.py

Do I always get a streaming socket? No. If the free server buffers the whole completion, t_first and t_done collapse, and wait swallows work. Then the splitter cannot save you. That is a limitation, not a footnote. Non-streaming free servers make this method weaker, not stronger. If your loop expects one JSON blob, you are timing a black box. Split nothing. Say so out loud.

How do I decide keep versus discard? I do not worship a magic cutoff. I look at the batch. If contamination sits under about 0.2, I treat wall clock as mostly work. If it wanders from 0.2 to 0.6, I report work time and I print the wait column in the same breath so nobody can hide. If a run clears 0.6, I discard it for ranking. I still log it. Hidden discards are how people fake a calm night.

Is 0.6 science? No. It is a fence I can defend in a code review. You can move the fence. You cannot skip the column. Would you ship a unit test that asserts elapsed < 2.0 on a machine you do not own? Then why is your model eval doing that?

I also refuse to mix cold and warm. The first call after a pause is a different animal. I tag it. I do not average it with the next nine. People love n=10 until you ask which one was the kettle boiling. A free server after idle is a library that has to turn the lights on. That is not a regression in the book.

Where this holds together: identical prompts, streaming responses, a client you control, and a reason to compare two setups on the same night. Where it falls apart: tool-call agents whose "first token" is an opening brace, judges that treat latency as quality, batches that span a traffic cliff, and any protocol that cannot show you the first byte. A free server is a city street. Rush hour is not a model regression. Off-peak is not a breakthrough.

Should you use this if you are publishing a public model card? Only as a hygiene check, never as the headline metric. Should you use this if you are shopping dedicated hardware? No. Buy a stopwatch that is not shared. Should you use this if your eval is a chatbot screenshot and a vibe? Please stop. The method will not fix a missing clock.

I still like free inference. I like it the way I like a public library. Useful. Crowded. A terrible place to time how fast you read. An open source project plus a free server is a gift for iteration. It is a trap for stopwatch science. Hold both thoughts in the same head.

If you already have a slot on that free server, do not start with a blog graph. Start with one prompt, ten streamed calls, and the wait column. If wait moves and work does not, your old dashboard was theatre. That is the only invitation I have.

I Imported the Patch. That's When It Died.

Jordan Liu — Sat, 19 Sep 2026 09:31:28 +0000

I imported the patch. That's when it died.

ast.parse had already smiled. The chat had already called the work complete. Neither of those is a gate I trust. If your coding-model eval still stops at "looks like Python," you are grading handwriting, not software.

Why does this keep happening? Because a lot of us are still scoring the story of a fix. Fluency is cheap. Import is not. I wanted a number I could defend without recruiting a second model to compliment the first one.

So I built a three-gate harness: parse, import, test. Same fixture. Same patch. Three increasingly rude questions. The code below is the experiment. You can run it without a GPU and without believing a single adjective.

The fixture is small on purpose

I do not need a web app to catch a liar. I need a module that parses, then explodes when Python executes the top level, then fails a real assertion if you get past that. Parse-time green is a vanity metric. Import is the first honest gate.

# shop.py  — broken on purpose
from decimal import Decimal, ROUND_HALF_UP

TAX = None  # populated only if you actually import this file correctly

def require_tax_table():
    if TAX is None:
        raise RuntimeError("tax table was never loaded")
    return TAX

# Module-level landmine. ast.parse will not run this.
require_tax_table()

def add_tax(cents: int) -> int:
    rate = require_tax_table()
    money = Decimal(cents) * (Decimal("1") + rate)
    return int(money.to_integral_value(rounding=ROUND_HALF_UP))

# test_shop.py
from shop import add_tax

def test_add_tax_rounds_half_up():
    assert add_tax(100) == 108
    assert add_tax(1) == 1

See the trap? ast.parse never calls require_tax_table(). A model can rewrite add_tax with gorgeous syntax and still leave the import path haunted. That gap is what I wanted to count. Chat will not count it for you.

The intended fix is boring. Load a Decimal("0.08") tax table before anyone calls add_tax, keep the rounding, do not invent a helper that is not in the file. Boring is the point. I am not grading creativity. I am grading whether the module exists in a Python process.

I ask for a diff. I refuse to read it.

The prompt is hostile on purpose. If the model wants to narrate, it can narrate into a zero.

Return a unified diff for shop.py only.
Do not explain. Do not wrap the diff in markdown.
The file must import and test_add_tax_rounds_half_up must pass.
Tax rate is 8%. Use decimal rounding already in the file.

Then I do not read the completion. I extract. I apply. I score. If that sounds unfriendly, good. An oracle should be unfriendly.

import re, subprocess, sys, tempfile, shutil, ast
from pathlib import Path

DIFF_RE = re.compile(r"(?ms)^--- .*^\+\+\+ .*?^(?=\Z)", re.M)

def extract_unified_diff(text: str) -> str | None:
    if text.lstrip().startswith("--- "):
        return text
    blocks = re.findall(r"```

(?:diff|patch)?\n(.*?)

```", text, re.S)
    for b in blocks:
        if b.lstrip().startswith("--- "):
            return b
    return None  # essays get nothing. no pity parse.

If the completion is a memoir with a fenced snippet in the middle, and that snippet is not a unified diff, the extractor returns None. Score zero at gate zero. No partial credit for confidence. I used to "helpfully" take the first code fence. That is how you score markdown as Python and then convince yourself the model improved.

Applying the patch is equally unromantic:

def apply_diff(src: Path, diff: str) -> Path | None:
    tmp = Path(tempfile.mkdtemp())
    shutil.copytree(src.parent, tmp, dirs_exist_ok=True)
    patch = tmp / "change.diff"
    patch.write_text(diff)
    r = subprocess.run(
        ["patch", "-p1", "--batch", "-d", str(tmp)],
        input=diff, text=True, capture_output=True,
    )
    return tmp if r.returncode == 0 else None

Need patch on the machine. That is a feature. If your eval loop cannot survive a 1970s Unix tool, it is not ready to survive a model that emits @@ headers for files you never asked it to touch.

Three gates, one script

def gate_parse(py: Path) -> bool:
    try:
        ast.parse(py.read_text())
        return True
    except SyntaxError:
        return False

def gate_import(py: Path) -> bool:
    r = subprocess.run(
        [sys.executable, "-c", f"import sys; sys.path.insert(0, {py.parent.as_posix()!r}); import shop"],
        capture_output=True, text=True,
    )
    return r.returncode == 0

def gate_test(root: Path) -> bool:
    r = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", str(root / "test_shop.py")],
        capture_output=True, text=True,
    )
    return r.returncode == 0

def score(completion: str, fixture: Path) -> dict:
    diff = extract_unified_diff(completion)
    if not diff:
        return {"extract": False, "parse": False, "import": False, "test": False}
    root = apply_diff(fixture, diff)
    if root is None:
        return {"extract": True, "parse": False, "import": False, "test": False}
    py = root / "shop.py"
    parsed = gate_parse(py)
    imported = parsed and gate_import(py)
    tested = imported and gate_test(root)
    return {"extract": True, "parse": parsed, "import": imported, "test": tested}

I run it against checked-in completions, not against a vibe:

python score_patch.py fixtures/essay.txt
python score_patch.py fixtures/parse_only.txt
python score_patch.py fixtures/import_ok_test_fail.txt
python score_patch.py fixtures/pass.txt

Those four files are oracles for the scorer. They are not a vendor bake-off. If my grader cannot separate them, I do not get to grade a model yet. The labeled output below is from those fixtures, which you can paste yourself. It is not a claim about any hosted model.

# fixture-driven scorer output (not a product benchmark)
essay                 extract=0 parse=0 import=0 test=0
parse_only            extract=1 parse=1 import=0 test=0
import_ok_test_fail   extract=1 parse=1 import=1 test=0
pass                  extract=1 parse=1 import=1 test=1

parse_only is the one that should make you nervous. It is valid syntax. It still cannot be imported. An LLM judge will often call it solid. ast.parse will call it fine. Python will not.

What the gates actually argue about

Parse-only completions are empty calories. They look like engineering. They are essays wearing parentheses. I used to stop there because it felt quantitative. It was not.

Import failures are more interesting. The model invents load_rates(), or it leaves a name it saw in a comment, or it deletes the landmine and forgets to put a tax table back. Fluency stays high. The process dies before pytest exists. That is not a "almost." That is a format error with extra steps.

Test failures after a clean import are the only failures I treat as the model having engaged the problem. Everything before that is packaging. I blurred those categories for months. That was on me, not on the model.

Want a one-liner while you iterate on the extractor?

python -c "import ast,pathlib; ast.parse(pathlib.Path('shop.py').read_text()); print('parsed')"
python -c "import shop" || echo "imported? no"
python -m pytest -q test_shop.py

Three commands. Three different truths. If your dashboard collapses them into one thumbs-up, your dashboard is lying with better typography.

Where a cheap generation loop helps, and where it breaks

I still need something to emit candidate diffs. Burning a paid endpoint to discover that a pile of samples never import is how you light money on fire. The mechanical grader is the expensive-looking part that is actually free. Generation is the part that tempts you to skip gates because "it read well."

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode only as a generation pipe: free model access and a free server option, so the loop can throw diffs at this harness without me standing up inference. The scorer does not call it. Delete the pipe and the gates still work. That is the only honest integration I can defend.

Where that setup performs well: volume. You can park messy completions against parse/import/test until the failure taxonomy stops moving. You are paying in attention, not in GPU mythology. Where it breaks: the free path will not save a weak fixture. If your test is assert True, import becomes theater again. Unstructured completions wrap diffs in sermons; if you relax the extractor, you will grade the sermon. Truncation dies at parse or import, which is good, unless you mark those as "needs review" instead of fail. Do not do that. Fail is a complete sentence.

Limitations, out loud

This harness does not measure taste. It will pass a brutal one-liner that satisfies test_add_tax. It will fail a correct redesign that renames the function. It is hostile to refactors. That is a bias, not a virtue. It also will not catch a wrong tax rate if you never asserted 108. Weak tests make import look like enlightenment.

Do not use this if you are evaluating docs, architecture notes, or UI copy. Do not use it as a hiring rubric. Do not use it to stamp "production ready" on a model because gate three went green on one file. If you need semantic review, add a linter, then a type checker, then stop before you add another LLM as judge. The last time I let a model grade a model, pretty prose still moved the score.

I also do not claim these gates outlive a moving product surface. Free access and a free server are operator-supplied availability notes, not a promise about models, hardware, duration, or quota. If the pipe changes, the scorer should not care. If the scorer cares, you coupled the wrong things.

The ranking that flipped

Once I ignored chat, my personal ranking of "this completion looks good" collapsed. The essay I would have merged in a hurry scored zeros. The ugly diff that never said "sure" imported and passed. That is the whole lesson, and it does not require a leaderboard.

So I am done asking whether AI outgrew our tests. I ask whether our tests ever loaded the code. Most days they did not. Parse is polite. Import is not. I grade the rude one.