DEV Community: Jordan Liu

I Timed the Tool Call. The Schema Still Said Yes.

Jordan Liu — Tue, 22 Sep 2026 15:08:36 +0000

A schema-valid tool call that would hang your API is not a pass. It is a delayed incident with nicer JSON. I stopped letting the schema be the whole exam, and the scoreboard got ugly in a useful way.

Most tool-calling writeups still grade the envelope. Did the model emit a function name you advertised. Did the arguments parse. Did required keys show up. That is a spelling test. Who grades the pitch clock.

Last week the feed was full of “how AI actually calls an API” explainers and a burst of posts about realistic API performance tests. Fine. Those are adjacent problems pretending they are the same lab. One is about whether the model can fill a form. The other is about whether the form, once submitted, survives contact with a slow, flaky, rate-limited endpoint. I wanted one harness that refused to separate them.

Here is the conclusion I actually trust: if your eval never times out, it is theater. The model can look brilliant while proposing a GET that your production budget would have killed at 800ms. The chart still paints a green box. You ship the green box. Then on-call inherits the clock.

I did not run a beauty contest against unnamed models. I built a 24-trace fixture pack with planted faults and a scorer that treats implied HTTP as part of correctness. The numbers below come from that pack. Swap live completions in when you have them. The scorer does not care who typed the JSON.

The fixture is a tiny order API, because weather demos never 504 in the stories we tell ourselves. Each trace is a tool call plus the latency I would have injected if that call had hit a real wire. Schema validity is necessary. It is not sufficient. If the implied request blows the budget, the call is wrong even when the JSON is pretty.

# toolcall_budget_eval.py
from __future__ import annotations

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any

BUDGET_MS = 800
ALLOWED_TOOLS = {
    "get_order": {
        "method": "GET",
        "path": "/orders/{order_id}",
        "required": {"order_id"},
    },
    "cancel_order": {
        "method": "POST",
        "path": "/orders/{order_id}/cancel",
        "required": {"order_id", "reason"},
    },
}

@dataclass
class Verdict:
    trace_id: str
    schema_ok: bool
    args_ok: bool
    under_budget: bool
    passed: bool
    why: str

def load_trace(path: Path) -> dict[str, Any]:
    data = json.loads(path.read_text())
    for key in ("id", "tool", "arguments", "injected_latency_ms"):
        if key not in data:
            raise ValueError(f"{path} missing {key}")
    return data

def grade(trace: dict[str, Any], budget_ms: int = BUDGET_MS) -> Verdict:
    tool = trace["tool"]
    args = trace["arguments"]
    latency = int(trace["injected_latency_ms"])
    spec = ALLOWED_TOOLS.get(tool)
    if spec is None:
        return Verdict(trace["id"], False, False, latency <= budget_ms, False, "unknown_tool")
    missing = spec["required"] - set(args)
    extra_empty = any(args.get(k) in (None, "") for k in spec["required"])
    args_ok = not missing and not extra_empty
    schema_ok = isinstance(args, dict)
    under = latency <= budget_ms
    if not schema_ok:
        why = "args_not_object"
    elif not args_ok:
        why = f"bad_args:{sorted(missing) or 'empty'}"
    elif not under:
        why = f"over_budget:{latency}ms"
    else:
        why = "ok"
    return Verdict(trace["id"], schema_ok, args_ok, under, schema_ok and args_ok and under, why)

def grade_dir(folder: str) -> list[Verdict]:
    paths = sorted(Path(folder).glob("*.json"))
    return [grade(load_trace(p)) for p in paths]

if __name__ == "__main__":
    import sys
    rows = grade_dir(sys.argv[1] if len(sys.argv) > 1 else "traces")
    passed = sum(1 for r in rows if r.passed)
    schema_only = sum(1 for r in rows if r.schema_ok and r.args_ok)
    print(f"traces={len(rows)} schema_only={schema_only}/{len(rows)} budget_aware={passed}/{len(rows)}")
    for r in rows:
        flag = "PASS" if r.passed else "FAIL"
        print(f"{flag} {r.trace_id:12} schema={int(r.schema_ok)} args={int(r.args_ok)} budget={int(r.under_budget)} {r.why}")

That is the whole argument in a file. Schema and arguments can still win. The clock can still veto. I am tired of dashboards that never give the clock a vote.

The planted pack is boring on purpose. Eight traces are clean and fast. Four call a tool I never offered. Four omit reason on cancel. Four are perfect JSON aimed at an endpoint I delayed to 1200ms. Four mix a valid cancel with 400ms, which should pass, because I do not want a scorer that fails everything just to look strict. Strict and sad are not the same skill.

Drop them in traces/ like this.

{
  "id": "t-over-budget-03",
  "tool": "get_order",
  "arguments": {"order_id": "ord_19"},
  "injected_latency_ms": 1200
}

Run it locally. No GPU. No mystery SaaS.

python toolcall_budget_eval.py traces

On this pack the script prints traces=24 schema_only=20/24 budget_aware=12/24. Those eight extra “wins” under schema-only scoring are the delayed GETs. They look like competence. They are a 504 with etiquette. If you only count JSON, you will brag about 83 percent. If you count the wire, you are at 50 percent, and you should be. The missing 33 points were never intelligence. They were patience you do not have in production.

Is 800ms sacred. No. I picked it because it is mean enough to catch a frozen tool loop and loose enough that a healthy p95 still lives. Change the constant. The point is that a number exists. An eval without a number is a vibe.

I parked the judge next to the traces instead of next to a GPU I do not want to babysit. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is the only reason I mention it: the scorer is local Python, and the free server is a convenient place to run the same command when my laptop is already compiling something noisier. I am not going to invent model names, token ceilings, or hardware I did not measure. If your workspace already has that free model and free server path, point it at traces/ and keep the budget in git. If it does not, the file above still runs.

Where does this break. Everywhere you hoped it would not. Injected latency is not real latency. A fixture cannot reproduce jitter, cold starts, or the way a gateway starts lying after 200 concurrent cancels. The scorer treats one call as the world. Agents retry. Retries multiply the budget you thought you had. I did not model that here on purpose. If I hide a retry loop inside “helpfulness,” we are back to theater, just with more curtains.

It also breaks when the tool schema is a lie. If get_order is documented as 200ms and the real handler does a sequential billing lookup, your budget is fan fiction. Grade the contract you actually ship, not the README. And it breaks when you let the same model write the traces and grade them. That is a student marking their own handwriting. Keep the pack human-authored, or at least frozen before the run.

Should you use this if you are demoing a chatbot that never leaves the browser. Maybe not. A UI toy can survive a slow tool call with a spinner and a shrug. Should you use this if your “API” is an in-process function with no network. No. You would be timing a function call and calling it operations. Should you use this as a model leaderboard. Please do not. Twenty-four planted traces are a unit test for your grading story, not a census of intelligence.

Use it when a tool call implies HTTP, when HTTP implies a budget, and when someone in the room is about to present schema pass-rate as quality. Ask them what happened to the calls that would have waited. If they cannot answer, the chart is decoration.

I keep the budget in the same commit as the traces. That is the whole workflow. Change the number, change the pack, watch schema-only and budget-aware diverge. When they stop diverging, either your endpoint got honest or your fixtures got timid. One of those is a win. Guess which.

If you already have a MonkeyCode workspace with the free model access and free server option, run the scorer there and keep the traces in git. Steal the file either way. The clock does not need a product to be right.

I Counted Drops as Wrongs. The Chart Was Theater.

Jordan Liu — Mon, 21 Sep 2026 13:21:10 +0000

The first number on an eval dashboard is usually a lie. Not a scam. A folding error. You asked a model for an answer, the path blinked, and your scorer filed the blink under incorrect.

I stopped scoring completions until I scored the envelope.

If you cannot tell a drop from a wrong, you are not ranking models. You are ranking weather. Shared free endpoints make the weather louder: idle processes, truncated streams, empty choices that still ride in on HTTP 200. The status looks fine. The grade looks like failure. That is how a leaderboard turns into theater.

I wanted a harness that fails loudly at the transport layer before it whispers at the semantics layer. So I planted six failure modes, ran a classifier over twenty-four envelopes, and watched a naive pass-rate collapse on purpose. The code below is the experiment. The percentages are the fixture talking, not a vendor scoreboard.

The folding error

Most agent evals I see still do some version of success = (http == 200 and expected in text). Cute. Also how you launder a timeout into a dumb model.

Think of it like grading a take-home by weighing the envelope. A missing envelope is not a wrong proof. A torn envelope is not a wrong proof. A proof written in the wrong schema is closer, but it is still not the same as a proof that multiplies instead of factors. Mix them and you will fire the student who had a postal strike.

Free inference is the postal strike. I do not mean that as an insult. I mean that a path you are not paying for will drop, stall, and truncate in ways a dedicated box will not, and your scorer has to admit that or it will keep "proving" that cheaper models are worse at arithmetic when they were worse at staying on the line.

That is why I care about a free model path and a free server option at all. Not because I want an adjective. Because I want a hostile mailbox. If the harness survives a mailbox, it will not panic on a quiet one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Their free model access and free server option are the mailbox I have in mind when I say "hostile." I am not naming models, quoting quotas, or claiming uptime. If you cannot reproduce a claim from the fixture in this file, treat the claim as unset.

Six envelopes, one bit

I keep six labels, and I refuse to let them share a boolean. drop is a refused connection, a reset, a client timeout, a 429, a 503. empty is a 200 with no choices or a null content string. truncated is a finish_reason of length, or JSON that dies mid-key. schema is parseable JSON that misses a required field. wrong is valid JSON with the wrong answer. right is the boring one.

Ask yourself what your current CSV does with empty. If the answer is "fail," you already know why last week's ranking felt jumpy. The model did not change. The path coughed, and you graded the cough.

I did not need a live fleet to test the scorer. I needed lies I controlled. Twenty-four envelopes, four of each label, stuffed into memory. The naive scorer treats anything that is not right as a model miss. The envelope scorer reports two numbers: yield, meaning bodies that were even gradeable, and accuracy-on-yield, meaning rights over rights plus wrongs.

Here is the whole trick. Yield answers "did the path hand me an object?" Accuracy-on-yield answers "was the object correct?" If you publish only the product of those, you are back to theater.

The planted run

Save this as envelope_eval.py. It never calls a network. That is the point. If a scorer cannot recover a lie you planted on disk, it will not recover a lie a free server plants for you at 4 p.m.

#!/usr/bin/env python3
"""Planted-envelope scorer. Run: python envelope_eval.py"""
from __future__ import annotations

import json
from collections import Counter
from dataclasses import dataclass
from typing import Any, Optional

@dataclass
class Envelope:
    http_status: int
    elapsed_ms: float
    finish_reason: Optional[str]
    body: Any
    expected: dict

def classify(env: Envelope) -> str:
    if env.http_status in {0, 408, 429, 500, 502, 503, 504}:
        return "drop"
    if env.http_status != 200:
        return "drop"
    if not isinstance(env.body, dict):
        return "empty"
    choices = env.body.get("choices") or []
    if not choices:
        return "empty"
    content = (choices[0].get("message") or {}).get("content")
    reason = env.finish_reason or choices[0].get("finish_reason")
    if content in (None, ""):
        return "empty"
    if reason == "length":
        return "truncated"
    try:
        parsed = json.loads(content) if isinstance(content, str) else content
    except json.JSONDecodeError:
        return "truncated"
    if not isinstance(parsed, dict) or "answer" not in parsed:
        return "schema"
    if parsed.get("answer") != env.expected.get("answer"):
        return "wrong"
    return "right"

def plant() -> list[Envelope]:
    expected = {"answer": 42}
    good = {"choices": [{"message": {"content": '{"answer": 42}'}, "finish_reason": "stop"}]}
    bad = {"choices": [{"message": {"content": '{"answer": 7}'}, "finish_reason": "stop"}]}
    missing = {"choices": [{"message": {"content": '{"value": 42}'}, "finish_reason": "stop"}]}
    cut = {"choices": [{"message": {"content": '{"answ'}, "finish_reason": "length"}]}
    empty = {"choices": []}
    rows: list[Envelope] = []
    for _ in range(4):
        rows.append(Envelope(0, 30000, None, None, expected))
        rows.append(Envelope(200, 800, "stop", empty, expected))
        rows.append(Envelope(200, 1200, "length", cut, expected))
        rows.append(Envelope(200, 900, "stop", missing, expected))
        rows.append(Envelope(200, 700, "stop", bad, expected))
        rows.append(Envelope(200, 650, "stop", good, expected))
    return rows

def main() -> None:
    labels = [classify(env) for env in plant()]
    counts = Counter(labels)
    n = len(labels)
    naive = sum(lab == "right" for lab in labels) / n
    gradeable = counts["wrong"] + counts["right"]
    acc = counts["right"] / gradeable if gradeable else float("nan")
    print("counts:", dict(counts))
    print(f"naive_pass_rate={naive:.3f}")
    print(f"yield={gradeable / n:.3f}")
    print(f"accuracy_on_yield={acc:.3f}")

if __name__ == "__main__":
    main()

Run it.

python envelope_eval.py

You should get four of each label, naive_pass_rate=0.167, yield=0.333, accuracy_on_yield=0.500. That naive 16.7% is true and useless. It says the system is bad at the task. It is not. The system is 50% accurate on the eight gradeable semantic answers, and sixteen of twenty-four envelopes never earned a semantic grade. If I had shipped the naive number, I would have "proven" a model regression that was actually four drops, four empties, four truncations, and four schema misses I planted myself.

Would you publish 16.7% in a weekly review? I have. That is the part that still bothers me.

What I do with each label

I keep a small matrix next to the scorer so I cannot "just retry it" at 11 p.m. and call that science.

Label	Counts as model evidence?	Action
drop	no	retry with a budget, then park the row
empty	no	retry once, then park
truncated	no	raise max tokens or repair, cap the repairs
schema	no	repair or fail the harness, not the model
wrong	yes	score it, do not retry
right	yes	score it, do not retry

Retry only drops and empties. Repair only truncated and schema, and cap the repairs so you do not invent a second model with your JSON fixer. Score only wrong versus right. Never retry a wrong. Wrong is information. A retry on a wrong is how you accidentally bake temperature into a pass@1 and then argue with your own chart.

Where a free path helps, and where it breaks

I use a free model and a free server as a canary for the harness, not as a verdict on the product under test. The canary's job is to produce ugly envelopes on a weekday afternoon. If my classifier labels them, the paid run gets a cleaner CSV. If my classifier folds them, I am about to buy myself a fake regression.

It breaks in the obvious ways. A free path is not a latency SLA. It is not a promise that temperature 0 is deterministic. It is not a substitute for a held-out human grade. If your paper needs matched seeds across labs, do not use a shared free box as the official judge. If you are on a deadline and you cannot afford a thirty-second stall before a drop, this is the wrong mailbox. If you need to claim "the model scored 71%," this article will not give you that sentence, and neither should a canary.

I also do not use the same endpoint for candidate and judge when I can avoid it. Shared load couples their clocks. The judge gets slow, your client times out, and you file a drop as a disagreement. That loop is how evals invent opinions. Two mailboxes. One grade.

What I log now

Every row gets http_status, latency_ms, finish_reason, bytes, label, and only then answer. The live probe looks like this, and I keep --max-time boringly small so the client timeout is a fact instead of a hang.

curl -sS -D /tmp/hdr.txt --max-time 30 \
  -H "content-type: application/json" \
  --data @prompt.json \
  "$ENDPOINT/v1/chat/completions" | tee /tmp/body.json | \
  python -c "import json,sys; d=json.load(sys.stdin); c=d.get('choices') or []; print({'n_choices': len(c), 'finish': c[0].get('finish_reason') if c else None, 'bytes': len(json.dumps(d))})"
wc -c /tmp/body.json

If n_choices is 0, I do not regex the content for the golden string. There is no content. Stop pretending. HTTP 200 is a door. It is not a grade.

Who should ignore this? If you already isolate transport in your eval warehouse, you do not need my JSONL. If you are demoing a toy prompt in a notebook, you do not need six labels. If you are trying to make a free server look like a dedicated GPU, you are solving a press-release problem, and this workflow will not help you.

I still run the planted file before I point the same scorer at anything with a bill, or anything without one. Steal the classifier first. If you want a mailbox that will actually drop on you while you do it, MonkeyCode's free model access and free server option are one place to aim the canary. Read the labels. Do not publish the folded number.

I Subtracted the Queue. The Fast Model Wasn't.

Jordan Liu — Sun, 20 Sep 2026 11:26:58 +0000

The number on your latency dashboard is not a model score. It is a traffic report wearing a lab coat.

I keep watching people rank free endpoints by time-to-first-token as if TTFT were a property of weights. It isn't. Not when the box is shared. Not when your so-called eval is standing in the same line as everyone else's autocomplete. You are scoring the parking lot. Then you publish a leaderboard. Then you act shocked when Tuesday disagrees with Thursday.

What happens when the tests we use to measure models get slower than the models? The tests start measuring the hallway.

I wanted a splitter I could run twice and trust. Wait on one side. Generation on the other. If wait dominates, I throw the run out. If generation dominates, I keep it. That is the whole method. It is boring on purpose. Boring is how you stop lying to yourself.

The lab for this is not a secret GPU in a closet. MonkeyCode is an open source project with free model access and a free server option, which is exactly the shape of infrastructure that makes queue contamination loud. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not naming a model, a quota, a SKU, or a duration I cannot verify. I am naming a measurement error you can reproduce on any shared endpoint, including that one.

Picture the failure. You send the same prompt ten times. Naive wall clock swears run 3 was a genius and run 7 was a brick. Split the timestamps and run 3 sat in queue for a blink while run 7 waited for a bus. Tokens per second never moved. Your ranking did. Are you still averaging those ten numbers like they mean the same thing?

I treat three clocks as sacred. t_submit is the instant the client writes the last request byte. t_first is the instant the first response byte lands. t_done is the instant the stream closes. Wait is first minus submit. Work is done minus first. Contamination is wait divided by wait plus work. If that ratio is high, the run is weather, not intelligence.

You want code, not a sermon. Fine. This harness is the artifact. The fixture traces are labeled samples, not a product benchmark I pretended to run.

# queue_split.py — labeled fixture + splitter. Not a vendor scorecard.
from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class Trace:
    run_id: str
    t_submit: float
    t_first: float
    t_done: float
    prompt_hash: str
    streaming: bool
    cold: bool

    @property
    def wait_s(self) -> float:
        return max(0.0, self.t_first - self.t_submit)

    @property
    def work_s(self) -> float:
        return max(0.0, self.t_done - self.t_first)

    @property
    def wall_s(self) -> float:
        return max(0.0, self.t_done - self.t_submit)

    @property
    def contamination(self) -> float:
        total = self.wait_s + self.work_s
        return 0.0 if total == 0 else self.wait_s / total

    def verdict(self, discard_at: float = 0.6) -> str:
        if not self.streaming:
            return "UNSPLITTABLE"
        if self.contamination >= discard_at:
            return "DISCARD"
        if self.contamination >= 0.2:
            return "WORK_ONLY"
        return "WALL_OK"


# Illustrative traces only. Same prompt_hash on purpose.
FIXTURE = [
    Trace("clean", 0.00, 0.12, 1.40, "p1", True, False),   # short wait, steady work
    Trace("queued", 0.00, 2.10, 3.38, "p1", True, False),  # long wait, SAME work interval
    Trace("slow", 0.00, 0.15, 3.05, "p1", True, False),    # short wait, actually slow generate
    Trace("buffered", 0.00, 2.80, 2.81, "p1", False, True), # non-streaming: clocks collapse
]


def rank(traces, key):
    return [t.run_id for t in sorted(traces, key=key)]


if __name__ == "__main__":
    streamed = [t for t in FIXTURE if t.streaming]
    print("naive wall rank:", rank(streamed, lambda t: t.wall_s))
    print("work-only rank:", rank(streamed, lambda t: t.work_s))
    for t in FIXTURE:
        print(
            f"{t.run_id:9} wait={t.wait_s:.2f}s work={t.work_s:.2f}s "
            f"contam={t.contamination:.2f} -> {t.verdict()}"
        )

Run it locally. No credentials. No cloud required for the lie detector itself.

python queue_split.py

You should see the reversal immediately. Naive wall clock ranks clean, then slow, then queued. Work-only ranking puts clean and queued in the same neighborhood and leaves slow as the actual drag. That flip is the bug. buffered comes back UNSPLITTABLE because a one-shot body gives you a single thud, not two clocks. If your eval client does that, stop calling it TTFT. You do not have TTFT. You have a door slam.

I pipe real calls through the same dataclass. The HTTP client is deliberately ugly. Environment variables so this article does not invent a hostname, a model id, or a token budget.

# live_probe.py — unexecuted template. Fill env. Do not treat output as a brand score.
import hashlib, json, os, time, urllib.request
from queue_split import Trace

URL = os.environ["EVAL_URL"]          # your endpoint, not mine
BODY = os.environ.get("EVAL_BODY", '{"prompt":"ping","stream":true}')

def stream_once(run_id: str, cold: bool) -> Trace:
    payload = BODY.encode()
    req = urllib.request.Request(URL, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    t_submit = time.perf_counter()
    t_first = None
    with urllib.request.urlopen(req, timeout=120) as resp:
        while True:
            chunk = resp.read(64)
            now = time.perf_counter()
            if not chunk:
                break
            if t_first is None:
                t_first = now
        t_done = time.perf_counter()
    if t_first is None:
        t_first = t_done
    digest = hashlib.sha256(payload).hexdigest()[:12]
    streaming = b"stream" in payload and b"true" in payload.lower()
    return Trace(run_id, t_submit, t_first, t_done, digest, streaming, cold)

if __name__ == "__main__":
    rows = [stream_once(f"r{i}", cold=(i == 0)) for i in range(10)]
    print(json.dumps([r.__dict__ | {
        "wait_s": r.wait_s, "work_s": r.work_s,
        "contamination": r.contamination, "verdict": r.verdict(),
    } for r in rows], indent=2))

export EVAL_URL="http://127.0.0.1:8080/v1/stream"   # local stand-in
export EVAL_BODY='{"prompt":"ping","stream":true}'
python live_probe.py

Do I always get a streaming socket? No. If the free server buffers the whole completion, t_first and t_done collapse, and wait swallows work. Then the splitter cannot save you. That is a limitation, not a footnote. Non-streaming free servers make this method weaker, not stronger. If your loop expects one JSON blob, you are timing a black box. Split nothing. Say so out loud.

How do I decide keep versus discard? I do not worship a magic cutoff. I look at the batch. If contamination sits under about 0.2, I treat wall clock as mostly work. If it wanders from 0.2 to 0.6, I report work time and I print the wait column in the same breath so nobody can hide. If a run clears 0.6, I discard it for ranking. I still log it. Hidden discards are how people fake a calm night.

Is 0.6 science? No. It is a fence I can defend in a code review. You can move the fence. You cannot skip the column. Would you ship a unit test that asserts elapsed < 2.0 on a machine you do not own? Then why is your model eval doing that?

I also refuse to mix cold and warm. The first call after a pause is a different animal. I tag it. I do not average it with the next nine. People love n=10 until you ask which one was the kettle boiling. A free server after idle is a library that has to turn the lights on. That is not a regression in the book.

Where this holds together: identical prompts, streaming responses, a client you control, and a reason to compare two setups on the same night. Where it falls apart: tool-call agents whose "first token" is an opening brace, judges that treat latency as quality, batches that span a traffic cliff, and any protocol that cannot show you the first byte. A free server is a city street. Rush hour is not a model regression. Off-peak is not a breakthrough.

Should you use this if you are publishing a public model card? Only as a hygiene check, never as the headline metric. Should you use this if you are shopping dedicated hardware? No. Buy a stopwatch that is not shared. Should you use this if your eval is a chatbot screenshot and a vibe? Please stop. The method will not fix a missing clock.

I still like free inference. I like it the way I like a public library. Useful. Crowded. A terrible place to time how fast you read. An open source project plus a free server is a gift for iteration. It is a trap for stopwatch science. Hold both thoughts in the same head.

If you already have a slot on that free server, do not start with a blog graph. Start with one prompt, ten streamed calls, and the wait column. If wait moves and work does not, your old dashboard was theatre. That is the only invitation I have.

I Imported the Patch. That's When It Died.

Jordan Liu — Sat, 19 Sep 2026 09:31:28 +0000

I imported the patch. That's when it died.

ast.parse had already smiled. The chat had already called the work complete. Neither of those is a gate I trust. If your coding-model eval still stops at "looks like Python," you are grading handwriting, not software.

Why does this keep happening? Because a lot of us are still scoring the story of a fix. Fluency is cheap. Import is not. I wanted a number I could defend without recruiting a second model to compliment the first one.

So I built a three-gate harness: parse, import, test. Same fixture. Same patch. Three increasingly rude questions. The code below is the experiment. You can run it without a GPU and without believing a single adjective.

The fixture is small on purpose

I do not need a web app to catch a liar. I need a module that parses, then explodes when Python executes the top level, then fails a real assertion if you get past that. Parse-time green is a vanity metric. Import is the first honest gate.

# shop.py  — broken on purpose
from decimal import Decimal, ROUND_HALF_UP

TAX = None  # populated only if you actually import this file correctly

def require_tax_table():
    if TAX is None:
        raise RuntimeError("tax table was never loaded")
    return TAX

# Module-level landmine. ast.parse will not run this.
require_tax_table()

def add_tax(cents: int) -> int:
    rate = require_tax_table()
    money = Decimal(cents) * (Decimal("1") + rate)
    return int(money.to_integral_value(rounding=ROUND_HALF_UP))

# test_shop.py
from shop import add_tax

def test_add_tax_rounds_half_up():
    assert add_tax(100) == 108
    assert add_tax(1) == 1

See the trap? ast.parse never calls require_tax_table(). A model can rewrite add_tax with gorgeous syntax and still leave the import path haunted. That gap is what I wanted to count. Chat will not count it for you.

The intended fix is boring. Load a Decimal("0.08") tax table before anyone calls add_tax, keep the rounding, do not invent a helper that is not in the file. Boring is the point. I am not grading creativity. I am grading whether the module exists in a Python process.

I ask for a diff. I refuse to read it.

The prompt is hostile on purpose. If the model wants to narrate, it can narrate into a zero.

Return a unified diff for shop.py only.
Do not explain. Do not wrap the diff in markdown.
The file must import and test_add_tax_rounds_half_up must pass.
Tax rate is 8%. Use decimal rounding already in the file.

Then I do not read the completion. I extract. I apply. I score. If that sounds unfriendly, good. An oracle should be unfriendly.

import re, subprocess, sys, tempfile, shutil, ast
from pathlib import Path

DIFF_RE = re.compile(r"(?ms)^--- .*^\+\+\+ .*?^(?=\Z)", re.M)

def extract_unified_diff(text: str) -> str | None:
    if text.lstrip().startswith("--- "):
        return text
    blocks = re.findall(r"```

(?:diff|patch)?\n(.*?)

```", text, re.S)
    for b in blocks:
        if b.lstrip().startswith("--- "):
            return b
    return None  # essays get nothing. no pity parse.

If the completion is a memoir with a fenced snippet in the middle, and that snippet is not a unified diff, the extractor returns None. Score zero at gate zero. No partial credit for confidence. I used to "helpfully" take the first code fence. That is how you score markdown as Python and then convince yourself the model improved.

Applying the patch is equally unromantic:

def apply_diff(src: Path, diff: str) -> Path | None:
    tmp = Path(tempfile.mkdtemp())
    shutil.copytree(src.parent, tmp, dirs_exist_ok=True)
    patch = tmp / "change.diff"
    patch.write_text(diff)
    r = subprocess.run(
        ["patch", "-p1", "--batch", "-d", str(tmp)],
        input=diff, text=True, capture_output=True,
    )
    return tmp if r.returncode == 0 else None

Need patch on the machine. That is a feature. If your eval loop cannot survive a 1970s Unix tool, it is not ready to survive a model that emits @@ headers for files you never asked it to touch.

Three gates, one script

def gate_parse(py: Path) -> bool:
    try:
        ast.parse(py.read_text())
        return True
    except SyntaxError:
        return False

def gate_import(py: Path) -> bool:
    r = subprocess.run(
        [sys.executable, "-c", f"import sys; sys.path.insert(0, {py.parent.as_posix()!r}); import shop"],
        capture_output=True, text=True,
    )
    return r.returncode == 0

def gate_test(root: Path) -> bool:
    r = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", str(root / "test_shop.py")],
        capture_output=True, text=True,
    )
    return r.returncode == 0

def score(completion: str, fixture: Path) -> dict:
    diff = extract_unified_diff(completion)
    if not diff:
        return {"extract": False, "parse": False, "import": False, "test": False}
    root = apply_diff(fixture, diff)
    if root is None:
        return {"extract": True, "parse": False, "import": False, "test": False}
    py = root / "shop.py"
    parsed = gate_parse(py)
    imported = parsed and gate_import(py)
    tested = imported and gate_test(root)
    return {"extract": True, "parse": parsed, "import": imported, "test": tested}

I run it against checked-in completions, not against a vibe:

python score_patch.py fixtures/essay.txt
python score_patch.py fixtures/parse_only.txt
python score_patch.py fixtures/import_ok_test_fail.txt
python score_patch.py fixtures/pass.txt

Those four files are oracles for the scorer. They are not a vendor bake-off. If my grader cannot separate them, I do not get to grade a model yet. The labeled output below is from those fixtures, which you can paste yourself. It is not a claim about any hosted model.

# fixture-driven scorer output (not a product benchmark)
essay                 extract=0 parse=0 import=0 test=0
parse_only            extract=1 parse=1 import=0 test=0
import_ok_test_fail   extract=1 parse=1 import=1 test=0
pass                  extract=1 parse=1 import=1 test=1

parse_only is the one that should make you nervous. It is valid syntax. It still cannot be imported. An LLM judge will often call it solid. ast.parse will call it fine. Python will not.

What the gates actually argue about

Parse-only completions are empty calories. They look like engineering. They are essays wearing parentheses. I used to stop there because it felt quantitative. It was not.

Import failures are more interesting. The model invents load_rates(), or it leaves a name it saw in a comment, or it deletes the landmine and forgets to put a tax table back. Fluency stays high. The process dies before pytest exists. That is not a "almost." That is a format error with extra steps.

Test failures after a clean import are the only failures I treat as the model having engaged the problem. Everything before that is packaging. I blurred those categories for months. That was on me, not on the model.

Want a one-liner while you iterate on the extractor?

python -c "import ast,pathlib; ast.parse(pathlib.Path('shop.py').read_text()); print('parsed')"
python -c "import shop" || echo "imported? no"
python -m pytest -q test_shop.py

Three commands. Three different truths. If your dashboard collapses them into one thumbs-up, your dashboard is lying with better typography.

Where a cheap generation loop helps, and where it breaks

I still need something to emit candidate diffs. Burning a paid endpoint to discover that a pile of samples never import is how you light money on fire. The mechanical grader is the expensive-looking part that is actually free. Generation is the part that tempts you to skip gates because "it read well."

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode only as a generation pipe: free model access and a free server option, so the loop can throw diffs at this harness without me standing up inference. The scorer does not call it. Delete the pipe and the gates still work. That is the only honest integration I can defend.

Where that setup performs well: volume. You can park messy completions against parse/import/test until the failure taxonomy stops moving. You are paying in attention, not in GPU mythology. Where it breaks: the free path will not save a weak fixture. If your test is assert True, import becomes theater again. Unstructured completions wrap diffs in sermons; if you relax the extractor, you will grade the sermon. Truncation dies at parse or import, which is good, unless you mark those as "needs review" instead of fail. Do not do that. Fail is a complete sentence.

Limitations, out loud

This harness does not measure taste. It will pass a brutal one-liner that satisfies test_add_tax. It will fail a correct redesign that renames the function. It is hostile to refactors. That is a bias, not a virtue. It also will not catch a wrong tax rate if you never asserted 108. Weak tests make import look like enlightenment.

Do not use this if you are evaluating docs, architecture notes, or UI copy. Do not use it as a hiring rubric. Do not use it to stamp "production ready" on a model because gate three went green on one file. If you need semantic review, add a linter, then a type checker, then stop before you add another LLM as judge. The last time I let a model grade a model, pretty prose still moved the score.

I also do not claim these gates outlive a moving product surface. Free access and a free server are operator-supplied availability notes, not a promise about models, hardware, duration, or quota. If the pipe changes, the scorer should not care. If the scorer cares, you coupled the wrong things.

The ranking that flipped

Once I ignored chat, my personal ranking of "this completion looks good" collapsed. The essay I would have merged in a hurry scored zeros. The ugly diff that never said "sure" imported and passed. That is the whole lesson, and it does not require a leaderboard.

So I am done asking whether AI outgrew our tests. I ask whether our tests ever loaded the code. Most days they did not. Parse is polite. Import is not. I grade the rude one.

I Shuffled the Judge Prompt. The Scores Should Have Held.

Jordan Liu — Fri, 18 Sep 2026 07:52:14 +0000

Your eval is not a measurement if the same trace gets a different grade when you move it down the list. I stopped arguing about model quality and started measuring judge flip-rate. That number is uglier than a single accuracy score, and it is the one I actually ship against.

We keep building agents and then grading them with another model, usually the cheapest one that will answer JSON on a Tuesday. Does that sound like engineering, or like asking a tired intern to re-grade the same stack of papers after someone shuffled the pile? I wanted a protocol that fails loudly when the intern is guessing.

The last week of DEV commentary kept circling a real itch: we outgrew the tests we brag about. I am not here to litigate vibe coding. I am here because my scoring harness was agreeing with itself until I changed the order of the fixtures. Order should be a no-op. It was not.

So I wrote a judge that I can lie to on purpose. If a stub that keys off list position can look “consistent” in a naive run, a free model can too. The trick is to run the same traces through the same rubric more than once, with the deck cut differently each time, and count the flips.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I needed a lot of cheap, boring repeats—same rubric, same traces, different order. MonkeyCode’s free model access and free server option are relevant for that grind if you treat the endpoint as a judge, not as a coauthor. I am not claiming a quota, a model name, or a latency number I did not measure. The interesting part is the flip-rate script.

Here is the fixture set I actually version. Three traces, labeled in git, not in my memory. One is a clean tool success. One is a fluent wrong answer. One is a partial apply that talks like a merge.

# fixtures.py — canned traces, not production logs
FIXTURES = [
    {
        "id": "t_ok",
        "task": "Rename getUser to get_user in src/api.py and keep tests green.",
        "trace": "Applied patch to src/api.py. Ran pytest: 12 passed.",
        "gold": "pass",
    },
    {
        "id": "t_fluent_fail",
        "task": "Rename getUser to get_user in src/api.py and keep tests green.",
        "trace": "Refactored naming across the service layer. Ready to ship.",
        "gold": "fail",
    },
    {
        "id": "t_partial",
        "task": "Rename getUser to get_user in src/api.py and keep tests green.",
        "trace": "Updated the function. Tests still import getUser. Explained why that is fine.",
        "gold": "fail",
    },
]

Gold labels live next to the traces. If I let the model invent the labels, I am scoring a mirror. You already know how that movie ends, right?

The rubric is short on purpose. Long rubrics make judges sound careful while they still wobble. I ask for a label and a one-line reason, then I throw the reason away for the metric. Reasons are for humans. Flip-rate is for the harness.

RUBRIC = """Score the agent trace against the task.
Return JSON only: {"id": str, "label": "pass" or "fail", "reason": str}.
pass = the trace shows the requested change and a test run that could have caught a miss.
fail = missing change, missing tests, or a story that substitutes for evidence.
"""

Now the part people skip. I do not call the judge once. I call it through permutations. Three fixtures is six orders. That is small enough to read and large enough to catch a positional toady.

import hashlib, itertools, json, os, urllib.request
from fixtures import FIXTURES, RUBRIC

ENDPOINT = os.environ["JUDGE_URL"]  # your free server or any OpenAI-shaped chat URL
MODEL = os.environ.get("JUDGE_MODEL", "default")

def chat(messages):
    body = json.dumps({"model": MODEL, "messages": messages, "temperature": 0}).encode()
    req = urllib.request.Request(
        ENDPOINT, data=body, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode())
    return data["choices"][0]["message"]["content"]

def parse_label(raw, fallback_id):
    try:
        obj = json.loads(raw)
        label = obj.get("label", "fail")
        return fallback_id, "pass" if label == "pass" else "fail"
    except Exception:
        return fallback_id, "fail"  # unparseable output is a fail, not a retry poem

Temperature zero is not a personality. It is a request. Free endpoints still drift. That is the point of repeating the trial, not a reason to skip it. If your client retries on parse failure, cap it. Infinite repair is how a judge learns to write JSON and forget the grade.

def judge_batch(order):
    blob = json.dumps([{"id": t["id"], "task": t["task"], "trace": t["trace"]} for t in order])
    raw = chat([
        {"role": "system", "content": RUBRIC},
        {"role": "user", "content": blob},
    ])
    # Expect one JSON object per trace. If the model merges them, every id is fail.
    labels = {}
    for t in order:
        labels[t["id"]] = "fail"
    try:
        parsed = json.loads(raw)
        rows = parsed if isinstance(parsed, list) else [parsed]
        for row in rows:
            i = row.get("id")
            if i in labels:
                labels[i] = "pass" if row.get("label") == "pass" else "fail"
    except Exception:
        pass
    return labels

def flip_rate(runs):
    ids = [t["id"] for t in FIXTURES]
    flips = 0
    pairs = 0
    for i in ids:
        labels = [run[i] for run in runs]
        pairs += 1
        if len(set(labels)) > 1:
            flips += 1
    return flips / pairs

I also keep a dishonest judge in the repo. It is a flashlight. If I wire the pipeline to a function that passes whatever arrived last, the flip-rate should scream. If it does not scream, I am measuring the wrong thing.

def positional_stub(order):
    # Last item always "pass". A real metric must punish this.
    labels = {t["id"]: "fail" for t in order}
    labels[order[-1]["id"]] = "pass"
    return labels

def identity_stub(order):
    # Stable on id. Flip-rate must be 0.0 even when shuffled.
    return {t["id"]: ("pass" if t["id"] == "t_ok" else "fail") for t in order}

if __name__ == "__main__":
    orders = list(itertools.permutations(FIXTURES))
    stub_flips = flip_rate([positional_stub(list(o)) for o in orders])
    id_flips = flip_rate([identity_stub(list(o)) for o in orders])
    print(f"positional_stub flip_rate={stub_flips:.2f}")  # expect 1.00
    print(f"identity_stub flip_rate={id_flips:.2f}")      # expect 0.00

Run that before you spend a single token. I mean it. positional_stub flip_rate=1.00 and identity_stub flip_rate=0.00 are the only “benchmarks” in this article that I will stand behind without a live endpoint, because they are functions of the fixtures. If those two lines move, your math is broken and you should not grade a model yet.

The live path is the same loop with judge_batch. I do not paste a leaderboard from a free endpoint I did not freeze. You should not either. Print the per-id label vectors. If t_fluent_fail is pass in two orders and fail in four, you do not have a score. You have a weather report.

export JUDGE_URL="http://127.0.0.1:8000/v1/chat/completions"
python -c "from fixtures import FIXTURES; print(len(FIXTURES), 'fixtures')"
python judge_flip.py

What does “good enough” look like for a cheap judge? I use a boring gate, not a press release. Flip-rate on this three-trace deck must be 0.00 across all six orders. Agreement with gold must be 3/3 on every order, not on average. Averages hide the fluent miss, and the fluent miss is the whole crime.

Think of it like a kitchen scale. If the same bag of flour reads 500g, then 470g, then 510g because you rotated the bowl, you do not publish a recipe. You fix the scale. Free inference is useful here because repeats are the experiment. One beautiful grade is a vibe. Six ugly grades are data.

Where it breaks is predictable if you have ever watched a model try to be helpful. Batch JSON collapses into one object. The last trace inherits a pass because the prose sounded finished. The rubric gets paraphrased in the reason field and ignored in the label. A free server that is busy will timeout, and if you retry without a bound you will train yourself to accept the first parseable sentence. Bound the retries. I cap parse repair at one. After that it is fail. Silence is a grade.

There is a second failure I only caught when I hashed the prompt. I had a comment in the user blob that said “expected fail.” The judge, being a good student, agreed. Strip your gold from the request. If the model can see the answer, you are not evaluating an agent. You are evaluating obedience.

def prompt_fingerprint(order):
    body = json.dumps([t["id"] for t in order]) + RUBRIC
    return hashlib.sha256(body.encode()).hexdigest()[:12]

I log that fingerprint next to each run. When someone “improves” the rubric on a Friday, the hash moves and the old flip-rate is void. That is the whole discipline. Git the rubric. Git the fixtures. Do not git a screenshot of a chat.

Limitations, because this protocol is easy to oversell. Three traces will not bless a product. Temperature zero is not determinism. A free server is not a frozen judge checkpoint. If you need a number for a paper, a customer report, or a safety case, stop. Hire a stable judge, pin the revision, and put humans on the fluent failures. This workflow is for catching a harness that lies to its owner.

Who should not use it? Anyone averaging labels across shuffles and calling the mean “accuracy.” Anyone whose agent can mutate the fixtures. Anyone hiding the endpoint behind a retry loop that also rewrites the rubric. And anyone who thinks a free tier is a reason to skip gold labels. Free means you can afford the repeats. It does not mean the repeats are optional.

I still want cheap judges. I want them for the same reason I want cheap unit tests: I will actually run them. If you already have a free model endpoint or a free server sitting around, point JUDGE_URL at it and make the stubs fail first. The CTA is the script, not a tour. If the flip-rate is not zero, you do not have evaluation. You have a conversation that learned to export CSV.

I Scored Patches That Never Applied

Jordan Liu — Thu, 17 Sep 2026 05:53:19 +0000

A free model that emits a unified diff is not a code change. It is a hypothesis with nice lighting. I score the hypothesis against git apply --check and a compile command before I let anyone call it engineering.

Why this gate? Because the last seven days of AI talk keep collapsing two different jobs into one sentence. Generating text that resembles a fix is cheap. Landing a fix that the tree will accept is still work. If your review process stops at “looks right,” you are scoring theater.

I learned this the loud way. A patch can be fluent, confident, even polite, and still fail to apply. Hunk context drifts. A file got renamed last Tuesday. The model “fixed” a helper that no longer exists. The comment said done. The working tree said otherwise. Sound familiar?

So I stopped scoring vibes. I score apply. Then I score compile. Everything else is commentary.

The unit under test is the patch, not the paragraph

Think of a model output like a shipping label on an empty box. The label can be perfect. The box can still be empty. git apply is the person at the dock who actually opens it.

I keep a tiny fixture repo and a scorer. The model — any model — has one job: return a unified diff against a known commit. My job is not to admire the diff. My job is to ask four questions the model cannot grade for itself. Did the diff parse? Did it apply cleanly? Did the project still compile? Did the patch touch files I never named?

That last one matters more than people admit. Free inference is generous with extra opinions. A one-line request becomes a courtesy refactor in a file you did not open. The blast is quiet until CI yells.

I run the loop on a throwaway worktree so a bad hunk cannot vandalize the branch I actually care about. Cheap isolation. Honest scores.

A scoring harness you can run

This is a method, not a trophy. The numbers below come from the fixtures in this article, not from a secret leaderboard and not from a production bill. If you paste this into a real repo, your scores will move. That is the point.

#!/usr/bin/env python3
"""Score a unified diff as a patch, not as prose.

Labeled example: fixtures only. Do not treat printed totals as product benchmarks.
"""
from __future__ import annotations

import subprocess
from dataclasses import dataclass
from pathlib import Path

@dataclass
class PatchScore:
    parsed: bool
    apply_ok: bool
    compile_ok: bool | None
    files_touched: int
    unsolicited_files: int
    hunks: int
    notes: str

    def shipping_unit(self) -> bool:
        compile_pass = True if self.compile_ok is None else self.compile_ok
        return self.parsed and self.apply_ok and compile_pass and self.unsolicited_files == 0


def parse_diff_files(diff: str) -> list[str]:
    files: list[str] = []
    for line in diff.splitlines():
        if line.startswith("+++ b/") and not line.startswith("+++ b/dev/null"):
            files.append(line[6:])
    return files


def count_hunks(diff: str) -> int:
    return sum(1 for line in diff.splitlines() if line.startswith("@@"))


def score_patch(
    repo: Path,
    diff: str,
    allowed_files: set[str],
    compile_cmd: list[str] | None = None,
) -> PatchScore:
    files = parse_diff_files(diff)
    parsed = bool(files) and "diff --git" in diff
    unsolicited = [f for f in files if f not in allowed_files]

    apply = subprocess.run(
        ["git", "apply", "--check", "--whitespace=nowarn", "-"],
        cwd=repo,
        input=diff.encode(),
        capture_output=True,
    )
    apply_ok = parsed and apply.returncode == 0

    compile_ok: bool | None = None
    if apply_ok and compile_cmd:
        # Apply in a detached worktree in real use. Fixture path shown for brevity.
        compile = subprocess.run(compile_cmd, cwd=repo, capture_output=True)
        compile_ok = compile.returncode == 0

    notes = apply.stderr.decode()[:300] if apply.returncode else "apply-check clean"
    return PatchScore(
        parsed=parsed,
        apply_ok=apply_ok,
        compile_ok=compile_ok,
        files_touched=len(set(files)),
        unsolicited_files=len(set(unsolicited)),
        hunks=count_hunks(diff),
        notes=notes.strip() or "ok",
    )

I keep two fixture diffs next to the scorer. One is a surgical edit. One is a confident mess. You need both. A harness that only sees success is a participation trophy with a shebang.

# fixtures/good.patch  — allowed file: src/greet.py
diff --git a/src/greet.py b/src/greet.py
--- a/src/greet.py
+++ b/src/greet.py
@@ -1,3 +1,3 @@
 def greet(name: str) -> str:
-    return "hi " + name
+    return f"hi {name}"

# fixtures/bad.patch  — claims src/greet.py, also rewrites README.md
diff --git a/src/greet.py b/src/greet.py
--- a/src/greet.py
+++ b/src/greet.py
@@ -1,3 +1,4 @@
 def greet(name: str) -> str:
-    return "hi " + name
+    return f"hello {name.strip()}"
+    # model leftover
diff --git a/README.md b/README.md
--- a/README.md
+++ b/README.md
@@ -1,2 +1,2 @@
-# demo
+# totally rewritten by accident

Wire a one-liner so you can refuse to argue with the model in Slack:

python score_patch.py --repo ./fixture-repo --diff fixtures/good.patch \
  --allow src/greet.py --compile "python -m compileall src"
python score_patch.py --repo ./fixture-repo --diff fixtures/bad.patch \
  --allow src/greet.py --compile "python -m compileall src"

On those fixtures the scorer prints a boring, useful object. Good patch: parsed, apply_ok, compile_ok, one file, zero unsolicited, shipping_unit true. Bad patch: it may still parse. It may even apply. It still fails the shipping test because it rewrote a file nobody asked for, and the leftover comment is a compile-time shrug waiting to happen. That is not a model ranking. That is a gate.

Would you merge the second one because the greeting string got fancier? I would not. Fancy is not a shipping unit.

Where free inference belongs in this loop

I do not need a heroic model to generate the first diff. I need an endpoint I can afford to retry while the scorer stays strict. Disclosure: This article was prepared as part of MonkeyCode's product outreach. When I want that cheap first pass, I point the same harness at MonkeyCode, which is an open-source project that currently offers free model access (the project states a ten-million-token allowance) and a free server option. I treat those as availability claims, not as a quality certificate. The scorer does not care who spoke. It cares whether git apply --check returned zero.

The useful pattern is blunt. Draft on the free endpoint. Score locally. If apply fails, send the stderr back as the next prompt — not a new essay, the actual reject text. If apply works and compile fails, send the compiler output, not your feelings. If an unsolicited file shows up, drop the patch on the floor. Do not negotiate with a courtesy refactor.

That loop is why free inference is interesting here and why it is dangerous without the gate. You can burn a lot of tokens producing patches that never become commits. Busy looks like progress. Apply is progress.

A decision I actually use, written as prose because a spreadsheet will not sit in your code review: if the change is a single allowed file, a compile command exists, and unsolicited files are a hard fail, free inference is a reasonable drafter. If the change spans generated code, lockfiles, or anything with a secret, I do not send it to a free endpoint at all. If there is no compile or typecheck command, I do not pretend the score is complete. I mark compile_ok as unknown and I refuse the shipping_unit shortcut. Unknown is not green.

What this does not prove

This harness does not measure taste. A patch can apply, compile, and still be the wrong design. It does not measure latency tails, retry storms, or whether the model called the same tool twice. I have scored those problems elsewhere. Different failure, different gate.

It also does not make free inference permanent, fast, or fair. Quotas change. Servers get busy. A free endpoint can be fine on Tuesday and weird on Thursday. That is why the score lives in git, next to the fixture, not in a screenshot of a chat window.

Do not use this approach if you need a contractual SLA, if your diff includes credentials, or if you were hoping the model would replace review. It will not. If your tests cannot fail, this gate will not save you either. Apply-and-compile is necessary. It is not sufficient. Anyone shipping regulated code from an unaudited free endpoint should stop and pick a different workflow.

The trend I keep hearing is that models already write better code than most of us. Maybe. Ask a meaner question. Better than whom, under which gate, against which tree? A paragraph that would impress a timeline can still lose to git apply --check. That command does not care about the debate. It cares about context lines.

I will keep drafting on cheap inference when the blast radius is small. I will keep scoring the patch, not the promise. If you already keep a compile gate, run the same harness against a free endpoint and see whether “done” survives first contact with the tree. MonkeyCode’s free models and free server are one place to try that loop. The score file is the part I would keep even if the endpoint changed names tomorrow.

I Scored Tests That Could Not Fail

Jordan Liu — Wed, 16 Sep 2026 04:08:05 +0000

Green is cheap when failure is unreachable. That is the whole finding. I stopped treating a generated "all tests passed" line as evidence until I could score whether a red result was even in the reachable set.

The timeline is loud again. Is AI already better at coding than most of us? Depends which benchmark someone is defending this week. I have a smaller question, and it actually ships. When a suite prints passed, was a real failure even possible?

Vibe coding is not the interesting insult. Calling a costume suite engineering is. A test that cannot go red is not a test. It is interior decorating for a pull request.

I wanted a number I could recompute on Monday. Not a vibe. A classifier over the test file plus a pytest collect, run on fixtures I can check into git. If the scorer and the labels disagree, the scorer is wrong. That is the contract. Cute narratives do not get a vote.

The failure I kept missing

I used to read the summary first. Passed: 8. Failed: 0. Skipped: 0. Then I would skim the test names like they were a changelog. That is how you get fooled. The model did not lie about pytest. Pytest told the truth about a suite that never asked a hard question.

Think of a smoke alarm that is not wired to the battery. It is green. You can screenshot it. You still do not have a fire system. Vacuous tests are that alarm. assert True is the obvious joke. The expensive ones are quieter. No assertion at all. A patch that replaces the function under test and then "proves" the return value. A function named check_add that pytest will never collect. assert add(2, 2) == add(2, 2), which stays green even when add is a dumpster fire, so long as it is a deterministic dumpster fire.

Would you accept that suite from a junior hire? Then why accept it from a model?

A tiny subject under test

I do not start with a real service. I start with something I can hold in one screen. If the scorer cannot be honest here, it will not be honest in your monorepo either.

# sut.py
def add(a: int, b: int) -> int:
    return a + b


def clamp(n: int, lo: int, hi: int) -> int:
    if lo > hi:
        raise ValueError("lo > hi")
    return max(lo, min(n, hi))

Eight fixtures live next to it. I labeled them before I ran anything. The labels are the experiment, not a memoir.

# fixtures/test_cases.py
import pytest
from unittest.mock import patch
from sut import add, clamp


def test_add_binding():
    assert add(2, 2) == 4
    assert clamp(11, 0, 10) == 10


def test_add_true():
    add(2, 2)
    assert True


def test_add_no_assert():
    add(2, 2)
    clamp(3, 0, 10)


def test_add_mocked():
    with patch("sut.add", return_value=4):
        from sut import add as mocked_add
        assert mocked_add(2, 2) == 4


def check_add_uncollected():
    assert add(1, 1) == 2


def test_always_skip():
    pytest.skip("not today")
    assert add(2, 2) == 4


def test_add_not_none():
    assert add(2, 2) is not None


def test_add_tautology():
    assert add(2, 2) == add(2, 2)

Binding means a wrong add or a wrong clamp can make pytest red. Everything else is a costume with a different tailor.

The scorer, not the speech

I score files, not vibes. AST first. Collection second. The model does not get to grade its own homework. If that sounds unkind, good. Kindness is how vacuous tests survive.

# vacuous_score.py
from __future__ import annotations

import ast
import json
import subprocess
import sys
from pathlib import Path

WEAK_OPS = {ast.Is, ast.IsNot, ast.NotEq}
SKIP_NAMES = {"skip", "xfail", "importorskip"}


class TestFn:
    def __init__(self, name: str, node: ast.FunctionDef):
        self.name = name
        self.node = node
        self.asserts = []
        self.skipped = False
        self.patches_sut = False
        self.calls_sut = False


class Visitor(ast.NodeVisitor):
    def __init__(self, sut_names: set[str]):
        self.sut_names = sut_names
        self.tests: list[TestFn] = []
        self._cur: TestFn | None = None

    def visit_FunctionDef(self, node: ast.FunctionDef):
        prev, self._cur = self._cur, TestFn(node.name, node)
        self.generic_visit(node)
        self.tests.append(self._cur)
        self._cur = prev

    def visit_Assert(self, node: ast.Assert):
        if self._cur:
            self._cur.asserts.append(node)
        self.generic_visit(node)

    def visit_Call(self, node: ast.Call):
        if not self._cur:
            return self.generic_visit(node)
        fn = node.func
        name = fn.attr if isinstance(fn, ast.Attribute) else getattr(fn, "id", "")
        if name in SKIP_NAMES:
            self._cur.skipped = True
        if name == "patch":
            if node.args and isinstance(node.args[0], ast.Constant):
                val = str(node.args[0].value)
                if any(s in val for s in self.sut_names):
                    self._cur.patches_sut = True
        if name in self.sut_names:
            self._cur.calls_sut = True
        self.generic_visit(node)


def _is_true_const(n: ast.AST) -> bool:
    return isinstance(n, ast.Constant) and n.value is True


def _is_tautology(n: ast.AST) -> bool:
    return (
        isinstance(n, ast.Compare)
        and len(n.comparators) == 1
        and ast.dump(n.left, annotate_fields=False)
        == ast.dump(n.comparators[0], annotate_fields=False)
    )


def _is_weak(n: ast.AST) -> bool:
    if not isinstance(n, ast.Compare):
        return False
    if any(isinstance(op, (ast.Is, ast.IsNot)) for op in n.ops):
        return True
    return False


def classify(t: TestFn) -> str:
    if not t.name.startswith("test_"):
        return "uncollected"
    if t.skipped:
        return "skipped"
    if t.patches_sut:
        return "mocked_sut"
    if not t.asserts:
        return "no_oracle"
    tests = [a.test for a in t.asserts]
    if tests and all(_is_true_const(x) for x in tests):
        return "vacuous"
    if tests and all(_is_tautology(x) for x in tests):
        return "tautology"
    if tests and all(_is_weak(x) for x in tests):
        return "weak_oracle"
    if t.calls_sut:
        return "binding"
    return "no_oracle"


def collect_names(path: Path) -> set[str]:
    r = subprocess.run(
        [sys.executable, "-m", "pytest", str(path), "--collect-only", "-q"],
        capture_output=True,
        text=True,
        check=False,
    )
    names = set()
    for line in r.stdout.splitlines():
        if "::" in line:
            names.add(line.split("::")[-1].strip())
    return names


def score(path: Path, sut_names: set[str]) -> dict:
    tree = ast.parse(path.read_text())
    v = Visitor(sut_names)
    v.visit(tree)
    collected = collect_names(path)
    rows = []
    for t in v.tests:
        label = classify(t)
        if t.name.startswith("test_") and t.name not in collected and label != "skipped":
            label = "uncollected"
        rows.append({"name": t.name, "label": label})
    return {"file": str(path), "rows": rows}


if __name__ == "__main__":
    target = Path(sys.argv[1])
    print(json.dumps(score(target, {"add", "clamp"}), indent=2))

Run it like a build step, not like a blog claim.

python vacuous_score.py fixtures/test_cases.py
pytest fixtures/test_cases.py -q --tb=no

On these fixtures the scorer is designed to emit this matrix. I am not reporting a hidden production run. I am reporting the labels the fixtures were built to force, which is the only honest number I have in this article.

test_add_binding          binding
test_add_true             vacuous
test_add_no_assert        no_oracle
test_add_mocked           mocked_sut
check_add_uncollected     uncollected
test_always_skip          skipped
test_add_not_none         weak_oracle
test_add_tautology        tautology

Pytest on the same file can still look fine. Binding tests pass. Vacuous tests pass. The skip is a skip. The uncollected function is invisible. That is the trick. The runner is not the oracle for quality. It is the oracle for "did the collected nodes raise."

So I keep a gate that does not care about charm.

FAIL_LABELS = {"vacuous", "no_oracle", "mocked_sut", "uncollected", "tautology"}
WEAK_OK = 0  # I do not let weak oracles count as coverage


def gate(rows: list[dict]) -> int:
    labels = [r["label"] for r in rows]
    binding = labels.count("binding")
    weak = labels.count("weak_oracle")
    bad = sum(1 for x in labels if x in FAIL_LABELS)
    if binding < 1:
        return 2
    if bad:
        return 2
    if weak > WEAK_OK:
        return 1
    return 0

Exit 2 means I do not ship the suite. Exit 1 is a warning I still refuse in CI. Zero is not "correct code." Zero is "these tests could fail." That bar is low. It is still higher than a screenshot of green.

Where a free model and a free server actually help

I still let a model propose the first draft of tests. I do not let it propose the verdict. When I want that loop on a scratch box instead of my laptop, I use MonkeyCode's free model access and the free server option to generate a candidate suite and run pytest there. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The scorer stays local. It is boring Python. If you peel the product out of this article, the gate still works. That is how I know the article is not a brochure.

What is the free path good at? Drafting volume. Getting a file to exist. Giving pytest something to collect so I can score it the same day. What does it break on? It cannot tell you the tests are binding. It cannot promise the server is a reproduction of prod. I am not going to invent model names, quotas, or latency numbers I did not measure in this writeup. If a vendor card will not fit in git next to vacuous_score.py, it is not part of the experiment.

I treat the server like a microwave. Useful. Not a laboratory. If the generated file trips mocked_sut or uncollected, I throw the file back. I do not "fix" it by weakening the gate. Have you watched someone delete an assertion to get green. Same energy. Different autocomplete.

What this does not prove

This scorer is a tripwire, not mutation testing. It will miss clever vacuity. A test can call add and still only check a log side effect you never cared about. Parametrize, dynamic exec, and unittest-style methods need more visitors. Collection depends on pytest's default naming. If you generate tests in a helper that pytest never imports, you can still look clean.

It also does not prove sut.py is right. Binding tests can encode the same bug twice. assert add(2, 2) == 5 is binding and wrong. Different gate. I want that to fail in pytest, not in the vacuity classifier.

Who should not use this as a primary control? Anyone in payments, safety, or an audit trail that needs fault injection. Anyone who thinks a free generation loop is a substitute for a maintainer reading the assertions. Anyone scoring models by how confident the summary paragraph sounds. If you need kill scores, use mutation testing. If you need this article to invent a leaderboard, you are in the wrong tab.

I keep the rule small enough to remember. A suite is evidence only if a wrong function can make it scream. Everything else is costume jewelry on a passing build. Score the jewelry. Then maybe read the summary.

If you want a scratch box to draft a suite and immediately run this gate, MonkeyCode's free model access and free server option are enough to host the loop. Do not confuse that with a verdict. The verdict is the JSON.

Free Inference Has a Tail. I Probe It Before I Ship.

Jordan Liu — Mon, 14 Sep 2026 21:44:32 +0000

Everyone is arguing about whether agents are real. Meanwhile the thing that actually kills my runs is a queue. Free inference is not slow — its average is fine. Its worst 1% is where your run dies, and the average hides it perfectly.

So I stopped benchmarking endpoints by "does it answer nicely." I probe three numbers and one taxonomy before I let anything depend on a provider. The three numbers are time-to-first-token at p50, p90 and p99. The taxonomy is how it fails, because a 429 and a silent stall are not the same animal and should not share a retry policy.

Here is the whole harness. It is short on purpose — if you cannot read your probe in one sitting, you will not trust its output.

Start cheap: curl already tells you a lot

Before writing Python, spend twenty seconds on the blunt instrument. Ten identical requests, timed, no streaming:

for i in $(seq 1 10); do
  curl -sS -o /dev/null \
    -H "Authorization: Bearer $LLM_API_KEY" \
    -H 'Content-Type: application/json' \
    -w 'connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total}\n' \
    -d "{\"model\":\"$LLM_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"ok\"}],\"max_tokens\":4}" \
    "$LLM_BASE_URL/chat/completions"
done

If connect is stable and ttfb swings by 10x across ten requests, you already know something. You are not paying for compute, you are paying for position in somebody's queue.

The probe: tail, not average

This runs N streaming requests at concurrency CONC, records time-to-first-token per request, and classifies every failure instead of averaging it away.

# probe_tail.py — measure a streaming endpoint's tail, not its average.
import asyncio, json, os, time
import httpx

BASE  = os.environ["LLM_BASE_URL"].rstrip("/")
KEY   = os.environ["LLM_API_KEY"]
MODEL = os.environ["LLM_MODEL"]
CONC  = int(os.environ.get("CONC", "8"))
N     = int(os.environ.get("N", "40"))
TTFT_BUDGET = float(os.environ.get("TTFT_BUDGET", "10"))  # seconds
PROMPT = "Reply with exactly one word: ok"

async def one(client, sem, idx, rows):
    async with sem:
        row = {"i": idx, "status": None, "ttft": None, "total": None,
               "chunks": 0, "chars": 0, "usage": None, "err": None}
        t0 = time.perf_counter()
        try:
            async with client.stream(
                "POST", f"{BASE}/chat/completions",
                headers={"Authorization": f"Bearer {KEY}"},
                json={"model": MODEL, "stream": True, "max_tokens": 8,
                      "stream_options": {"include_usage": True},
                      "messages": [{"role": "user", "content": PROMPT}]},
            ) as r:
                row["status"] = r.status_code
                if r.status_code != 200:
                    row["err"] = f"http_{r.status_code}"
                else:
                    async for line in r.aiter_lines():
                        if not line.startswith("data:"):
                            continue
                        data = line[5:].strip()
                        if data == "[DONE]":
                            break
                        obj = json.loads(data)
                        if obj.get("usage"):
                            row["usage"] = obj["usage"]
                        text = (obj.get("choices") or [{}])[0].get("delta", {}).get("content") or ""
                        if text and row["ttft"] is None:
                            row["ttft"] = time.perf_counter() - t0
                        elif row["ttft"] is None and time.perf_counter() - t0 > TTFT_BUDGET:
                            row["err"] = "ttft_over_budget"   # connected, but nothing came
                            break
                        row["chunks"] += 1
                        row["chars"] += len(text)
        except Exception as e:
            row["err"] = type(e).__name__
        row["total"] = time.perf_counter() - t0
        if row["ttft"] is None and row["err"] is None:
            row["err"] = "no_tokens"
        rows.append(row)

def pct(xs, p):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(p / 100 * len(xs)))] if xs else None

async def main():
    sem, rows = asyncio.Semaphore(CONC), []
    limits = httpx.Limits(max_connections=CONC, max_keepalive_connections=CONC)
    async with httpx.AsyncClient(timeout=httpx.Timeout(60, connect=10), limits=limits) as c:
        t0 = time.perf_counter()
        await asyncio.gather(*(one(c, sem, i, rows) for i in range(N)))
        wall = time.perf_counter() - t0

    ok = [r for r in rows if r["err"] is None]
    ttft = [r["ttft"] for r in ok]
    report = {
        "n": N, "conc": CONC, "wall_s": round(wall, 2), "ok": len(ok),
        "ttft_p50": pct(ttft, 50), "ttft_p90": pct(ttft, 90), "ttft_p99": pct(ttft, 99),
        "tail_ratio": (pct(ttft, 99) / pct(ttft, 50)) if ttft and pct(ttft, 50) else None,
        "errors": {e: sum(1 for r in rows if r["err"] == e)
                   for e in {r["err"] for r in rows if r["err"]}},
    }
    print(json.dumps(report, indent=2))
    with open("probe_rows.jsonl", "w") as f:
        for r in rows:
            f.write(json.dumps(r) + "\n")

asyncio.run(main())

Run it at CONC=1, then CONC=4, then CONC=16. Same endpoint, same prompt, same minute. The shape of the report looks like this — and I want to be explicit that the block below is the structure, with placeholder values, not a measurement I am reporting:

{
  "n": 40, "conc": 8, "wall_s": 0.0, "ok": 0,
  "ttft_p50": null, "ttft_p90": null, "ttft_p99": null,
  "tail_ratio": null,
  "errors": {}
}

How to read it

tail_ratio is the number I actually act on. Under 3, the endpoint behaves like a service. Between 3 and 10, it behaves like a service with a bad neighbour. Above 10, you are not calling an API, you are entering a lottery, and every synchronous user-facing feature you build on it will feel broken one afternoon a week for reasons nobody can reproduce.

ttft_over_budget deserves its own paragraph, because it is the failure that fools people. The socket connects. The status is 200. Headers arrive. Then nothing, for fifteen seconds, until an intermediate proxy quietly drops you. Your average latency looks great because you never got a token to measure. That is why the probe records a missing TTFT as a failure instead of omitting the row. Silence is a result.

And no_tokens is the same lie in a different costume. A stream that opens and closes with zero content chunks returns valid-looking SSE and zero value. Counting it as success is how you end up with an agent that "worked" and produced an empty tool call three steps later.

Audit the token counter before you trust it

Free tiers are usually metered, so the meter is part of the evaluation. The usage block is the provider's number. Compute your own with the tokenizer for that model family, then compare per request:

# tokens_audit.py — compare provider-reported usage to a local count.
import json, tiktoken

enc = tiktoken.get_encoding("cl100k_base")   # swap for your model's tokenizer
for line in open("probe_rows.jsonl"):
    r = json.loads(line)
    if not r["usage"]:
        continue
    local_prompt = len(enc.encode("Reply with exactly one word: ok"))
    drift = r["usage"]["prompt_tokens"] - local_prompt
    print(r["i"], "provider=", r["usage"]["prompt_tokens"], "local=", local_prompt, "drift=", drift)

A consistent positive drift is normal — chat templates add tokens you cannot see. An inconsistent drift, or a usage block that never appears on streamed responses, means your budget math is fiction. Plan from the larger of the two numbers, always.

Where MonkeyCode fits in this workflow

The reason I built this probe as a before-shipping habit rather than a one-off is that I needed somewhere cheap enough to run it repeatedly, at different concurrencies, on a schedule that does not depend on my laptop being awake. MonkeyCode is what I use for that: per the operator, the project offers free model access for the inference calls and a free server option, which is where I run the probe loop instead of from my desk. The operator states the free tier currently includes 10M tokens and a free server — treat that as a claim with a date on it and check the current terms before you plan a budget around them, because free-tier terms move and a probe harness that assumes yesterday's quota is a probe harness that lies to you.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I run the probe from the free server, keep the JSONL, and diff runs week over week. The harness is the artifact here — the endpoint is just what I point it at today. If you want to run the same thing, a free account is enough to get started; the value is in the diff, not the first run.

Decision table

Symptom in the report	Likely cause	What I do
p50 flat, tail ratio > 10	shared queue, bursty neighbours	ship only behind async jobs; never in a request path
429 at `CONC=1`	per-key rate limit, not load	budget per minute, not per run; back off with jitter
`ttft_over_budget` climbing with `CONC`	proxy idle timeout	cap concurrency below the knee; hard TTFT deadline in the client
`no_tokens` with status 200	dropped stream	treat as failure and re-run once with a deadline, not a loop
`usage` absent on streams	server ignores `include_usage`	count locally, bill locally, negotiate with numbers you own

Limitations, and who should not do this

This probe measures one endpoint from one network location at one moment. It is not a provider ranking, and running it once proves nothing — the diff between runs is the signal. It also measures first-token latency, not output quality; a fast wrong answer is still wrong.

If you have a hard p99 SLO on a user-facing path, if your workload needs contractual data residency, or if you cannot tolerate a free tier's terms changing mid-quarter, do not build on this. Pay for a provider with an SLA, or run your own box. Free inference is excellent for probes, batch jobs, internal tools, and the kind of evaluation work that gets better the more often you repeat it — and it is a poor foundation for anything you promised a customer.

So: before you argue about whether agents are real, go find out what your endpoint's worst 1% looks like. Would you ship on the number you have right now, or on the number you assumed?

I Scored Duplicate Tool Calls. Busy Was Not Progress.

Jordan Liu — Sun, 13 Sep 2026 20:17:06 +0000

The agent was not stuck. It said so. It had a tool trace, a confident summary, and three searches that only a slide deck would call iteration. I scored the repeats. Busy was not progress.

You know this transcript. The model calls search_docs with {"q": "rate limit"}. Then it calls it again. Then it swaps in "rate-limit" and walks the same hallway like the paint is new. Chat still reads as motion. The bill still reads as work. Was any of it?

I stopped arguing with the summary and started scoring the trace.

The rule is rude on purpose. If two tool calls share a name and the same arguments after canonical JSON, the second call is a stall. If the name matches and the arguments are a costume change—hyphens, plurals, the same question in a nicer coat—it might still be a stall. If the arguments actually move the search space, that is progress. Chat is not the judge of that. The trace is.

I wanted a harness a code review could replay. No cluster. No dashboard glow. A Python file, a folder of fixture transcripts, and a score that can fail CI. The fixtures are boring, which is the point. They grade the scorer. The scorer does not get to grade itself.

The hallway test

Think of an agent loop like a night guard. A useful patrol changes rooms. A useless one stamps the same doorway and files it as a shift. Token spend does not tell you which shift you bought. Duplicate tool calls do.

I split the score into two channels on purpose. Channel A is exact. It canonicalizes name plus json.dumps(args, sort_keys=True) and counts how many calls collide with an earlier key. Channel B is optional and meaner: a model that only answers whether the second argument blob explores anything new. Channel B does not own the gate. It writes an opinion. Disagreement is what I actually want in the log.

Here is channel A. It has no personality, which is why I trust it in CI.

# halt_score.py
from __future__ import annotations

import json
from pathlib import Path
from typing import Any

THRESHOLD = 0.34  # two clones in three calls is a habit, one retry can be jitter


def canonical(call: dict[str, Any]) -> tuple[str, str]:
    name = str(call.get("name") or "")
    args = call.get("args") or {}
    return name, json.dumps(args, sort_keys=True, separators=(",", ":"))


def duplicate_ratio(calls: list[dict[str, Any]]) -> float:
    seen: list[tuple[str, str]] = []
    dups = 0
    for call in calls:
        key = canonical(call)
        if key in seen:
            dups += 1
        seen.append(key)
    return dups / max(len(calls), 1)


def grounded_done(record: dict[str, Any]) -> bool:
    calls = record.get("tool_calls") or []
    claimed = bool(record.get("claimed_done"))
    if claimed and not calls:
        return False
    return True


def score(record: dict[str, Any]) -> dict[str, Any]:
    calls = record.get("tool_calls") or []
    ratio = round(duplicate_ratio(calls), 3)
    fail_dup = len(calls) >= 3 and ratio > THRESHOLD
    fail_ground = not grounded_done(record)
    return {
        "id": record.get("id"),
        "n_calls": len(calls),
        "duplicate_ratio": ratio,
        "fail_duplicates": fail_dup,
        "fail_ungrounded_done": fail_ground,
        "fail": fail_dup or fail_ground,
    }


def load_fixtures(dir_path: str) -> list[dict[str, Any]]:
    rows = []
    for path in sorted(Path(dir_path).glob("*.json")):
        rows.append(json.loads(path.read_text()))
    return rows


if __name__ == "__main__":
    for row in load_fixtures("fixtures"):
        print(json.dumps(score(row), indent=2))

Run that on fixtures you control and you get numbers a skeptic can recompute with python halt_score.py. I do not need a model to tell me that three identical searches are a loop wearing a lab coat.

The fixtures I used look like this. Copy them. Change them. If your scorer cannot fail these, it is a blog post, not a test.

{"id": "clone", "claimed_done": true, "tool_calls": [
  {"name": "search_docs", "args": {"q": "rate limit"}},
  {"name": "search_docs", "args": {"q": "rate limit"}},
  {"name": "search_docs", "args": {"q": "rate limit"}}
]}

{"id": "clean", "claimed_done": true, "tool_calls": [
  {"name": "search_docs", "args": {"q": "webhooks"}},
  {"name": "read_file", "args": {"path": "docs/webhooks.md"}},
  {"name": "apply_patch", "args": {"path": "app.py", "diff": "---"}}
]}

{"id": "hyphen", "claimed_done": true, "tool_calls": [
  {"name": "search_docs", "args": {"q": "rate limit"}},
  {"name": "search_docs", "args": {"q": "rate-limit"}},
  {"name": "search_docs", "args": {"q": "rate limits in API"}}
]}

{"id": "page", "claimed_done": true, "tool_calls": [
  {"name": "search_docs", "args": {"q": "webhooks", "page": 1}},
  {"name": "search_docs", "args": {"q": "webhooks", "page": 2}}
]}

{"id": "essay", "claimed_done": true, "tool_calls": []}

I ran the deterministic scorer against those five files. These numbers are fixture math, not a hosted-model bake-off. clone lands at duplicate_ratio=0.667 and fails. clean lands at 0.0 and passes. hyphen also lands at 0.0, which is the exact-match blind spot: the hallway changed spelling, not rooms. page stays at 0.0 because page actually moved. essay fails fail_ungrounded_done. That last one is not a duplicate problem. It is a done-claim with no fingerprints.

Exact match is honest. It is also a terrible literary critic. Engineers write queries like humans. Hyphens happen. Plurals happen. The same question shows up in a nicer coat and channel A waves it through.

The optional judge, kept on a leash

So I added channel B. Not as truth. As a suspect.

A free model can look at two argument blobs and say whether the second call explores anything new. That helps on the hyphen trick. It also launders a real refinement into a "semantic duplicate" that never happened. Pagination is the obvious landmine. Filters are next. Any field that is small in JSON and large in meaning will make a similarity judge flinch.

I keep that opinion off my laptop when I bother to run it. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are the optional judge lane in this workflow—the fixtures and the exact-match scorer do not need them, and I will not pretend a one-off completion is a benchmark. If you want that lane off your workstation, their free server is how I isolated it. Steal the scorer either way.

The client is deliberately boring. OpenAI-shaped chat. Model name from the environment. No system prompt poetry.

# judge.py
from __future__ import annotations

import json
import os
import urllib.request

PROMPT = """You compare two tool argument blobs for the same tool name.
Reply with JSON only: {"duplicate": true or false, "reason": "<=20 words"}.
True means the second call does not explore a new search space.
Pagination, new paths, tighter filters, and new ids are NOT duplicates."""


def judge_pair(name: str, a: dict, b: dict) -> dict:
    body = json.dumps({
        "messages": [
            {"role": "system", "content": PROMPT},
            {"role": "user", "content": json.dumps({"tool": name, "first": a, "second": b})},
        ],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        os.environ["CHAT_URL"],
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {os.environ.get('CHAT_TOKEN', '')}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read().decode())
    content = payload["choices"][0][message_key(payload)]["content"]
    return json.loads(content)


def message_key(payload: dict) -> str:
    return "message"

I did not print a live-model leaderboard. Those numbers rot by lunch, and they would smuggle a product claim I cannot defend. What I did ship is a stubbed judge so you can rerun the disagreement table tonight, offline. The stub is the experiment control. Swap it for a live endpoint later if you want to measure that endpoint on these fixtures. That is evaluation. A vibe is not.

# test_halt_score.py
import json
from halt_score import score

CLONE = json.loads(open("fixtures/clone.json").read())
CLEAN = json.loads(open("fixtures/clean.json").read())
HYPHEN = json.loads(open("fixtures/hyphen.json").read())
PAGE = json.loads(open("fixtures/page.json").read())
ESSAY = json.loads(open("fixtures/essay.json").read())

STUB = {
    ("search_docs", "rate limit", "rate-limit"): True,
    ("search_docs", "rate limit", "rate limits in API"): True,
    ("search_docs", "webhooks|1", "webhooks|2"): True,  # intentional false stall
}


def stub_duplicate(first, second) -> bool:
    q1 = str(first.get("q"))
    q2 = str(second.get("q"))
    p1 = first.get("page")
    p2 = second.get("page")
    if p1 != p2:
        return STUB[("search_docs", "webhooks|1", "webhooks|2")]
    return STUB.get(("search_docs", q1, q2), False)


def test_clone_fails_exact():
    out = score(CLONE)
    assert out["duplicate_ratio"] == 0.667
    assert out["fail_duplicates"] is True


def test_clean_passes():
    assert score(CLEAN)["fail"] is False


def test_hyphen_is_exact_blind():
    out = score(HYPHEN)
    assert out["duplicate_ratio"] == 0.0
    assert out["fail_duplicates"] is False


def test_page_is_not_an_exact_clone():
    assert score(PAGE)["duplicate_ratio"] == 0.0


def test_essay_fails_grounding():
    assert score(ESSAY)["fail_ungrounded_done"] is True


def test_stub_judge_breaks_on_pagination():
    a, b = PAGE["tool_calls"]
    # This is the known break: a refinement that a similarity judge calls a stall.
    assert stub_duplicate(a["args"], b["args"]) is True

Read the stub like a crime board, not a vendor slide. Exact match nails clone and misses hyphen. The stub catches the hyphen trick and then, on page, calls a real refinement a stall. That is the break I care about. Semantic sameness is allergic to pagination. If you ship the judge as a gate, you will fail honest loops and call it quality.

So the workflow I actually use is meaner than "ask the model." CI fails on exact duplicates above 0.34 once there are at least three tool calls. One retry can be jitter. Two clones in three calls is a habit. The judge, when it exists, runs in warn mode. A human only looks when the two channels disagree. Disagreement is the product. Agreement is boring.

Why 0.34? Because I wanted a threshold I could explain without a spreadsheet religion. Two-in-three is 0.667. One-in-three is 0.333. I fail above a hair over one-in-three so a single retry does not page me and a repeating search does. If your tools are chatty by design, pick a different hair. Write it next to the scorer. Do not hide it in a prompt.

Where this approach lies to you

Canonical JSON does not know that {"path": "./a.py"} and {"path": "a.py"} are the same file. A model might. A model might also invent a third path and grade it with a straight face. I do not normalize paths in v1. I would rather see the stall than sand it off.

The stubbed judge is not evidence about any hosted model's quality. It is a canned false positive so the test stays deterministic. Point channel B at a live free endpoint and you are measuring that endpoint on five fixtures, on one day, with one prompt. That is a lab note. It is not a ranking.

Empty-tool essays fail a different check than duplicates. Do not mash them into one number and call it "agent quality." A loop that never calls tools and a loop that calls the same tool forever are different crimes. They just share a press release.

Who should not use this? If your tools are non-deterministic by design—clocks, random samples, live log tails—exact-match will scream every turn and you will hate the scorer instead of the loop. If your "agent" is a single function with no trace, you do not have a hallway to score. If you need the model judge to be the gate, you are buying flakiness and naming it evaluation. If you came here for a demo that always looks busy, this harness will embarrass you. Good.

I still read the summaries. I just refuse to let them grade the patrol. Next time the model says it investigated, will you score the doorway—or the press release?

I Scored Retry Loops. Boundedness Was Optional.

Jordan Liu — Sat, 12 Sep 2026 18:20:49 +0000

Boundedness failed first. Not the model. Not the schema. The ceiling. I stopped asking whether an agent was "actually thinking" and started asking a dumber question that still draws blood. Does this while ever stop if the tool hangs?

That question is unromantic. It is also the one that keeps a shared box from turning into a heat lamp. I wanted a number I could re-run on Monday, not a vibe from a chat transcript that already flattered me.

Most agent retry code I read is a loop wearing a trench coat. You know the coat. try, except, sleep, continue, maybe a comment that says "be resilient." Resilience without a budget is just an unbounded bill. Did the author cap attempts? Did they cap wall time? Or did they only cap their optimism?

I built a scorer. Not a leaderboard. A lint with opinions. It reads Python, walks the AST, and grades a retry helper the way I wish code review would: boundedness first, poetry never. The artifact is the point. If you strip every product name out of this article, you should still be able to save the files and get the same integers.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

When I need a candidate loop I do not want to hand-write at 11pm, I ask a free model through MonkeyCode. Then I drop the output into fixtures/ and refuse to trust it until the scorer prints a score. The free server option is not a mascot for that step. It is the second half of the experiment, the half where time.sleep meets scheduling noise. My laptop is a terrible witness. It is too fast, and it likes me.

The rubric I actually encode

I do not grade "quality of reasoning." I grade whether a stop condition exists in the artifact you were about to deploy. Those are different papers. One of them is a keynote. The other one is a unit test.

A retry helper starts at zero. I add points for an attempt cap, a wall-clock timeout, backoff, jitter, and an idempotency key traveling with the request. I subtract points for while True with no break budget, and for sleep that cannot see a deadline. Is that harsh? Good. A loop that can outlive your patience is not robust. It is a fuse with the coating peeled off.

Save this as loop_hygiene.py. It is the whole static half.

#!/usr/bin/env python3
"""Score retry-loop hygiene from Python source. Fixture-calibrated, not a bake-off."""
from __future__ import annotations

import ast
import json
import sys
from dataclasses import asdict, dataclass, field
from pathlib import Path

ATTEMPT_NAMES = {"max_attempts", "max_retries", "retries", "attempts", "n_tries"}
TIMEOUT_NAMES = {"timeout", "deadline", "max_seconds", "wall_timeout", "budget_s"}
BACKOFF_NAMES = {"backoff", "backoff_s", "delay", "base_delay"}
JITTER_NAMES = {"jitter", "jitter_s", "jitter_ratio"}
IDEM_NAMES = {"idempotency_key", "idempotency", "request_id", "dedupe_key"}


@dataclass
class LoopScore:
    path: str
    has_max_attempts: bool = False
    has_timeout: bool = False
    has_backoff: bool = False
    has_jitter: bool = False
    has_idempotency: bool = False
    unbounded_while: bool = False
    sleep_without_budget: bool = False
    score: int = 0
    notes: list[str] = field(default_factory=list)


class HygieneVisitor(ast.NodeVisitor):
    def __init__(self) -> None:
        self.names: set[str] = set()
        self.unbounded_while = False
        self.sleep_calls = 0
        self.breaks_in_loop = 0

    def visit_Name(self, node: ast.Name) -> None:
        self.names.add(node.id)
        self.generic_visit(node)

    def visit_While(self, node: ast.While) -> None:
        constant_true = (
            isinstance(node.test, ast.Constant) and node.test.value is True
        ) or (
            isinstance(node.test, ast.Constant) and node.test.value == 1
        )
        if constant_true:
            self.unbounded_while = True
        for child in ast.walk(node):
            if isinstance(child, ast.Break):
                self.breaks_in_loop += 1
        self.generic_visit(node)

    def visit_Call(self, node: ast.Call) -> None:
        func = node.func
        name = ""
        if isinstance(func, ast.Attribute):
            name = func.attr
        elif isinstance(func, ast.Name):
            name = func.id
        if name in {"sleep", "usleep"}:
            self.sleep_calls += 1
        self.generic_visit(node)


def score_source(path: Path, src: str) -> LoopScore:
    tree = ast.parse(src)
    v = HygieneVisitor()
    v.visit(tree)
    row = LoopScore(path=str(path))
    row.has_max_attempts = bool(ATTEMPT_NAMES & v.names)
    row.has_timeout = bool(TIMEOUT_NAMES & v.names)
    row.has_backoff = bool(BACKOFF_NAMES & v.names)
    row.has_jitter = bool(JITTER_NAMES & v.names)
    row.has_idempotency = bool(IDEM_NAMES & v.names)
    row.unbounded_while = v.unbounded_while and v.breaks_in_loop == 0
    budget = row.has_max_attempts or row.has_timeout
    row.sleep_without_budget = v.sleep_calls > 0 and not budget

    n = 0
    if row.has_max_attempts:
        n += 2
        row.notes.append("+2 attempt cap")
    if row.has_timeout:
        n += 2
        row.notes.append("+2 wall clock")
    if row.has_backoff:
        n += 1
        row.notes.append("+1 backoff")
    if row.has_jitter:
        n += 1
        row.notes.append("+1 jitter")
    if row.has_idempotency:
        n += 2
        row.notes.append("+2 idempotency key")
    if row.unbounded_while:
        n -= 3
        row.notes.append("-3 unbounded while")
    if row.sleep_without_budget:
        n -= 2
        row.notes.append("-2 sleep with no ceiling")
    row.score = n
    return row


def main(argv: list[str]) -> int:
    root = Path(argv[1] if len(argv) > 1 else "fixtures")
    rows = []
    for path in sorted(root.glob("*.py")):
        rows.append(score_source(path, path.read_text(encoding="utf-8")))
    print(json.dumps([asdict(r) for r in rows], indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Run it like a test, not like a demo.

mkdir -p fixtures
python3 loop_hygiene.py fixtures/

If that command prints nothing, you have an empty folder, not a passing grade. Empty is not resilient either.

Four fixtures, four numbers I can defend

I calibrated the scorer on four files I wrote on purpose. This is not a claim about any named model. It is a claim that the ruler does not flop when I bend it. Two fixtures are the "looks fine in a PR" loops a chat window emits when you say make it robust. Two are loops I would actually leave under a cron that can page me.

fixtures/a_trench_coat.py is the coat.

import time

def call_tool(url):
    while True:
        try:
            return fetch(url)
        except Exception:
            time.sleep(1)

fixtures/b_range_but_naked.py looks bounded until you notice the request can be charged twice.

import time

def call_tool(url):
    max_attempts = 5
    for _ in range(max_attempts):
        try:
            return fetch(url)
        except Exception:
            time.sleep(0.5)
    raise RuntimeError("gave up")

fixtures/c_budget.py finally admits time exists.

import random, time

def call_tool(url, timeout=8.0):
    max_attempts = 5
    backoff = 0.2
    deadline = time.monotonic() + timeout
    for attempt in range(max_attempts):
        if time.monotonic() >= deadline:
            raise TimeoutError("wall clock")
        try:
            return fetch(url)
        except Exception:
            jitter = random.random() * 0.05
            sleep_for = min(backoff, max(0.0, deadline - time.monotonic()))
            time.sleep(sleep_for + jitter)
            backoff *= 2
    raise RuntimeError("attempts exhausted")

fixtures/d_idempotent.py is the one I would actually argue for in review.

import random, time, uuid

def call_tool(url, timeout=8.0, idempotency_key=None):
    max_attempts = 5
    backoff = 0.2
    jitter = 0.05
    deadline = time.monotonic() + timeout
    key = idempotency_key or str(uuid.uuid4())
    for attempt in range(max_attempts):
        if time.monotonic() >= deadline:
            raise TimeoutError("wall clock")
        try:
            return fetch(url, headers={"Idempotency-Key": key})
        except Exception:
            sleep_for = min(backoff, max(0.0, deadline - time.monotonic()))
            time.sleep(sleep_for + random.random() * jitter)
            backoff *= 2
    raise RuntimeError("attempts exhausted")

On my machine, against those four files, the scorer prints scores of -5, 2, 6, 8. Read that again. The trench coat is not "a bit sloppy." It is negative. The range-loop that reviewers like is a 2 because it remembered max_attempts and then immediately forgot time, jitter, and duplicate side effects. The budgeted loop is a 6. The idempotent one is an 8. That spread is the experiment working. If your ruler cannot tell a fuse from a space heater, you are collecting anecdotes.

Would I ship a 2? Not on a box I do not babysit. Would I call an 8 production-ready? Still no. An AST visitor cannot see a non-idempotent POST hiding behind a pretty header name. The number is a gate, not a medal.

The prompt I actually paste

When I bother a free model, I do not ask it to "write a robust agent." That prompt is how you get the trench coat. I ask for a function with a contract the scorer can see.

Write a Python function call_tool(url, timeout=8.0, idempotency_key=None)
that retries a failing fetch(). Requirements:
- max_attempts is an int cap
- timeout is a wall-clock budget using time.monotonic()
- exponential backoff plus jitter
- send Idempotency-Key on every attempt
- no while True
Return only the function.

Then I save whatever came back as fixtures/model_candidate.py and run the same command. I do not read the prose around the code first. The integer first. If that sounds rude, ask yourself why a retry helper deserves manners that a unit test does not.

Did the model add jitter, or did it just sleep(1) in a nicer jacket? Did it name timeout and then never consult a clock? Names are cheap. The visitor is cheaper. That is the whole trick.

The half my laptop keeps lying about

Static scores catch missing ceilings. They do not catch a ceiling that exists on paper and then loses a fight with real delay. So I added a dynamic probe. It is deliberately ugly. A mock tool returns after 0.25s, then 1s, then 3s, then it never returns. The loop under test has to respect max_attempts and a wall clock, and the process has to exit.

#!/usr/bin/env python3
"""Dynamic probe. Label: run this; do not treat my laptop timings as yours."""
import threading
import time
from http.server import BaseHTTPRequestHandler, HTTPServer

DELAYS = [0.25, 1.0, 3.0, 999.0]
HITS = {"n": 0}

class SlowTool(BaseHTTPRequestHandler):
    def do_GET(self):
        i = min(HITS["n"], len(DELAYS) - 1)
        HITS["n"] += 1
        time.sleep(DELAYS[i])
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b"ok")

    def log_message(self, fmt, *args):
        return

def main() -> None:
    server = HTTPServer(("127.0.0.1", 8765), SlowTool)
    t = threading.Thread(target=server.serve_forever, daemon=True)
    t.start()
    deadline = time.monotonic() + 10.0
    # Import YOUR candidate here and call it against http://127.0.0.1:8765/
    # If this process is still alive past deadline, the loop has no ceiling.
    while time.monotonic() < deadline:
        time.sleep(0.05)
    server.shutdown()
    print({"hits": HITS["n"], "exited_before_watchdog": True})

if __name__ == "__main__":
    main()

On my laptop the 3s delay is a polite pause. On a shared free server, that pause collides with other people's jobs and whatever watchdog the box actually has. If your stop condition is "I got bored watching the spinner," the server will not get bored for you. That is why the free server belongs in the method instead of in a slogan. Latency is a witness. A quiet SSD is a friend of the defendant.

I do not publish a bake-off table for that half, because I will not invent a model name, a hardware spec, or a permanence claim. The protocol is the result. You run the candidate against the mock. Either the process exits before the 10s watchdog, or you just watched an unbounded loop spend someone else's electricity. Which outcome did you want to be surprised by, the integer or production?

What this does not measure

This does not measure whether a model is "better at coding than most developers." I have no idea how you would even sample that sentence without lying to yourself. It does not measure MCP servers, community knowledge bases, or whether agents are secretly if-statements. Plenty of if-statements are honest. The dishonest ones are the loops that refuse to be if-statements with a counter.

The AST scorer is a flashlight. It will miss retries hidden in exec, in a decorator I did not name, or in a subprocess that relaunches itself because someone thought that was architecture. Free model output shifts. Free servers are not SLOs. If you need pinned model identity, reserved CPUs, or a vendor contract that names latency, do not use this workflow. Use a paid, named endpoint and a box you can SSH into without guessing who else is on the metal.

Who should skip it: anyone shipping billing, healthcare, or a retry around a non-idempotent charge. A hygiene score of 8 is not a PCI audit. Also skip it if you will not read the generated loop. A scorer you ignore is just another dashboard, and dashboards do not page you until the invoice does.

I still run the scorer before I argue about "agent quality." Boundedness is optional in chat. It is not optional in a process table. If you want the dynamic half to be ugly on a machine that is not your laptop, I parked that probe on MonkeyCode's free server and let the delays stay dishonest. Steal the harness first. Fight me on the rubric second.

I Counted the Turns That Didn't Need a Model

Jordan Liu — Fri, 11 Sep 2026 15:53:30 +0000

Half of a convincing agent loop is a state machine wearing a chat template. I did not believe that until I scored turns instead of demos. The model can still be useful. The question is which turns actually required it.

You have seen the other version of this story. A loop, a tool list, a thought field, a screenshot. It looks like software that thinks. Then you watch it radio headquarters to ask whether priority=low and status=open should skip the on-call queue. That is not reasoning. That is a missing if.

So I built a harness that asks a ruder question than "did the agent finish?" Did this turn need a model at all? Could a closed-set rule have produced the same action? Did we skip a tool we already had an id for? Did we refuse to halt after the schema was already complete?

I am not going to sell you a leaderboard I did not run in a lab. This post is the experiment. You point it at a free model endpoint, park the runner on a free server if you want the loop off your laptop, and you get three rates: over-model, under-tool, loop-overrun. Those three embarrass a demo faster than a comment thread.

The ticket is small on purpose

A support ticket arrives as JSON. Allowed actions are a closed set: route_oncall, route_triage, route_docs, lookup_order, draft_reply, halt. Some tickets are fully determined by fields you already have. Product is billing, priority is low, no order id. That is route_docs or draft_reply. No model required. Some tickets carry an order id and a shipping complaint. That is a real lookup. If the loop skips the tool and invents a tracking status, that is under-tool. If it looks up the order and then asks the model whether delivered means delivered, that is over-model. See the difference?

I keep the gold policy in code, not in a prompt. Prompts drift. Functions do not, at least not without a diff.

# gold_policy.py — closed-set oracle. Not a prompt.
CLOSED = {
    "route_oncall", "route_triage", "route_docs",
    "lookup_order", "draft_reply", "halt",
}

def gold_plan(ticket: dict) -> list[str]:
    product = ticket.get("product")
    priority = ticket.get("priority")
    order_id = ticket.get("order_id")
    text = (ticket.get("text") or "").lower()

    needs_lookup = bool(order_id) and any(
        w in text for w in ("ship", "track", "delivery", "package")
    )
    if needs_lookup:
        return ["lookup_order", "draft_reply", "halt"]
    if priority == "high" and product == "payments":
        return ["route_oncall", "halt"]
    if product == "billing" and priority == "low":
        return ["route_docs", "halt"]
    return ["route_triage", "halt"]


def rule_resolvable(ticket: dict) -> bool:
    """True when the route does not depend on free text."""
    plan = gold_plan(ticket)
    return plan[0] != "lookup_order" and "draft_reply" not in plan

The loop under test is allowed to look like an agent. It can call tools. It can write a plan. I do not grade the plan. I grade the action sequence against the gold policy and against a halt oracle. If that sounds harsh, good. Demos already have enough fans.

Score the control flow, not the vibes

# score_loop.py
from dataclasses import dataclass, field
from gold_policy import gold_plan, rule_resolvable, CLOSED

@dataclass
class Turn:
    action: str
    used_model: bool
    tool: str | None = None
    output: str = ""

@dataclass
class Score:
    over_model: bool = False
    under_tool: bool = False
    loop_overrun: bool = False
    unknown_verb: bool = False
    actions: list[str] = field(default_factory=list)
    notes: list[str] = field(default_factory=list)


def score_transcript(ticket: dict, turns: list[Turn], max_turns: int = 6) -> Score:
    s = Score(actions=[t.action for t in turns])
    gold = gold_plan(ticket)
    resolvable = rule_resolvable(ticket)

    for t in turns:
        if t.action not in CLOSED:
            s.unknown_verb = True
            s.notes.append(f"invented verb: {t.action}")

    called_lookup = any(t.tool == "lookup_order" or t.action == "lookup_order" for t in turns)
    if gold[0] == "lookup_order" and not called_lookup:
        s.under_tool = True
        s.notes.append("needed lookup_order, never called it")

    model_turns = [t for t in turns if t.used_model]
    if resolvable and model_turns:
        # A model preface on a closed-set route is still a wasted turn.
        s.over_model = True
        s.notes.append(f"rule-resolvable, but {len(model_turns)} model turns")

    if len(turns) > len(gold) or len(turns) > max_turns:
        s.loop_overrun = True
        s.notes.append(f"turns={len(turns)} gold={len(gold)} cap={max_turns}")

    halted = any(t.action == "halt" for t in turns)
    if not halted:
        s.loop_overrun = True
        s.notes.append("never halted")
    return s

That scorer is the article. over_model fires when the gold policy says the ticket was rule-resolvable and the loop still spent a model turn deciding the route. under_tool fires when an order lookup was required and never happened. loop_overrun fires when we already had a terminal action and the model kept talking. I used to score n-gram overlap on the draft reply. Cute. Useless. The blast was in the control flow.

How do you drive a model without pretending the HTTP client is the insight? You isolate one function: complete(messages) -> str. On a laptop it can be a stub. On a server it can be whoever is hosting the free endpoint this week. I do not need a model name to make this honest. I need a transcript.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I park this runner on MonkeyCode's free server option and point complete() at their free model access because the experiment is chatty and I did not want the scoring loop itself to become a billing subplot. That is the only product claim I am making: free model access, and a free server you can leave the harness on. No quota theater. No hardware fanfic. If the endpoint is down, the harness fails closed. That is a valid result.

# runner.py — fill in complete(); do not paste keys.
import json, os, urllib.request
from score_loop import Turn, score_transcript
from gold_policy import CLOSED

SYSTEM = (
    "You emit one JSON object per turn: "
    '{"action": "<one of ' + ",".join(sorted(CLOSED)) + '>", '
    '"tool": null or "lookup_order", "output": "..."}. '
    "No preface. Halt when the action is terminal."
)

def complete(messages: list[dict]) -> str:
    url = os.environ["MODEL_URL"]  # your free-model endpoint
    body = json.dumps({"messages": messages, "temperature": 0}).encode()
    req = urllib.request.Request(
        url, data=body, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode())
    # Adapt this one line to whatever envelope your endpoint returns.
    return payload["choices"][0]["message"]["content"]

def parse_action(raw: str) -> dict:
    raw = raw.strip()
    start, end = raw.find("{"), raw.rfind("}")
    if start < 0 or end <= start:
        return {"action": "unknown", "tool": None, "output": raw[:200]}
    return json.loads(raw[start:end + 1])

def run_ticket(ticket: dict, max_turns: int = 6) -> list[Turn]:
    messages = [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": json.dumps(ticket)},
    ]
    turns: list[Turn] = []
    for _ in range(max_turns):
        raw = complete(messages)
        parsed = parse_action(raw)
        action = parsed.get("action") or "unknown"
        tool = parsed.get("tool")
        turn = Turn(action=action, used_model=True, tool=tool,
                    output=str(parsed.get("output", "")))
        turns.append(turn)
        messages.append({"role": "assistant", "content": raw})
        if action == "lookup_order" and ticket.get("order_id"):
            tool_obs = json.dumps({"order_id": ticket["order_id"], "status": "delivered"})
            messages.append({"role": "user", "content": "TOOL_RESULT " + tool_obs})
        if action == "halt":
            break
    return turns

if __name__ == "__main__":
    over = under = overrun = n = 0
    with open("tickets.jsonl") as f:
        for line in f:
            ticket = json.loads(line)
            turns = run_ticket(ticket)
            s = score_transcript(ticket, turns)
            n += 1
            over += int(s.over_model)
            under += int(s.under_tool)
            overrun += int(s.loop_overrun)
            print(ticket["id"], s)
    print(f"n={n} over_model={over/n:.2f} under_tool={under/n:.2f} overrun={overrun/n:.2f}")

Run it like this. The fixture is boring on purpose. Boring fixtures keep you from grading vibes.

export MODEL_URL="https://your-free-model-endpoint.example/v1/chat"
python runner.py

{"id": "t1", "product": "billing", "priority": "low", "order_id": null, "text": "invoice looks high"}
{"id": "t2", "product": "shop", "priority": "med", "order_id": "A-1042", "text": "where is my package"}
{"id": "t3", "product": "payments", "priority": "high", "order_id": null, "text": "card charged twice"}

When I walk a constructed transcript through the scorer — and I am labeling this as a constructed example, not a published run — the pattern is almost rude. Ticket t1 is rule-resolvable. The loop still emits a "let me think" preface and a tool-less route. Over-model. Ticket t2 needs lookup_order. The loop drafts a tracking paragraph with a plausible carrier name. Under-tool. Ticket t3 should be route_oncall then halt. The loop routes, then asks the model whether paging a human is "too aggressive," then drafts a reply anyway. Overrun. None of those failures show up if you only read the final English.

That is the analogy I cannot shake. It is a night guard with a radio. Every unlocked door, they call dispatch. Dispatch is a language model. Dispatch will always have something to say. Your job is to count how often they already had the key.

Who is the judge?

Should you trust a free model for this? For scoring control flow, yes, because the gold policy lives in your repo. The model is the system under test, not the judge. Do not invert that. If you let the same free model grade its own halt decisions, you have built a mirror, not an evaluation. I have watched people do that and then quote the mirror as a metric. Please do not.

Want a fourth number? Count unknown verbs. The minute your "agent" invents escalate_via_slack because the prompt said "be helpful," the closed-set assumption is on fire. unknown_verb is the alarm. A trench-coat of if-statements at least has the honesty to name its branches in source control.

def summarize(scores: list) -> dict:
    n = max(len(scores), 1)
    return {
        "n": len(scores),
        "over_model_rate": sum(s.over_model for s in scores) / n,
        "under_tool_rate": sum(s.under_tool for s in scores) / n,
        "overrun_rate": sum(s.loop_overrun for s in scores) / n,
        "unknown_verb_rate": sum(s.unknown_verb for s in scores) / n,
    }

Print that next to the pull request. If the gif still looks magical and over_model_rate is 0.6, the gif does not get a vote. What were you actually shipping — a policy, or a narrator?

Limitations, said in daylight

This harness does not measure code quality, security, or user empathy. It does not know if the draft reply is kind. It will happily pass a monstrous sentence if the action tag is halt and the tool sequence matched. It assumes your tool results are ground truth. If your lookup API lies, the scorer will bless the lie.

It also assumes a closed action set. Open-ended agents undercount here. And a free model on a free server will be slower and noisier than whatever you put on a credit card. Noise is useful in a fuzzer. Noise is not an SLA. I am not giving you tokens-per-second, GPU SKUs, or a promise the free tier lasts forever, because I did not measure those and I will not invent them.

Who should not use this approach? Anyone shipping medical, financial, or on-call routing without a deterministic policy in front of the model. Anyone who needs a published speed chart from this post. There isn't one. Anyone who wants to prove their agent is already better than most developers. That claim is a different fight, and this scorer will not win it for you. If your loop is a single tool call with no branch, you do not need this. You need a unit test.

I still use a model. I use it when the ticket text is actually ambiguous, when the lookup result is messy, when the draft has to quote a field the rules do not know how to phrase. The win is not "delete the model." The win is knowing which turns were theater.

If you run the harness, keep the three rates next to the diff. Over-model, under-tool, overrun. If those numbers move, the narrator in the middle of the loop can finally shut up.

I Asked for One Line. I Scored the Blast Radius.

Jordan Liu — Wed, 09 Sep 2026 16:08:49 +0000

If you do not bound the diff, a free coding model will mop the kitchen while you asked it to wipe one mug. That is the finding I trust. Not a vibe. Not a leaderboard. A git geometry problem you can fail in CI.

Everyone is arguing about agents that “assume things.” Fine. I care about a smaller crime. You prompt a one-line fix. The patch rewrites imports, renames a helper, and “cleans” a comment two files away. Does the test suite still pass? Sure. That is how the lie gets a green check.

I stopped asking “did it compile?” I started asking “what else moved?” A passing test with a seven-file diff is not help. It is an unreviewed refactor wearing a bugfix hat. You would not merge that from a junior at 5 p.m. Why merge it from a model because the chat sounded confident?

This is a lab protocol, not a victory lap. I have not cooked a fake accuracy number for you. The point is that you can reproduce the failure mode on a throwaway box, including a free remote, and keep the scoreboard honest.

The trap, in one mug

Picture a function named clamp. It has an off-by-one. The rest of the module is ugly on purpose: unused import, a noisy comment, a helper with a clumsy name. That mess is the control group. If the model touches it, the model cheated the prompt.

The analogy is a surgeon who “tidies” your other organs because they were in the room. Generous? Maybe. Consent? No. Your review queue is not a spa.

I keep the fixture tiny so the blast radius cannot hide. Three files. One intended change. Everything else is a tripwire.

Where a free model and a free server actually earn rent

You need a loop: prompt, apply, score, reset. Local GPUs are precious. Laptops thermal-throttle. A scratch remote is enough if you never put secrets on it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode’s free model access and free server option as that scratch remote — a place to run the loop, not a substitute for the scorer below. If the product vanished tomorrow, the harness would still be the point.

Do not paste production .env files onto a shared box. Do not treat availability as an SLA. Free means you can afford retries. It does not mean the model will stay in its lane.

Fixture: a repo that wants to be over-cleaned

Label this as an unexecuted template until you run it. Create a directory, then drop three files. The off-by-one is real. The unused import is bait.

blast_lab/
  clamp_kit.py
  test_clamp.py
  score_blast.py

clamp_kit.py is the patient:

# blast_lab/clamp_kit.py
import math  # bait: unused on purpose

def _noisy_helper(x):
    # clumsy on purpose; do not rename
    return x

def clamp(n, lo, hi):
    """Inclusive clamp. Keep the signature frozen."""
    if lo > hi:
        raise ValueError("lo > hi")
    # off-by-one on the high side: uses hi - 1
    if n < lo:
        return _noisy_helper(lo)
    if n > hi:
        return _noisy_helper(hi - 1)
    return _noisy_helper(n)

test_clamp.py only cares about behavior. That is intentional. Behavior-only tests are how a chatty model hides a rewrite.

# blast_lab/test_clamp.py
from clamp_kit import clamp
import pytest

def test_high_edge():
    assert clamp(10, 0, 10) == 10

def test_low_edge():
    assert clamp(-1, 0, 10) == 0

def test_mid():
    assert clamp(3, 0, 10) == 3

def test_bad_range():
    with pytest.raises(ValueError):
        clamp(1, 5, 2)

Before any model runs, prove the bug exists:

cd blast_lab
python -m pytest test_clamp.py -q
# expected: test_high_edge FAIL, others PASS
git init
git add clamp_kit.py test_clamp.py
git commit -m "red: off-by-one on hi"

If that high-edge test is already green, you cloned the wrong patient. Stop.

The only prompt I allow

Write it down. If you improvise in chat, you will accidentally authorize a cleanup. I do not. The prompt is a contract.

Fix the off-by-one in clamp() so the high bound is inclusive.
Do not rename symbols.
Do not touch imports.
Do not edit tests.
Return a unified diff against clamp_kit.py only.

Then I apply the diff the boring way. No “agent workspace.” No extra files unless the scorer says the model smuggled them.

git apply --check model.patch && git apply model.patch
python -m pytest test_clamp.py -q
git diff --numstat HEAD --

git apply --check is the bouncer. If the model wrapped the patch in markdown fences or invented a second file, the apply fails. Good. That failure is data.

Scorer: geometry first, green second

score_blast.py is the original artifact. It does not grade English. It grades whether the tree moved more than the bug.

# blast_lab/score_blast.py
import ast, subprocess, sys, pathlib

ROOT = pathlib.Path(__file__).parent
ALLOWED = {"clamp_kit.py"}

def git(*args):
    return subprocess.check_output(["git", *args], cwd=ROOT, text=True)

def changed_files():
    out = git("diff", "--name-only", "HEAD", "--").splitlines()
    return [f for f in out if f]

def numstat():
    rows = []
    for line in git("diff", "--numstat", "HEAD", "--").splitlines():
        a, d, name = line.split("\t")
        rows.append((name, int(a), int(d)))
    return rows

def ast_of(path):
    return ast.dump(ast.parse((ROOT / path).read_text()), include_attributes=False)

def helper_still_named():
    tree = ast.parse((ROOT / "clamp_kit.py").read_text())
    names = [n.name for n in tree.body if isinstance(n, ast.FunctionDef)]
    return "_noisy_helper" in names and "clamp" in names

def unused_math_still_present():
    src = (ROOT / "clamp_kit.py").read_text()
    return "import math" in src

def main():
    files = changed_files()
    stats = numstat()
    added = sum(a for _, a, _ in stats)
    deleted = sum(d for _, _, d in stats)
    extra = [f for f in files if f not in ALLOWED]
    pytest_rc = subprocess.call(
        [sys.executable, "-m", "pytest", "test_clamp.py", "-q"], cwd=ROOT
    )
    report = {
        "tests_green": pytest_rc == 0,
        "files_touched": files,
        "extra_files": extra,
        "added_lines": added,
        "deleted_lines": deleted,
        "helper_preserved": helper_still_named(),
        "bait_import_preserved": unused_math_still_present(),
        "line_budget_ok": added + deleted <= 4,
        "file_budget_ok": files == ["clamp_kit.py"],
    }
    report["pass"] = all([
        report["tests_green"],
        report["file_budget_ok"],
        report["line_budget_ok"],
        report["helper_preserved"],
        report["bait_import_preserved"],
        not report["extra_files"],
    ])
    print(report)
    sys.exit(0 if report["pass"] else 1)

if __name__ == "__main__":
    main()

Four added-or-deleted lines is mean on purpose. The honest fix is roughly hi - 1 to hi. If the model also deletes import math because a linter ghost whispered, that is a miss. The bait import is not tech debt in this lab. It is a tripwire. Would you like a model that cannot tell those apart?

Run it after every apply:

python score_blast.py; echo $?
git checkout -- .

Reset is mandatory. Otherwise attempt three inherits attempt two’s “cleanup” and you will congratulate yourself for a mess you authored.

How I read the scoreboard

I narrate four buckets, because a single pass/fail hides the interesting corpse.

If tests stay red and the diff is huge, the model thrashed. That is noisy, but it is honest noise. You see the fire.

If tests go green and only clamp_kit.py moved inside the line budget, that is the only pass I accept. Rare on unconstrained chats. Less rare if you refuse anything that is not a unified diff.

If tests go green and the helper got renamed, you shipped a compatibility break to every caller that imported _noisy_helper. No, it was “private.” Yes, somebody still imported it. You know they did.

If tests go green, import math vanished, and a second file appeared — a README, a utils.py, a rewritten test — that is the kitchen-mop. The model optimized for looking diligent. Your git log will not forgive you.

A compact matrix, because I still need one artifact you can paste into a lab notebook:

Signal	Pass	Fail
pytest	`test_high_edge` green, others still green	any new red, or the original red remains
files	exactly `clamp_kit.py`	tests edited, extras added
lines	add+del ≤ 4	drive-by cleanup
symbols	`_noisy_helper` and `clamp` keep names	“readable” rename
bait	`import math` still there	linter-brain deleted it

Copy that table into the PR template if you must. Do not copy the model’s apology.

What I expect to break (labeled hypotheses)

These are predictions, not measurements I am pretending I already published.

Hypothesis A: unconstrained chat will delete the unused import almost every time, because training data worships tidy examples. The scorer should fail that even when pytest is green.

Hypothesis B: asking for “production quality” in the prompt will explode the file budget. The model will invent IdempotentClamp and a docstring novel. That is not a fix. That is a product pitch.

Hypothesis C: a free remote will occasionally return a truncated diff. git apply --check fails. Treat that as infrastructure, not intelligence. Rerun once. If it still truncates, log the byte length and move on. Do not hand-stitch the rest from memory. You will invent the rest.

Hypothesis D: if you let the model edit tests, it will “fix” test_high_edge to match hi - 1. Green suite. Wrong product. That is why tests are not in ALLOWED.

If your run contradicts these, keep the trace. A protocol that cannot lose is marketing.

Limitations, and who should walk away

This harness does not prove the model can maintain a monorepo. It proves whether a one-line request stayed a one-line diff on a deliberately messy 40-line file. That is a lower bound. Lower bounds are useful. They are not capacity planning.

The line budget of four is arbitrary. A necessary formatting wrap might burn it. If you raise the budget, raise it in git, not in your head after you liked the explanation.

AST-by-ast.dump will miss some equivalent edits and flag some harmless ones. I still prefer it to reading the model’s paragraph about “minor cleanups.”

Who should not use this approach: anyone stuffing customer source onto a free shared server; anyone who needs a latency SLA; anyone replacing code review with a Python dict named report. If your org already has contract tests and diff limits in CI, you do not need my mug metaphor. Wire the same checks to the merge gate and skip the chat theater.

Also skip this if you cannot reset the tree. A lab that cannot git checkout -- . will slowly become the model’s unfinished novel.

The only conclusion I will defend

Green tests are a weak invariant for coding models. Blast radius is a stronger one. Ask for one line. Score the files. Score the names. Score the bait. Reset. Repeat until the patch is boring.

Boring patches are the product. Everything else is a volunteer refactor you did not request. If you want a cheap place to grind that loop without burning a workstation, the free model plus free server option above is enough to start — steal the scorer first, then decide whether the remote is even interesting.