A familiar scene plays out in a team channel: someone pastes a forty-line refactor that an assistant produced on the first try, and the room quietly agrees the tool is ready for real work. Two weeks later the same tool opens pull requests that ignore the team's error-handling convention, and nobody can explain the gap. The gap is not mysterious, because the first run measured a demo and the second run measured a workflow. What follows is a myth-busting FAQ about that gap, plus a small canary harness you can run yourself before you standardize on anything.
Myth 1: a strong one-shot output means a strong model
The demo prompt was written by someone who already knew the answer, it usually contained the one relevant file, and the output you saw survived a private selection step. What you measured is the author's skill at prompting plus one lucky sample, not the model's behavior under your conventions. A single output is an anecdote about a distribution, and you need a dozen attempts at tasks you did not author before the distribution is worth describing. The cheap correction is to keep the prompts you did not write and count how many survive review unchanged.
Myth 2: a bigger token allowance means more shipped work
Tokens are an input to the process, not an output of it, so an allowance tells you how many attempts you can afford rather than how many patches will land. The number worth tracking is yield: accepted patches per million tokens, measured on tasks whose acceptance criteria you fixed in advance. Teams that track yield usually discover their expensive prompts are expensive for context they never needed, which is a fixable habit rather than a model limitation.
MonkeyCode's free tier, as its operator describes it, pairs free model access with a free server option, which makes it a practical place to run this kind of measurement without paying per seat while you calibrate. Disclosure: This article was prepared as part of MonkeyCode's product outreach. One caution before you plan a budget around any allowance: read the current quota on the product page, because free-tier numbers move faster than blog posts do, including this one.
Myth 3: a green test run means the patch is safe
Tests fail to catch a change for boring reasons: the file has no coverage, the assertion was written by the same agent minutes earlier, or the suite never exercises the branch that changed. A passing run therefore proves only that your selected checks still pass, which is a weaker claim than most reviewers hear. The correction is to pair every task with a check command you wrote before the agent ran, and to record the check exit code separately from the agent exit code.
Myth 4: the free server is the variable that matters most
Environment matters, but it is one variable among several: model, prompt, repository state, check selection, and wall-clock limits all move results too. If you vary the model and the machine at once, you cannot attribute the difference to either, and the argument in review becomes unfalsifiable. Pinning the environment while you vary one thing is the entire discipline, and a free server is useful precisely because it is cheap to hold constant across runs.
Myth 5: public leaderboard rankings transfer to your repository
Leaderboards reward tasks with short prompts, unambiguous answers, and no house style, which is the opposite of most production tickets. Your repository encodes conventions that never appear in a benchmark: naming, module boundaries, migration order, and the reviewer who rejects a diff for touching three files. Treat rankings as a filter for candidates and your own suite as the decision, because only the second one sees your conventions.
The artifact: a twelve-task canary suite with a token ledger
The harness below is a reference implementation, not something I executed for this article, so treat the code as a starting point you run and correct. Each task pins a base commit, a prompt, an agent command, and a check command you wrote yourself, and every run appends one JSON line so the ledger survives across sessions.
# canary/run_canary.py - reference implementation, unexecuted here
import json, subprocess, time, hashlib, pathlib, argparse
TASKS = json.loads(pathlib.Path("canary/tasks.json").read_text())
OUT = pathlib.Path("canary/results.jsonl")
def sh(cmd, cwd):
start = time.time()
p = subprocess.run(cmd, cwd=cwd, shell=True, capture_output=True, text=True)
return p.returncode, (p.stdout + p.stderr).strip(), round(time.time() - start, 2)
def run(task, model, root):
work = root / f"{task['id']}-{int(time.time())}"
subprocess.run(["git", "worktree", "add", "--detach", str(work), task["base_sha"]], check=True)
agent_exit, agent_log, secs = sh(task["agent_cmd"].format(model=model, prompt=task["prompt"]), work)
check_exit, check_log, _ = sh(task["check_cmd"], work)
diff = sh("git diff --shortstat", work)[1]
return {
"task": task["id"], "model": model, "base_sha": task["base_sha"],
"prompt_sha": hashlib.sha256(task["prompt"].encode()).hexdigest()[:12],
"agent_exit": agent_exit, "check_exit": check_exit,
"tokens": task.get("tokens_reported"), "seconds": secs, "diff": diff,
}
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("--model", required=True)
ap.add_argument("--repeat", type=int, default=1)
ap.add_argument("--only", default=None)
args = ap.parse_args()
root = pathlib.Path("canary/work")
root.mkdir(parents=True, exist_ok=True)
with OUT.open("a") as fh:
for task in TASKS:
if args.only and task["id"] != args.only:
continue
for _ in range(args.repeat):
row = run(task, args.model, root)
fh.write(json.dumps(row) + "\n")
print(row["task"], "check_exit=", row["check_exit"], row["diff"])
A single task definition is small enough to review in a pull request, which is what keeps the suite honest over time.
[
{
"id": "retry-boundary",
"base_sha": "<commit sha pinned by the team>",
"prompt": "Add retry with exponential backoff to the HTTP client; do not change public signatures.",
"agent_cmd": "monkeycode run --model {model} --prompt {prompt}",
"check_cmd": "pytest -q tests/test_retry.py && python scripts/lint_conventions.py",
"tokens_reported": null
}
]
Run it twice against the same pinned commits, first with a model you trust and then with a candidate, and keep both files: python canary/run_canary.py --model <trusted-id> --repeat 2 followed by python canary/run_canary.py --model <candidate-id> --repeat 2. The repeat is not decoration, because two runs of the same task already tell you whether the check is stable. If the trusted model fails a task twice, your task is broken, and you should fix the task before you judge anyone.
python - <<'PY'
import json, collections
rows = [json.loads(l) for l in open("canary/results.jsonl")]
by_model = collections.defaultdict(lambda: {"runs": 0, "accepted": 0})
for r in rows:
b = by_model[r["model"]]
b["runs"] += 1
b["accepted"] += 1 if r["check_exit"] == 0 and r["agent_exit"] == 0 else 0
for model, b in by_model.items():
print(model, b["runs"], "runs", round(b["accepted"] / b["runs"], 2), "accepted rate")
PY
Reading the numbers without overreading them
Suite size determines what you may claim, and the table below is the part most teams skip before drawing conclusions.
| Tasks | What it can support | What it cannot support |
|---|---|---|
| 5 | Spotting a broken prompt or an unusable environment | Any ranking between two similar models |
| 12 | A repair decision on one workflow, with named failures | Regression claims across repositories |
| 30+ | Weaker but repeatable comparisons, if tasks are stable | Vendor claims you publish as general truth |
Twelve tasks is enough to learn that a candidate fails your migration convention on three of them, which is actionable, and it is not enough to declare one model better than another by a few percentage points. Report the failures by name rather than the aggregate rate, because reviewers act on "missed the retry boundary" and ignore "87 percent accepted".
Limitations and who should skip this
This harness assumes you already have a test command worth running, so teams without meaningful coverage should invest there first and treat the agent as a draft generator meanwhile. Small prototypes gain little from a twelve-task suite, because the maintenance cost of pinned commits and stable checks exceeds the value of the comparison. Anyone needing vendor-grade statistics, confidence intervals, and cross-repository generalization needs a much larger suite and should not cite a canary run as evidence. The suite also rots quietly when dependency updates break old tasks, so schedule a monthly run against your trusted model and delete tasks that no longer discriminate.
If you want to try the harness without provisioning anything first, MonkeyCode's free server option is one reasonable place to hold the environment constant while you vary the model, and publishing the failures alongside the wins is what makes the ledger worth reading.
Top comments (1)
"The first run measured a demo and the second measured a workflow" is the clearest framing of this I've seen. Counting how many outputs survive review unchanged, on prompts you didn't write, is a metric any team can start this week.