DEV Community: Emery Chen

A Free Token Does Not Buy a Blind Retry

Emery Chen — Sun, 11 Oct 2026 21:47:38 +0000

A free retry is still a decision you own.
Cheap tokens do not erase a repeated mistake.
You need a stop rule before the first call.

The claim you can test

Do not retry an agent only because the call is free.
A free allowance changes the bill, not the evidence.
Evidence is the only reason a second call exists.

This piece argues from failure classes, not from scores.
It does not rank models, vendors, or hosted runners.
You can reject the opinion after you count your own logs.

Why retry talk is the wrong comfort

Public posts this week keep circling retries and swaps.
Treat those titles as untrusted signals, not as facts.
No headline here is used as a measurement.

Better inference does not dissolve a local lock.
Missing files and wrong paths stay on your machine.
A remote model cannot invent a file you never sent.

What free access tempts you to skip

A paid meter used to force a short pause.
You read the error before you spent again.
A free pool removes that pause unless you rebuild it.

That pause was doing classification work for you.
It made you name the failure in plain words.
Without a name, you resend the same prompt again.

Excuses that should not spend a call

Reject a retry that rests on these excuses.
Write the excuse down, then throw the call away.

The last reply looked almost right to you.
The log was long, so another summary feels due.
The allowance is large, so waste feels harmless.

Each excuse avoids naming even one new fact.
A retry without a new fact is noise.
Noise gets cheaper when the meter reads zero.

Classify the failure before the network

Read the failure on your own machine first.
Put it in one class from the table.
Only then decide whether the network is owed.

Rows you can apply today

Class	What you saw	Retry	Why
Local lock	Dependency or file lock	No	Another call will not release it
Missing input	Empty diff or absent file	No	Fix the fixture, then call once
Tool mismatch	Command is not on PATH	No	Repair the host before any rephrase
Flaky check	Local rerun already passed	Once	Record the flake and cap it
Wrong file	Edit landed outside the brief	No	Narrow the path list first
New compiler fact	Fresh error after a real edit	Yes	The error text is new evidence

Copy this table as a starting proposal only.
It is not a recorded result from production traffic.
Change every row until it matches your repo.

Keep a referee next to the table

Keep the referee on the machine you trust.
Keep the classifier in code you can diff.
The script below is an unexecuted local example.

#!/usr/bin/env python3
"""Local retry referee. Proposal only. Not a recorded run."""

from dataclasses import dataclass

STOP = "stop"
ONCE = "once"
ALLOW = "allow"

@dataclass(frozen=True)
class Verdict:
    action: str
    reason: str

def classify(kind: str, evidence_changed: bool, attempts: int) -> Verdict:
    local = {"lock", "missing_input", "tool_mismatch", "wrong_file"}
    if kind in local:
        return Verdict(STOP, "local fact is unchanged")
    if kind == "flake" and attempts < 1:
        return Verdict(ONCE, "one local rerun, then stop")
    if kind == "new_compiler_fact" and evidence_changed and attempts < 2:
        return Verdict(ALLOW, "new error text justifies one more call")
    return Verdict(STOP, "no new evidence or cap reached")

if __name__ == "__main__":
    samples = [
        ("lock", False, 0),
        ("missing_input", False, 0),
        ("flake", False, 0),
        ("flake", False, 1),
        ("new_compiler_fact", True, 0),
        ("new_compiler_fact", True, 2),
        ("wrong_file", True, 0),
        ("unknown", True, 0),
    ]
    for kind, changed, attempts in samples:
        verdict = classify(kind, changed, attempts)
        print(f"{kind} changed={changed} attempts={attempts} -> {verdict.action}: {verdict.reason}")

Save the file as retry_referee.py beside your repo.
Run the samples before you trust the gate.

python3 retry_referee.py

What those samples should show

Expect stop on locks, missing files, and wrong paths.
Expect one local chance when the class is flake.
Expect allow only when compiler text actually changed.

If the kind is unknown, the referee must stop.
An unknown kind is not a prompt to invent a class.
You add a row, then you run the samples again.

Put the referee in front of the call

Wrap the referee in a thin shell gate.
The gate reads a class you already wrote down.
It refuses the network when the verdict is stop.

#!/usr/bin/env bash
set -euo pipefail

kind="${1:-unknown}"
changed="${2:-false}"
attempts="${3:-0}"

action="$(python3 -c '
import sys
from retry_referee import classify
kind, changed, attempts = sys.argv[1:]
verdict = classify(kind, changed == "true", int(attempts))
print(verdict.action)
' "$kind" "$changed" "$attempts")"

if [[ "$action" != "allow" && "$action" != "once" ]]; then
  echo "refused: ${kind} has no new evidence"
  exit 2
fi

echo "permitted: ${action} for ${kind}"
# Place one real model call below this line.
# Do not hide a retry loop inside the client.

Save that file as retry_gate.sh next to the referee.
Try the gate with a lock before you try a model.

bash retry_gate.sh lock false 0
bash retry_gate.sh new_compiler_fact true 0

Run both commands from the directory that holds the scripts.
The first command should exit with status two.
The second should print a permit and then wait.

This gate is a sketch, not a finished product.
Label it unexecuted until it runs on your tree.
A printed permit is not a passing integration test.

Spend free access only after allow

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project for coding agents.
The outreach brief states free model access for readers.
It also states a free server option you can try.

The outreach brief states a free allowance of ten million tokens.
Confirm the live figure in the current project docs.
Do not budget a quarter from a copied number.

Free access can change without this article updating.
Model names, hardware, and duration are not assumed.
Permanence is not a feature you should plan on.

Use the free model only after the referee allows it.
Use the free server to host the agent loop.
Do not skip local classification because the host is free.

A practical order keeps the pool for new evidence.
Write the class down before you open a remote session.

Classify the failure on your own machine first.
Refuse the call when the class is local.
Send one bounded prompt only when evidence changed.
Re-run the same local checks after the patch.
Stop at two remote attempts even if tokens remain.

Two remote attempts is a hard cap in this opinion.
Leftover tokens are not an argument for a third call.
Stop even when the allowance still looks large.

Shrink the prompt the referee allows

When the referee allows a call, shrink the prompt.
Send the new error, the paths, and the last diff.
Do not resend the repository because the call is free.

Failure class: new_compiler_fact
Attempts so far: 1
Changed evidence: yes
Touched paths:
- src/retry_gate.py
New compiler text:
- src/retry_gate.py:40: NameError: verdict
Constraint:
- Edit only the touched paths.
- Do not add a retry loop.
- Stop after one patch.

If you cannot point at a new line, do not send it.
Abundance is not the same thing as context.
Context means the fact that changed since last time.

The opinion, without a hedge

Cheap inference rewards impatience more than careful work.
You should grow stricter as the meter disappears.
The stop rule matters most when retries feel free.

A free server does not outrank your worktree.
A large token pool does not outrank a missing file.
Another paraphrase does not outrank a held lock.

If you disagree, sample five failures from last week.
Count how many retries carried no new evidence.
Let that count accept or reject this opinion.

Limits that stay attached to the rule

This workflow does not measure model quality at all.
It does not compare hosts, prices, or latency.
It does not prove a free tier will remain available.

The script knows only the classes you pass in.
A wrong class produces a confident wrong stop.
You still have to read the log yourself.

Network flakes and partial writes need their own rows.
Add those rows before a shared branch uses the gate.
Never paste secrets or customer logs into the sample.

This draft was not executed against a live hosted agent.
Treat every command here as a local proposal.
Verify install steps and terms in the project docs.

A free server still needs your auth and your retention rules.
This draft does not describe those controls for you.
Read them before any source leaves your laptop.

Who should leave this workflow alone

Skip this when the edit is a typo you already see.
A referee is wasted on a fix you can type.
Human eyes are enough for that narrow case.

Skip this if you will not maintain the class list.
An abandoned table becomes theater within a week.
An unused table will not protect a shared branch.

Skip this if policy forbids a third-party agent server.
A free option is still a remote service.
Read the terms before you send repository source.

Skip this if you need audited numbers for a decision.
This article offers a rule, not a study.
Do not cite it as a performance result.

One check before you try the pool

Label five recent failures with the table above.
Run the referee against those labels before any call.
Call a free model only if the verdict is allow.

If current project terms still match the brief, use them.
If they do not, keep the referee and skip the host.
The stop rule stays on your machine either way.

Do Not Let the Free Model Grade Its Own Work

Emery Chen — Sat, 10 Oct 2026 18:27:15 +0000

A free model call is a sample, not a verdict.
You should freeze every fail rule before that call.
If the writer model grades itself, you learned nothing.

The position

Free access does not make a weak check valid.
It only makes that weak check cheaper to repeat.
You still need a grader that you wrote yourself.

Keep a hard split between the writer and the judge.
One model may draft the change you asked for.
A separate program must accept or reject that change.

Headlines are a bad oracle

You may see a small model post a clean run.
People then stop looking at the judge behind it.
That is the wrong lesson from a cheap success.

Public model names shift every few product weeks.
Prices and limits shift along with those names.

Your review should not depend on today's brand.
Pin the rubric instead of the weekly headline.

No fresh benchmark table was supplied for this draft.
Treat every quota line and speed line as unverified.

Read the vendor page on the same day you run.
Do not paste old numbers into a release note.

What a free stack may do

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The operator states two availability facts, and nothing more.
MonkeyCode offers free model access for you to try.

MonkeyCode also offers a free server option to try.
This draft invents no quota, hardware, or duration.

Use that free model as a writer, and nothing more.
Use that free server as a clean process boundary.

Keep the grader on a machine that you control.
Do not ask either service to define success for you.

If either offer disappears, this method still stands.
The files below do not require that specific vendor.
Swap the writer endpoint and keep the same judge.

Confirm the project license on the official page yourself.
Do not vendor a client from a memory of last month.

Freeze three files before any token

You need three local files before any network call.
Write each file by hand, then hash the rubric.

The writer model may read those frozen files.
The writer model must not edit the fail rules.

task.md holds the job in plain, testable language.
rubric.json holds fail rules with stable public ids.
fixture/ holds inputs the writer is forbidden to replace.

Record those hashes in freeze.txt before you call out.
If a later hash differs, you throw that run away.
A rubric edited after the call is a new experiment.

Put the verdict in code

A prompt that says be strict is not a verdict.
The model can agree with that line and still drift.
An exit code cannot flatter you and then drift.

Field contract

You should require three fields, and ignore extra keys.
files_touched must be an integer count of edited paths.
stdout must be captured text, not a model essay.

paths must be the list the prefix rule reads.
If a required field is missing, exit non-zero at once.
A partial JSON object is a failed contract, not a pass.

Proposal grader, not a logged run

The script below is a proposal, not an executed log.
Nothing here was run against a live vendor endpoint.
Copy it, then point it at your own fixture tree.

Expect failures until your paths match this layout.
This local proposal targets Python 3.10 or newer.

#!/usr/bin/env python3
"""Local grader. Proposal only. Does not call a network."""

import hashlib
import json
import sys
from pathlib import Path

ROOT = Path(".").resolve()
RUBRIC = ROOT / "rubric.json"
FREEZE = ROOT / "freeze.txt"
OUTPUT = ROOT / "out" / "result.json"
REQUIRED = ("files_touched", "stdout", "paths")


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def load_freeze() -> dict:
    pairs = {}
    for line in FREEZE.read_text(encoding="utf-8").splitlines():
        if not line.strip():
            continue
        name, digest = line.split()
        pairs[name] = digest
    return pairs


def assert_frozen() -> None:
    frozen = load_freeze()
    current = sha256(RUBRIC)
    expected = frozen.get("rubric.json")
    if current != expected:
        sys.exit("rubric changed after freeze; discard this run")


def missing_keys(payload: dict) -> list:
    return [key for key in REQUIRED if key not in payload]


def check(rule: dict, payload: dict) -> str | None:
    kind = rule["kind"]
    if kind == "max_files":
        if int(payload["files_touched"]) > int(rule["limit"]):
            return rule["id"]
    elif kind == "must_include":
        if rule["token"] not in payload["stdout"]:
            return rule["id"]
    elif kind == "forbidden_path":
        for path in payload["paths"]:
            if str(path).startswith(rule["prefix"]):
                return rule["id"]
    else:
        sys.exit(f"unknown rule kind: {kind}")
    return None


def main() -> int:
    assert_frozen()
    rubric = json.loads(RUBRIC.read_text(encoding="utf-8"))
    payload = json.loads(OUTPUT.read_text(encoding="utf-8"))
    absent = missing_keys(payload)
    if absent:
        print(json.dumps({"missing": absent}))
        return 1
    failed = []
    for rule in rubric["rules"]:
        hit = check(rule, payload)
        if hit:
            failed.append(hit)
    report = {"failed": failed, "rule_count": len(rubric["rules"])}
    print(json.dumps(report, indent=2))
    return 1 if failed else 0


if __name__ == "__main__":
    raise SystemExit(main())

A matching rubric can stay this small on purpose.
Three rules are enough to prove the judge can fail.

{
  "rules": [
    {"id": "F1", "kind": "max_files", "limit": 3},
    {"id": "F2", "kind": "must_include", "token": "EXIT=0"},
    {"id": "F3", "kind": "forbidden_path", "prefix": "secrets/"}
  ]
}

Commands that enforce the order

Run the hash step before any writer process starts.
Do not skip that hash, even on a tiny fixture.
Do not let the writer process touch rubric.json at all.

mkdir -p out fixture
python3 - <<'PY'
import hashlib
import pathlib
path = pathlib.Path("rubric.json")
digest = hashlib.sha256(path.read_bytes()).hexdigest()
out = pathlib.Path("freeze.txt")
out.write_text("rubric.json %s\n" % digest, encoding="utf-8")
PY
# Writer step stays outside this page.
# Save structured writer output to out/result.json.
python3 grade.py
echo "grader_exit=$?"

That shell comment is deliberate, not an omission.
This page does not ship a live vendor client.
You already know your endpoint, or you do not.

Add a client only after the grader passes a fake file.
A fake file is a hand-built result, not a model reply.

Known pass, then known fail

Build out/result.json by hand before any model call.
You want a known pass, then a known fail.
That pair proves the judge can say both words.

{
  "files_touched": 2,
  "stdout": "built fixture\nEXIT=0\n",
  "paths": ["src/app.py", "fixture/input.txt"]
}

Change files_touched to 9 and run the grader again.
You should see rule F1 and a non-zero exit.
If that edit still exits zero, your judge is blind.

Point one path at secrets/keys and expect rule F3.
A missing fail on that path means the prefix check is dead.

Two fixtures, one judge

You do not need another model to challenge the writer.
You need two frozen inputs and one local diff.
If the outputs disagree, keep both and merge neither.

Run the writer on fixture A and save the output.
Run the writer on fixture B with the same rubric.
Grade each output file with that same frozen rubric.
Diff the two grader reports, not the two chat stories.

A disagreement is a result, not a reason to relax rules.
You inspect both diffs, then you pick neither automatically.

How you should read the exit

A clean exit only clears your frozen rule list.
It does not close design review or threat review.
You still read every changed path with your own eyes.

Result you see	Move you make	Reason to move
Exit 0 and the freeze still matches	Keep the diff for human review	The sample met rules you wrote
Exit 1 with known rule ids	Repair the diff and leave the rules	The writer missed a frozen check
Freeze hash does not match the file	Discard the run and start again	The experiment changed mid-flight
`out/result.json` is missing entirely	Fix the writer output contract	Silence must never count as a pass
Grader prints an unknown rule kind	Stop and repair the rubric first	A vague judge is not a judge

Refuse the self-graded demo

You should refuse any demo that grades itself.
A self-score from the writer is only a story.
Those stories can stay inside a chat window.

They are not evidence inside a pull request.
Free tokens make that story easier to produce.
That ease is the danger, not the gift itself.

Spend the free writer on alternate drafts only.
Spend your own code on the verdict every time.

A free server helps only as a process boundary.
It can run the writer away from your laptop.
It should not hold the only copy of the rubric.

If you cannot re-run the grader offline, you stop.
Offline means the rubric and the grader stay local.
A server copy alone is not a review record.

What this check will not catch

This grader checks output shape, not writer intent.
A bad rule will happily pass a bad patch.

You can freeze a mistake just as easily as a truth.
Review the rubric text before you trust any exit.

Free access can change without notice or your consent.
Do not plan a release around an unnamed quota.

Do not put production secrets inside the fixture tree.
Do not send customer rows to a trial host.

The script ignores concurrency, retries, and clock skew.
It will not catch a writer that flakes between calls.

Run the same frozen input two times in a row.
If those two outputs differ, you have no result yet.

Who should not use this

Skip this flow if you need a contracted uptime promise.
Skip it when data cannot leave your private network.
Skip it when nobody on the team can read the rubric.
Skip it when you wanted a score, not a fail list.

Also skip it for medical, legal, or payment logic.
Those domains need stronger oracles than token rules.

A string check will not save a bad payment path.
Build a real suite before you spend any model call.

One next step, then stop

Read the current MonkeyCode access terms yourself today.
Confirm the free model offer and the free server offer.
Then run this grader against a fake result file first.
Spend a writer call only after that local pass holds.

Spend the Free Allowance on Hostile Fixtures

Emery Chen — Fri, 09 Oct 2026 14:53:06 +0000

Spend the free allowance on hostile fixtures only. Do not spend it on flattering model summaries. A summary that praises your diff is cheap comfort.

A fixture that breaks your spec is the actual work. That is the position, not a soft suggestion. If your free-tier log is mostly recap text, you wasted the pool.

Say the rule before the first call

You get a free model path and a free server. You also get a strong urge to request a nice summary. Refuse that urge before you write the first prompt.

Ask for a contradiction inside the spec instead. Ask for a missing field or a broken boundary. Then store the failure beside the claim you named.

A kind paragraph does not stress any contract. A hostile fixture does stress the contract directly. You should point at the broken case during review.

Why summaries feel like progress

Summaries are fast, and they read like finished judgment. You paste them into the pull request body. Reviewers see fluent prose and relax their questions.

The spec never met a single counterexample there. Free access makes this failure cheaper to repeat often. There is no invoice to embarrass the tenth recap.

You can generate twenty summaries and learn nothing new. That pattern is decoration, not real technical exploration. Hostile fixtures burn tokens because each call falsifies one claim.

If the claim survives, you learned one small thing. If the claim fails, you learned something you can file. Either result beats another flattering recap of the diff.

What hostile means here

Hostile does not mean illegal, abusive, or aimed outward. It means unfriendly to your own written specification. You are attacking the contract, not a person or company.

Use three fixture families and keep each family boring. Boring fixtures are easier to rerun after a model move. Clever prompts are harder to defend in review later.

Three families, nothing cleverer

Contradiction means two rules in the spec cannot both hold.
Omission means a required field, state, or error path is missing.
Boundary means an empty input, duplicate event, or hard limit appears.

Do not ask the model to invent crimes or exploit steps. Stay inside your repository and your documented rules only. If a fixture needs a real secret, stop and use a fake.

A harness you can copy

The script below is an unexecuted local example only. It does not call a model or open a socket. It checks that your log spent calls on hostile classes.

You run it locally before anyone cites the session. Adapt the filename if your repo uses another path. Do not treat a green exit as proof of product truth.

#!/usr/bin/env python3
"""Reject a free-tier log that is mostly summaries. Unexecuted example."""
import json
import sys
from collections import Counter
from pathlib import Path

ALLOWED = {"contradiction", "omission", "boundary"}
BANNED = {"summary", "praise", "rewrite"}

def load(path: Path):
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        if line.strip():
            rows.append(json.loads(line))
    return rows

def main() -> int:
    if len(sys.argv) != 2:
        print("usage: hostile_gate.py calls.jsonl")
        return 2
    rows = load(Path(sys.argv[1]))
    if len(rows) < 3:
        print("FAIL: need at least three hostile attempts")
        return 1
    kinds = Counter()
    for i, row in enumerate(rows, start=1):
        kind = str(row.get("fixture", "")).strip().lower()
        claim = str(row.get("claim_id", "")).strip()
        if not claim:
            print(f"FAIL: row {i} has no claim_id")
            return 1
        if kind in BANNED:
            print(f"FAIL: row {i} is a {kind}, not a fixture")
            return 1
        if kind not in ALLOWED:
            print(f"FAIL: row {i} fixture {kind!r} is not allowed")
            return 1
        kinds[kind] += 1
    missing = ALLOWED - set(kinds)
    if missing:
        print("FAIL: missing families: " + ", ".join(sorted(missing)))
        return 1
    print("OK " + " ".join(f"{k}={kinds[k]}" for k in sorted(kinds)))
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

python3 hostile_gate.py calls.jsonl
echo "exit=$?"

A synthetic log you can run

Create a synthetic log before you point the gate at real calls. These three lines are fake fixtures, not live model output. They exist so you can see a passing exit on your machine.

mkdir -p spec-lab && cd spec-lab
cat > calls.jsonl <<'EOF'
{"claim_id":"retry-once","fixture":"contradiction","input":"spec says at-least-once and exactly-once","survived":false,"note":"both sentences are present"}
{"claim_id":"retry-once","fixture":"omission","input":"no rule for a null receipt","survived":false,"note":"null path is unspecified"}
{"claim_id":"retry-once","fixture":"boundary","input":"duplicate delivery","survived":false,"note":"handler applied the side effect twice"}
EOF
python3 hostile_gate.py calls.jsonl

What a passing log must contain

Three logged calls are a minimum, not a quality score. You still need one contradiction, one omission, and one boundary. A log of fifty summaries fails this gate immediately.

That immediate failure is the feature you wanted. It stops you from mistaking volume for coverage. Add more fixtures after the three families already exist.

The line format

Keep each attempt on one single JSON line. Name the claim you tried to break in that call. Name the fixture family and whether the spec survived.

{"claim_id":"retry-once","fixture":"boundary","input":"duplicate delivery","survived":false,"note":"handler applied the side effect twice"}

{"claim_id":"retry-once","fixture":"contradiction","input":"spec says at-least-once and exactly-once","survived":false,"note":"both sentences are present"}

{"claim_id":"retry-once","fixture":"omission","input":"no rule for a null receipt","survived":false,"note":"null path is unspecified"}

Quote the spec line in the note when you need a pointer. Do not store customer payloads or live credentials there. Do not store prompts copied from other people's private chats.

Check the file before you cite it

Check that every line parses before you trust the mix. Then run the gate and keep the printed exit code. Neither command proves your product behavior is correct.

They prove you spent the session trying to break a claim. That is a lower bar than truth, and it is still useful. A session without that bar is just generated prose.

python3 -c 'import json,sys; [json.loads(line) for line in sys.stdin if line.strip()]' < calls.jsonl
python3 hostile_gate.py calls.jsonl
echo "exit=$?"

Decision table for the allowance

Use this table before you send another free call. If the row says stop, you stop without bargaining. Do not bargain with a fluent model answer.

You are about to ask for	Spend a free call?	Why
A short summary of the diff	No	Decoration, not a fixture
Praise for the approach	No	Flattery hides missing cases
A contradiction inside the spec	Yes	One family you must cover
A missing error path	Yes	Omission is a real bug class
A boundary input against one claim	Yes	This is the useful burn
A rewrite of the whole module	No	That is a new project, not a test
Exploit steps against a live company	No	Out of scope and harmful
A score that ranks models	No	This harness is not a benchmark

Read the yes rows as your actual token budget. Your free calls should cluster on those rows. If they do not, the allowance taught you nothing.

How a free server fits the habit

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is described here as an open-source coding project. The brief states free model access and a free server option.

The same brief states a free allowance of ten million tokens. Treat that figure as a brief claim, not a measurement. Nobody measured the live quota, model list, or hardware for this piece.

Check the current terms before you depend on any number. Point the free server at this harness, not at release prose. Hostile fixtures consume calls quickly, so a free pool helps.

Keep production secrets off that server even when the price is zero. A zero price does not change your data handling duties. Synthetic inputs belong in the log, not customer records.

Do not invent a model winner from one free session. Model names move, and published limits move with them. Your calls file is the artifact you can still defend.

If you try this, read MonkeyCode's current free-server terms first. Then run the gate against a fresh hostile log. Use the allowance to break one claim already in your repo.

A workflow for one claim

Pick one claim and run the loop below without shortcuts. The order matters because the label comes before the call. If you label the run after you like the answer, you will cheat.

Seven steps, in order

Pick one sentence from your spec and assign a claim id.
Write a contradiction prompt that quotes only that sentence.
Write an omission prompt for a missing field or error path.
Write a boundary prompt using an empty, duplicate, or maximum input.
Send those three prompts to the free server and save every reply.
Mark survived yourself, and do not let the model mark it.
Run the gate and commit the log beside the spec change.

Step six matters more than the wording of the model answer. The model can suggest a possible break in the spec. You decide whether that spec sentence actually failed.

If you skip that human decision, you outsourced judgment. A scratch host should not own your release standard. Keep the decision in the note field, in your own words.

Limits of this habit

This gate counts fixture families, not real-world truth. A silly contradiction can still pass the script cleanly. You must read the note before you trust the OK line.

Three calls can easily miss the dangerous case. The minimum exists to stop summary spam, not to certify coverage. Add a fixture when a bug escapes, and keep the old ones.

Free tiers change without asking for your permission first. A workflow that needs today's pool next month will stall. Store the prompts in git so any later host can rerun them.

The script trusts the fixture label you typed yourself. You can lie in the log and still receive OK. That lie fools a script, not a reviewer who reads the input.

What this is not

This piece reports no latency, accuracy, or savings figure. None of those quantities were measured for this workflow. If a quote adds a number, that number was invented later.

Who should not use it

Skip this habit if your change has no written claim. Hostile fixtures need a sentence you can actually falsify. Pure visual polish often has no such sentence yet.

Skip this if the diff holds secrets or personal data. Do not upload that material to a free server to save money. Rewrite the fixture with synthetic values before any call.

Skip this if counsel has not approved third-party model hosts. A free server is still an external processor until policy allows it. A zero invoice is not a legal argument for uploading code.

Skip this if you wanted a public model ranking. The decision table already forbids that use of the gate. Measure on a set you control, and publish the method separately.

Hold the line in review

Ask one question when a teammate pastes a model summary. Which claim did this call try to break today? If they cannot name the claim id, the call was entertainment.

Ask to see all three families inside the saved log. If a family is missing, send them back to the free bench. Do not block the merge on the model's stated confidence.

Block the merge on a missing fixture you already required. Free tokens are a research stipend for your own spec. Spend them on contradictions, omissions, and hard boundaries.

Leave the flattering summary unsent, unquoted, and out of the review. A free run that only comforts you did not earn a citation. Write the broken case down, or admit you did not look.

Pin the Tool Host Before You Spend a Token

Emery Chen — Thu, 08 Oct 2026 10:27:24 +0000

Stop spending model quota on chats you cannot rerun. A free model call is not an environment. Pin the tool host first, then spend the tokens.

You are not short of demos this month. You are short of proof that another person can rebuild. A transcript from one warm process is a story.

The claim

Here is the position, stated with no hedge. An agent run counts only when both sides are disposable. The model call must be logged as its own record.

The tool host must restart without your personal shell. If either side is "my machine that day," discard the run. Do not cite that run in a pull request.

Do not paste it into a contest writeup as proof. This is not a plea for a bigger cluster. This is a plea for a named host and a frozen contract.

You can hold that bar on a small server. You cannot hold it with a warm laptop session. A session dies, and the proof dies with it.

Why the split matters

Developers keep collapsing two products into one sentence. They say the model itself built the feature. The model emitted text, and a host may have acted.

Those two failures are not the same failure. A fluent model can still call a missing binary. A healthy host can still receive a broken call.

If you grade them as one blob, you learn nothing. You only learn that one afternoon was kind to you. Kind afternoons do not survive a clean reboot.

Agent contests reward a screenshot more than a boot log. Vibe-coded side projects reward that same bad habit. The missing artifact is a host you can boot again.

What pinned means

Pinned does not mean large, costly, or permanent. Pinned means a stranger can rebuild the tool side. They should not need your shell history to do it.

Use this checklist before you spend any quota.

Give the host a name that is not your laptop.
Keep the tool list in a versioned file.
Give each tool a timeout and a real error path.
Keep secrets in the host environment, not the transcript.
Write a manifest before you write any verdict.
Reboot the host and confirm the contract still holds.

Miss one item and the run is only a rehearsal. Rehearsals are useful while you design the loop. Rehearsals are not evidence you can cite later.

Score the host before the model

Score the host before you argue about model quality. If the host column fails, stop spending tokens. Model taste cannot repair a missing boot path.

Question	Pass	Fail
Can you name the host image?	Image id is in the manifest	"My dev box"
Are tools declared outside the prompt?	Schema file, versioned	Tools described in chat
Does a tool crash return structured error?	Exit code plus stderr field	Empty string or hang
Can a second person boot the host?	Documented start command	Needs your dotfiles
Is the model call logged separately?	Request id, token count, latency	Only the final answer
Did you rerun after a host restart?	Same schema, new process	One warm process only

Two failed rows mean you do not have a result. You have a demo that happened to look finished. Demos can guide the next hour of work.

They should not guide a merge decision alone. A green answer on a dead host is still a miss. Write the miss down, then fix the host.

The order you should follow

Follow this order, and do not swap the last steps forward. Quota spent before a restart is a sunk demo. The model cannot refund that kind of confusion.

Put the tool schema in the repo and review it like code.
Boot the tool host on a named machine you can restart.
Run the harness and demand a manifest on disk.
Restart the host and demand a second manifest.
Attach model access and append one logged call.
Score the table, then decide whether the answer matters.

Three failures this rule catches

Warm process

The warm-process failure looks like a clean success. Your tools stay imported, so the second call is cheap. A restart would have shown the missing package.

Secret in the prompt

Putting a secret in the prompt looks like convenience. The model repeats a token it should never have seen. The host should have read that secret from its own env.

Schema only in chat

Keeping the schema in chat looks like speed. You change a tool argument and forget the old transcript. The file in git would have made the break obvious.

A harness you can copy

The samples below are only a proposed workflow. This article does not report an executed benchmark. Copy them, then run them on a host you control.

Refuse early

The harness does three jobs and no more. It refuses to start without a host base URL. It writes a manifest before any model call.

It treats tool output as data, not as a vibe. It also refuses a laptop loopback as a cited host. A loopback probe can exist, but it cannot be the proof.

#!/usr/bin/env python3
"""Proposed harness. Not executed for this article."""

import json
import os
import sys
import time
import urllib.request
from pathlib import Path

ROOT = Path(__file__).resolve().parent
MANIFEST = ROOT / "run-manifest.json"
SCHEMA = ROOT / "tools.json"

def die(msg: str) -> None:
    print(msg, file=sys.stderr)
    raise SystemExit(2)

def require_env(name: str) -> str:
    value = os.environ.get(name, "").strip()
    if not value:
        die(f"missing {name}")
    return value

def load_schema() -> dict:
    if not SCHEMA.is_file():
        die("tools.json missing; refuse to invent tools in a prompt")
    data = json.loads(SCHEMA.read_text())
    if not data.get("tools"):
        die("tools.json has no tools")
    return data

def probe_host(base: str) -> dict:
    url = base.rstrip("/") + "/health"
    req = urllib.request.Request(url, method="GET")
    try:
        with urllib.request.urlopen(req, timeout=5) as resp:
            body = resp.read(2048).decode("utf-8", "replace")
            return {"ok": resp.status == 200, "status": resp.status, "body": body}
    except Exception as exc:
        die(f"host probe failed: {exc}")

def write_manifest(host: str, schema: dict, probe: dict) -> None:
    doc = {
        "host": host,
        "schema_tools": [t.get("name") for t in schema["tools"]],
        "probe": probe,
        "started_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "model_calls": [],
        "tool_results": [],
        "verdict": "incomplete",
    }
    MANIFEST.write_text(json.dumps(doc, indent=2) + "\n")

def main() -> None:
    host = require_env("TOOL_HOST")
    lowered = host.lower()
    if "127.0.0.1" in lowered or "localhost" in lowered:
        die("refusing loopback; pin a named host")
    schema = load_schema()
    probe = probe_host(host)
    write_manifest(host, schema, probe)
    print(f"manifest written: {MANIFEST}")
    print("next: call the model only after this file exists")

if __name__ == "__main__":
    main()

Contract file

Pair the harness with a tool contract in git. Do not hide that contract inside a system prompt. A system prompt is not a versioned interface.

{
  "tools": [
    {
      "name": "run_check",
      "timeout_sec": 20,
      "input": {},
      "output": {"exit_code": "int", "stdout": "string", "stderr": "string"}
    }
  ]
}

An incomplete verdict is the only honest status before a restart. Fill model calls only after the second boot. Leave the list empty rather than inventing a call.

Fixed check

The host process can stay boring on purpose. Boring is a feature when you need a rerun. A free server is enough if the process can restart.

You do not need a clever runtime to prove the split. The sample runs one fixed check, not a shell string. Clever runtimes hide the reboot you are trying to test.

#!/usr/bin/env python3
"""Proposed tool host. Not executed for this article."""

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
import subprocess

HOST = os.environ.get("BIND_HOST", "127.0.0.1")
PORT = int(os.environ.get("BIND_PORT", "8080"))
CHECK = ["python3", "-m", "pytest", "-q"]

class Handler(BaseHTTPRequestHandler):
    def _send(self, code: int, payload: dict) -> None:
        raw = json.dumps(payload).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def do_GET(self) -> None:
        if self.path != "/health":
            self._send(404, {"ok": False})
            return
        self._send(200, {"ok": True, "check": CHECK})

    def do_POST(self) -> None:
        if self.path != "/tools/run_check":
            self._send(404, {"ok": False})
            return
        try:
            proc = subprocess.run(
                CHECK,
                capture_output=True,
                text=True,
                timeout=20,
                check=False,
            )
        except subprocess.TimeoutExpired:
            self._send(200, {"exit_code": 124, "stdout": "", "stderr": "timeout"})
            return
        self._send(
            200,
            {
                "exit_code": proc.returncode,
                "stdout": proc.stdout[-2000:],
                "stderr": proc.stderr[-2000:],
            },
        )

if __name__ == "__main__":
    ThreadingHTTPServer((HOST, PORT), Handler).serve_forever()

Do not expose this sample to the public internet. It is a lab listener, not a production gateway. Bind it on a host you can turn off.

Local bind is only for a private smoke test. A cited run still needs a named host URL. Do not launder a laptop loopback through a tunnel and call it pinned.

Commands that should fail closed

Run the negative cases before you celebrate a green file. A harness that only passes on the happy path is theater. Expect exit code 2 when the pin is missing.

python3 pin_host.py
# expect exit 2: missing TOOL_HOST

TOOL_HOST=http://localhost:8080 python3 pin_host.py
# expect exit 2: loopback refused

TOOL_HOST=http://tool-host.example:8080 python3 pin_host.py
# expect run-manifest.json only after /health returns 200

Then restart the host process and probe again. The new manifest needs a new start timestamp. Same schema, new process, or the pin failed.

Only after that file exists should you attach model access. Append one model record with an id and a latency. If you cannot fill those fields, do not score the answer.

Where a free option fits

You can do this split without buying hardware first. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant here only as a place to try the split.

The operator reports free model access and a free server option. This article does not name models, quotas, or hardware. Those details move, and stale numbers waste your week.

Read the current project page before you depend on either offer. You bring the host process, or you adapt the probe. Nothing here assumes a vendor route named health.

Use the free server as the named tool host if it fits. Use the free model access only after the manifest exists. If the live terms are tighter than you hoped, shrink the loop.

Do not invent capacity the page does not list. Do not treat a free tier as a permanent contract. Confirm the live limits, then throw the run away and rerun.

Who should skip this

Skip this approach if the repo holds regulated data. Skip it if customer secrets would enter the tool host. A free shared option is a lab, not a vault.

Skip it if you need a contract, a region, or a retention promise. Free availability is not an SLA, and this article does not claim one. If your policy team needs paper, get paper elsewhere.

Skip it if you want a leaderboard or a quality score. This workflow checks disposability, not the answer's goodness. A pinned host can still emit a wrong patch.

Skip it if you only needed a single chat reply. Standing up a host for one prompt is wasted motion. Use a chat box, and do not pretend it was an eval.

Limits you should say out loud

The code samples were not run for this writeup. A health check does not prove the tool did useful work. You still need a task oracle you trust outside the model.

A free server can throttle, sleep, or change under you. Treat that as a known limit, not as a surprise betrayal. Pin what you can, and record when the pin breaks.

The localhost refusal is a harness policy, not a firewall. Anyone can delete that check and lie to themselves. The value is the habit, not the ten lines that enforce it.

No benchmark numbers appear here, because none were measured. Do not cite this article as speed, cost, or quality proof. Cite your own manifest, or cite nothing at all.

Hold the line

Keep the screenshots if they help you remember the idea. Do not call those screenshots results in a review. Spend free tokens after the host survives a restart.

If the host cannot restart, the model was never the point. A free call does not repair a host you cannot name. Pin the host, or do not spend the quota.

A Free Agent Server Is a Stranger Until You Pin It

Emery Chen — Thu, 08 Oct 2026 08:00:42 +0000

A free model pass is not a promotion ticket. You may use it to explore a failure. You may not cite it as proof.

This is an opinion, and the line is sharp. Cheap repetition is a gift to your curiosity. Unpinned repetition is gossip with a green badge.

The position

You should refuse promotion from a free run. The run must name its server, image, and tools. If any name is missing, the run is a note.

Free access lowers the cost of trying again. It does not lower the cost of being wrong. A green transcript can still be a local accident.

Why the temptation is worse now

Agent demos are crowded on boards this month. Demo culture rewards a working screenshot over a contract. It rarely rewards a pinned host record at all.

You feel pressure to try it on free capacity. That pressure is normal for a busy week. It is also how bad evidence spreads online.

A remote pass can fail for reasons you never see. You do not control that machine's noisy neighbors. Shared capacity can still change underneath your single run.

What you may claim

Disclosure: This article was prepared as part of MonkeyCode's product outreach. You can claim you used free model access. You can claim you used a free server option.

You cannot claim a quota, a chip, or a duration. Those two availability claims are enough for a lab loop. They are not a benchmark you can publish.

They are not a promise that the offer stays fixed. If the vendor page changes, your article must change. Do not paste last quarter's numbers into this week.

Price and limits belong in a link you recheck. A remembered grant is not a citable source. Recheck the page on the day you publish.

The three pins

Pin the prompt file before you send it. Pin the tool schema before the model sees tools. Pin the server image before you trust the host.

A pin is a hash, not a vibe. You store the hash next to the transcript. You refuse the transcript when a hash is absent.

Fields that must exist

Keep these fields in every record you might cite. A missing field is a failed run, not a footnote. Write them before you admire the answer text.

Store prompt_sha256 as the hash of the exact prompt file.
Store schema_sha256 as the hash of the tool schema file.
Store image_digest as the digest the free server reports.
Set host_role to free-server, never to localhost here.
Record model_route as the route you selected, not a guess.

Leave model names out until you can read them back. A route label you typed is not a model identity. Identity comes from the response metadata, or it stays unknown.

A decision you can apply

Use this table before you quote a run. The table is a proposal, not a measured study. Apply it even when the transcript looks clean.

Pins present	Host role	Decision
All three	free-server	Lab evidence, still not a benchmark
All three	localhost	Scratch note only
Missing image	any	Do not cite
Missing schema	any	Do not cite
Missing prompt	any	Do not cite
All three, route changed	free-server	New experiment, old notes die

Read the last row twice before you merge notes. A route change is a new system under test. Your old green file does not travel with it.

The objection

Some developers will call this pin check theater. They want speed, and the capacity is free. They say a pin check slows the point of the offer.

Speed is fine for a private scratch file. Citation is a different act with a higher price. The moment you publish a result, the pin is due.

You can still explore without running the script. You just cannot smuggle that exploration into a claim. Label the folder scratch and keep it out of the post.

A check you can run

The script below is an unexecuted proposal only. It does not call a model or a network. It only refuses a dirty record on disk.

#!/usr/bin/env python3
"""Refuse a run record that lacks three pins. Unexecuted proposal."""

import hashlib
import json
import sys
from pathlib import Path

REQUIRED = ("prompt_sha256", "schema_sha256", "image_digest", "host_role")


def sha256_file(path: Path) -> str:
    data = path.read_bytes()
    return hashlib.sha256(data).hexdigest()


def main() -> int:
    if len(sys.argv) != 4:
        print("usage: pincheck.py record.json prompt.txt tools.json")
        return 2
    record = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    prompt_hash = sha256_file(Path(sys.argv[2]))
    schema_hash = sha256_file(Path(sys.argv[3]))
    missing = [key for key in REQUIRED if not record.get(key)]
    if missing:
        print("reject: missing " + ",".join(missing))
        return 1
    if record["host_role"] != "free-server":
        print("reject: host_role must be free-server")
        return 1
    if record["prompt_sha256"] != prompt_hash:
        print("reject: prompt hash drift")
        return 1
    if record["schema_sha256"] != schema_hash:
        print("reject: schema hash drift")
        return 1
    digest = str(record["image_digest"])
    if not digest.startswith("sha256:"):
        print("reject: image_digest must start with sha256:")
        return 1
    print("accept: pins match; still not a benchmark")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it against a fixture you wrote by hand. A failing exit is the result you should want first. Fix the record, then rerun the same command.

python3 pincheck.py run-record.json prompt.txt tools.json
echo "exit=$?"

A zero exit means the record is internally consistent. It does not mean the agent is correct. It does not mean the free server stayed private.

Build the record without guessing

Hash the fixture files on your side first. Ask the free server for its image digest. Write the digest exactly as the server returned it.

# proposal commands; use the tool your OS actually ships
sha256sum prompt.txt tools.json 2>/dev/null || shasum -a 256 prompt.txt tools.json

Do not clean it up to match a blog post. A small record looks like the JSON below. Replace every placeholder before you run the check.

Fixture files

Start with a tiny prompt and a tiny schema. Both files must stay byte-stable during the pair of runs. If you edit either file, you start over.

Read fixture/hello.txt and report the first line only.
Do not call any tool except read_fixture.

{
  "tools": [
    {
      "name": "read_fixture",
      "description": "Read one file inside the fixture tree.",
      "arguments": {
        "path": "string"
      }
    }
  ]
}

{
  "prompt_sha256": "<hash from prompt.txt>",
  "schema_sha256": "<hash from tools.json>",
  "image_digest": "sha256:<digest the server returned>",
  "host_role": "free-server",
  "model_route": "<route you selected>",
  "note": "proposal fixture, not a live result"
}

If the server will not tell you the digest, stop. You do not have a pin you can defend. You have a story, not a citable record.

Drift during one session

Ask for the digest before the run starts. Ask for the digest after the run ends. If the two strings differ, discard both transcripts.

Do not pick the closer digest and move on. A moving image means you have a moving system. Your diff is then comparing two different strangers.

# proposal: capture digests around one attempt
printf '%s\n' "$DIGEST_BEFORE" > digest-before.txt
printf '%s\n' "$DIGEST_AFTER" > digest-after.txt
cmp -s digest-before.txt digest-after.txt || echo "reject: image moved"

Keep those two digest files beside the transcript. If cmp fails, the attempt never enters the table. Do not paste the rejected text into a chart.

How to use the free loop

Use free model access for repetition, not for glory. Run the same pinned prompt twice on purpose. Diff the tool calls, not the polished adjectives.

Freeze prompt.txt and tools.json before you send either.
Record the free server image digest in the note.
Send the same files through the free model route.
Save raw tool calls as run-a.json and run-b.json.
Diff the call names and argument keys only.
If the diff is nonempty, keep both runs as failures.
Do not average them into a success rate.

# Unexecuted proposal. Adapt keys if your log shape differs.
import json


def names(path):
    with open(path, encoding="utf-8") as handle:
        data = json.load(handle)
    return [call.get("name") for call in data.get("calls", [])]


left = names("run-a.json")
right = names("run-b.json")
print("same_names", left == right)
print("a", left)
print("b", right)

That snippet is also an unexecuted proposal for you. It assumes a log shape you already control. If your logs use another shape, adapt the keys.

Do not invent a pass to match the printout. A printed true is not a tested agent. Save the raw files even when the names match.

What free capacity must not touch

Give the free server a boring fixture repo. Use fake tokens, fake hostnames, and a read-only tree. Boring inputs keep the lesson about evidence, not drama.

Keep production credentials out of that environment entirely.
Keep customer data out of the prompt text.
Allow no network calls except the chosen model route.
Deny writes outside the temp fixture directory path.

A free server is shared capacity until proven otherwise. Treat it as hostile in the security sense. Courtesy from a vendor is not a private enclave.

Limits of this opinion

This check catches missing pins and hash drift. It does not catch a lying image digest. It does not score answer quality at all.

Two identical tool-name lists can still be wrong. You still need a human fixture with known outputs. That fixture is your job, not the model's job.

Free model access can disappear or change route. When the route changes, burn the comparison set. Start a new folder for the new route.

Do not mix old and new transcripts in one chart. This piece claims no latency number at all. This piece claims no token grant of any size.

This piece claims no win over any other stack. Treat every number you remember as unverified until sourced. A blog memory is not a primary source.

Who should not use this

Skip this flow if you need a certified environment. A free server option is the wrong home for secrets. It is the wrong home for a release gate.

Skip it if you cannot read the image digest. Skip it if your tool schema changes every prompt. Skip it if you wanted a demo, not a record.

Also skip it if you will ignore a nonzero exit. The script is useless once you bypass it. Discipline is the real gate, not the file.

What you should say afterward

Lead with the decision, not the tool brand. Say the run was pinned, or call it a scratch note. Do not let a free badge do the arguing.

If you mention the free model route, mention the date. If you mention the free server, mention the digest you saw. If either fact is missing, delete the paragraph.

One soft next step is enough for this piece. If you already have MonkeyCode free model access, use it here. Pair it with the free server option you were given.

Run the pin check on one fixture before you quote anything. Then stop writing product sentences into the lab note. More brand mentions will not make the hash tighter.

Do Not Cite an Agent Run You Cannot Price

Emery Chen — Thu, 24 Sep 2026 21:10:33 +0000

You should reject an agent run that cannot name its cost. A free server does not waive that recording rule. Zero dollars is still a number you must store.

Stop praising unlabeled results

Clean prompts are not the same as clear evidence. A short instruction can hide a messy execution path. You need the bill, the placement, and the model id.

This piece argues for a hard gate, not a soft tip. Incomplete manifests should fail before anyone reads the answer. A clever result without placement is not evidence.

Free access creates a new lie

Free model access feels harmless during the first hour. You try another prompt and skip the usage log. No invoice arrives, so the missing fields feel harmless.

That habit damages an eval set within days. Later you cannot separate a local fluke from a hosted fluke. You also cannot separate a free host from a paid host.

A missing amount should fail the test immediately. A missing server class should fail the test immediately. A missing model id should fail the test too.

Placement belongs in the artifact

The same prompt can shift when the server changes. Timeouts, tool reach, and rate limits all move. Your notes should name the placement beside the output.

A screenshot cannot support a serious merge decision. A pasted chat log is not an audit record either. Write the placement into the file you actually keep.

Stick to three labels and refuse extras mid-week. Use local for a machine you control yourself. Use free_hosted or paid_hosted for a vendor machine.

Extra labels quickly become excuses for lucky runs. Your reviewer cannot compare runs that invent names. Pick the three labels and keep them stable.

How to read the table

Use the table before you store any agent run. Every class still needs a model id and token counts. Only free_hosted may store an amount of zero.

Question	local	free_hosted	paid_hosted
Record model id?	Yes	Yes	Yes
Record token counts?	Yes	Yes	Yes
Amount field	Measured or estimated	Must be 0	Must be above 0
Merge evidence?	Only after the checker	Only after the checker	Only after the checker
Blank cost allowed?	No	No	No

Local runs must show a measured or estimated amount. Paid hosted runs must show an amount above zero. None of these rows prove the answer itself is right.

Fields that earn trust

Keep one JSON manifest beside each saved output. Do not bury those fields inside a chat scrollback. A pretty trace without required fields is only a story.

Require these fields on every cited run:

Require a run id so retries do not overwrite evidence.
Require the model id exactly as the provider named it.
Require a server class from the three labels above.
Require input tokens and output tokens as integers.
Require a currency code and a numeric amount.
Require a UTC timestamp and a prompt hash.

Optional notes can wait until the next revision. Required fields cannot wait for a cleanup pass. If a host omits usage, the run is not evidence.

Sample manifest

The sample below is illustrative, not a captured run. Replace every value before you treat it as proof. Do not commit a toy hash and call it evidence.

{
  "run_id": "run-2041",
  "model_id": "provider-model-id",
  "server_class": "free_hosted",
  "input_tokens": 812,
  "output_tokens": 146,
  "currency": "USD",
  "amount": 0,
  "started_at": "2026-09-25T09:00:00Z",
  "prompt_sha256": "replace-with-real-hash"
}

Run the proposed checker

The following script is a proposal, not a lab result. It was not executed while this article was written. Put it in your repo and run it on real files.

import hashlib
import json
import sys

REQUIRED = (
    'run_id',
    'model_id',
    'server_class',
    'input_tokens',
    'output_tokens',
    'currency',
    'amount',
    'started_at',
    'prompt_sha256',
)
CLASSES = {'local', 'free_hosted', 'paid_hosted'}

def fail(message):
    raise SystemExit(message)

def validate(manifest_path, prompt_path):
    data = json.load(open(manifest_path, encoding='utf-8'))
    missing = [key for key in REQUIRED if key not in data]
    if missing:
        fail('missing fields: ' + ', '.join(missing))
    if data['server_class'] not in CLASSES:
        fail('unknown server_class')
    if data['input_tokens'] < 0 or data['output_tokens'] < 0:
        fail('token counts cannot be negative')
    amount = data['amount']
    if amount < 0:
        fail('amount cannot be negative')
    if data['server_class'] == 'free_hosted' and amount != 0:
        fail('free_hosted must record zero')
    if data['server_class'] != 'free_hosted' and amount == 0:
        fail('zero amount requires free_hosted')
    digest = hashlib.sha256(open(prompt_path, 'rb').read()).hexdigest()
    if digest != data['prompt_sha256']:
        fail('prompt hash mismatch')
    print('manifest ok')

if __name__ == '__main__':
    validate(sys.argv[1], sys.argv[2])

CI command

Run the checker before you comment on any result. Pass the manifest path and the prompt file path. A non-zero exit must fail the job in CI.

python check_manifest.py runs/run-2041.json prompts/run-2041.txt

A green chat reply is not a green CI job. Keep the command in the same repo as the prompt. Reviewers should rerun it instead of trusting a screenshot.

A workflow you can repeat

Use this order for every run you might cite:

Save the prompt bytes to a file before the call.
Call the model on local or on the free server.
Write the manifest with class, ids, tokens, and amount.
Hash the prompt file and store that digest.
Run the checker and keep the exit code.
Share the answer only after the checker passes.

Stop at the first missing field. Do not patch the file after the review starts. A late edit is a new run, so give it a new id.

Limits you should say out loud

A valid manifest does not make the answer correct. It only shows that you recorded the run honestly. You still need assertions on the output text.

Vendor token counts can disagree with your own estimate. Do not write a guess into the token fields. Unknown usage means the run fails this gate.

Prices move, and free offers move as well. Record what was true for that specific run. Do not describe a free offer as a permanent promise.

This gate will reject casual exploratory notes on purpose. Keep scratch notes outside the evidence folder entirely. Mixing scratch files with evidence creates false confidence.

Use a free server without self-deception

A free hosted server can still teach you something. It helps when your laptop is a weak stand-in. It helps nothing if you delete the manifest afterward.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project with free model access. It also offers a free server option for hosted runs. Those availability claims are the only product facts used here.

This article does not claim quotas, hardware, or fixed model lists. Check current project docs before you depend on either offer. Terms can change, so your manifest must stay specific.

Label that path as free_hosted and keep the amount at zero. Copy the model id from the response, not from memory. Then run the checker before you share the output.

If the free path omits token counts, do not backfill them. Mark that run unusable for any merge evidence. Measure on a host that returns usage, or stay local.

Put the checker in the repo before you chase clever prompts. Point one non-production prompt at the free server. See whether the manifest can pass without hand edits.

Who should skip the gate

Skip this gate if your tests never call a model. Fixture checks do not need a cost field at all. Adding a fake cost field would be theater, not engineering.

Skip it when a platform already signs usage exports. Import those exports instead of typing amounts by hand. Hand-typed bills are easy to fake under deadline pressure.

Skip it when you want a demo with no audit trail. This method will slow that kind of demo down. The slowdown is intentional, so pick another format.

Do not use the checker as a safety filter. It does not judge quality or policy risk. It judges whether the run can be priced and placed.

Four questions before you share

Ask these questions before you share a result.

Can you name the model id without opening a chat log?
Can you name the server class in a single word?
Is the amount zero only because the class is free_hosted?
Does the prompt hash match the file you sent?

If any answer is no, withhold the conclusion. Run the prompt again with the manifest in place. Let the checker speak before you write the conclusion.

Exploration can stay fast inside a scratch folder. Evidence stays in the checked folder, separate from scratch. Do not mix those folders when you ask for a merge.

Free access is useful for iteration and dangerous for memory. Write the zero down, or do not cite the run. That is the whole position, and it is not optional.

If the Round Trip Was Instant, You Cheated

Emery Chen — Wed, 23 Sep 2026 17:06:22 +0000

Your agent did not pass. It never waited for anything.
Local tool calls hide timeouts, retries, and partial payloads.
An instant round trip is not evidence. It is a missing network.

Instant success is a different program

You run the agent on your laptop first.
Every tool is a function in the same process.
The fake API returns in one millisecond. No packet left the box.

That program is not the one you will ship.
The shipped program waits, aborts, retries, and sometimes gets half a JSON body.
Those branches never run when the network is a function call.

Browser-only agent demos make the cheat quieter.
The loop stays in one origin, one event loop, one warm cache.
You are grading a rehearsal, not a call path.

The claim you should argue with

Here is a clear position, not a hedge.
If your eval never blocked on I/O, you do not have an eval.
You only have a unit test of prompt text.

Timeouts are not a late ops concern.
They change which tool the model chooses next.
Retries can duplicate side effects on the wire.
Slow tools starve later steps under a wall clock.

You cannot see that failure on localhost.
A green local run can still be a silent network.

Record waits, not just answers

Stop logging only the final assistant string.
Log the wait that produced each tool result.
If you cannot replay the waits, you cannot debug the agent.

Capture at least these fields on every call:

tool name and attempt number
monotonic start time and end time
timeout budget for that attempt
abort flag and retry flag
bytes received before the first parse
HTTP status or transport error, if any

If attempt number is always one, you learned little.
If duration is always under 20ms, you learned nothing.
That second case is the silent cheat in most laptop evals.

Three waits you must treat as bugs

Do not dump every delay into one p95 bucket.
Name the wait, then write an assertion for it.

Deadline miss. The tool exceeds its budget. The agent must abort and choose a fallback. A hang is a failed eval, not a slow pass.
Retry storm. The first call dies. The second call repeats a POST. Unless you sent an idempotency key, you may have charged a card twice.
Partial body. Bytes arrive, then the socket dies. Your parser throws. The agent must not treat a truncated object as success.

Localhost almost never produces case two or three.
A remote hop produces them without extra theatre.
That is why a quiet laptop is a biased lab.

Copy this harness

The following is a proposed example, not a scored benchmark.
Wire it to your real tool runner before you trust it.
Do not treat the 20ms cutoff as a universal law.

# latency_harness.py
# Example only. Enforce timeout_s inside your real transport.

from __future__ import annotations

import json
import time
import uuid
from dataclasses import asdict, dataclass
from typing import Callable

@dataclass
class ToolEvent:
    run_id: str
    tool: str
    attempt: int
    timeout_s: float
    started_at: float
    ended_at: float
    aborted: bool
    error: str | None
    bytes_in: int
    implausible_local: bool

    @property
    def wait_ms(self) -> float:
        return (self.ended_at - self.started_at) * 1000.0


IMPLAUSIBLE_MS = 20.0  # example: in-process cheat detector


def call_with_budget(
    run_id: str,
    tool: str,
    attempt: int,
    timeout_s: float,
    fn: Callable[[], bytes],
) -> ToolEvent:
    started = time.monotonic()
    aborted = False
    error = None
    payload = b""
    try:
        payload = fn()
    except TimeoutError as exc:
        aborted = True
        error = str(exc)
    except Exception as exc:
        error = str(exc)
    ended = time.monotonic()
    wait_ms = (ended - started) * 1000.0
    return ToolEvent(
        run_id=run_id,
        tool=tool,
        attempt=attempt,
        timeout_s=timeout_s,
        started_at=started,
        ended_at=ended,
        aborted=aborted,
        error=error,
        bytes_in=len(payload),
        implausible_local=wait_ms < IMPLAUSIBLE_MS and error is None,
    )


def write_trace(path: str, events: list[ToolEvent]) -> None:
    with open(path, "w", encoding="utf-8") as handle:
        for event in events:
            handle.write(json.dumps(asdict(event)) + "\n")


def assert_eval_was_real(events: list[ToolEvent]) -> None:
    if not events:
        raise AssertionError("no tool events; this is not an agent eval")
    waits = [e.wait_ms for e in events]
    if all(w < IMPLAUSIBLE_MS for w in waits):
        raise AssertionError(
            "every tool returned under 20ms; you never left the process"
        )
    if all(e.attempt == 1 and not e.aborted for e in events):
        print("warning: no abort or retry observed")

Keep the assertion in CI, not in a slide.

# test_agent_waits.py
# Example assertions. Replace fake_remote() with a real invocation.

import uuid
from latency_harness import assert_eval_was_real, call_with_budget


def test_remote_search_must_honor_a_deadline():
    run_id = str(uuid.uuid4())

    def fake_remote() -> bytes:
        raise TimeoutError("deadline exceeded")

    event = call_with_budget(run_id, "search", 1, 8.0, fake_remote)
    assert event.aborted is True
    assert event.error is not None


def test_reject_all_instant_successes():
    run_id = str(uuid.uuid4())
    events = [
        call_with_budget(run_id, "search", 1, 8.0, lambda: b'{"ok":true}')
    ]
    try:
        assert_eval_was_real(events)
        raise AssertionError("instant local success should not pass")
    except AssertionError as exc:
        assert "never left the process" in str(exc)

Run the tests the boring way:

python -m pytest test_agent_waits.py -q

You still need two traces, not one green file.
One trace from the laptop process.
One trace from a host that is not your laptop.

python - <<'PY'
import json, sys

def load(path):
    with open(path) as f:
        return [json.loads(line) for line in f]

local = load("trace-local.jsonl")
remote = load("trace-remote.jsonl")
local_waits = [row["ended_at"] - row["started_at"] for row in local]
remote_waits = [row["ended_at"] - row["started_at"] for row in remote]

if local and max(local_waits) < 0.02 and remote and max(remote_waits) > 0.2:
    print("local eval was a rehearsal; remote eval actually waited")
    sys.exit(1)
print("next: diff abort flags and attempt counts")
PY

That comparison script is a proposal.
Tune the numbers after you collect your own traces.
Do not copy 20ms into production policy blindly.

A decision table, not a vibe

Use this table before you trust a green agent build.

Signal you recorded	What it usually means	Ship it?
All tools under 20ms, same process	You tested a function, not a network	No
Remote p95 is high, no timeout handler	The model will hang or skip steps	No
Timeout fires, retry repeats a POST	You may have duplicated a side effect	No
Timeout fires, retry sends an idempotency key	You have a real recovery path	Maybe
Abort mid-body, parser throws, agent stops	Partial JSON is an untested branch	No
Abort mid-body, agent requests a retry tool	You tested the ugly path	Yes, if traces agree
Browser demo only, no remote hop	You tested UI glue	No for backend agents
Local and remote traces pick different tools	The pass was environment-specific	No

Read the last row twice.
Latency is not the only drift you will see.
Tool choice drifts as soon as waits appear.
The model is not independent of time.

Idempotency is part of the latency test

A timeout without an idempotency key is an incident draft.
You do not know whether the first call committed.
The agent also does not know. It will guess.

Give every mutating tool a key you control:

# Example header. Your API may use a different name.
headers = {
    "Idempotency-Key": f"{run_id}:{tool}:{attempt}",
}

Then assert the remote trace.
A retry must reuse the key, not mint a second one.
If the key changes on attempt two, fail the job.
That bug is invisible when the first call never dies.

Why a free remote hop belongs in this method

You do not need a cluster for this control.
You need a machine that is not yours.
You also need a model endpoint that is not a local stub.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option.
That pair is useful as a control, not as a slogan.
Run the same harness against the free server.
Keep the laptop run as the suspect baseline.

If the free remote trace never waits either, your tools are still in-process.
If it waits and your code has no abort path, you found the bug.
That is the only reason a free box belongs in this article.

Do not treat free access as a scoreboard.
Do not treat it as a capacity promise.
Use it once to make the network real.

Limitations

This approach will waste time in some shops.
Skip it when the wait cannot exist.

Skip it if:

your agent has no tools and only writes text
every tool already has an injected delay in unit tests
you already run evals in an isolated remote job with real deadlines
policy forbids sending traces or prompts off the laptop

Do not mix unlike waits in one assertion.
A 20ms cutoff only hunts in-process cheats.
A multi-second DNS blip is a different test.
This harness does not prove task correctness.
It only proves the agent was forced to wait.

No speedup is claimed here.
No model ranking is claimed here.
The claim is narrower: your green bar may be a missing sleep.

What you should do tomorrow

Pick one agent path that always works locally.
Wrap each tool with the event recorder.
Run it on the laptop. Save trace-local.jsonl.

Then run the same path on a host you did not bootstrap by hand.
Save trace-remote.jsonl.
Diff abort flags, attempt counts, and wait buckets.

If local waits are all tiny, fail the job.
If remote waits exist and your code ignores them, fail the job.
If both traces wait and recover, you finally have an eval.

A free remote endpoint is enough to start that control.
If you already have MonkeyCode's free server, point the harness there once and keep the traces.

Do not announce a pass until something had to wait.

If Only One Host Passed, the Agent Did Not

Emery Chen — Tue, 22 Sep 2026 15:20:23 +0000

Your agent test is not a result yet. It is still a rumor from one host.

A second machine must run the same contract. Only then should you merge the change.

The hidden input you keep ignoring

Most agent evals pin the prompt text. Some teams even pin the tool schema.

Almost nobody pins the actual execution host. That gap is how broken loops ship.

The model is not the only moving part. Filesystem layout and network paths move too.

Clock skew quietly changes your retry behavior. DNS caches change what a tool returns.

Shell locale changes how paths expand. Home directories leak straight into tool arguments.

You currently treat those facts as noise. They are hidden inputs to every call.

Opinion: one host cannot certify an agent

Adopt a hard rule on agent evals. Fail the suite without a second host.

Local green builds hide host coupling fast. They reward agents that sniff your laptop.

Browser-only demos repeat that same mistake daily. A blessed runtime is still a single host.

The tab is not a second environment. It is one more blessed container with secrets.

If the agent needs your files, it failed. If it needs your GPU paths, it failed.

If it needs your dotenv layout, it failed. The contract has to travel without souvenirs.

Tool-calling tutorials often stop at the happy path. That is how you ship a laptop goblin.

The interesting failure is not a wrong sentence. The interesting failure is a path that only exists here.

Anatomy of a false green

Here is a common false green pattern. The local agent labels issues with flair.

It also reads your private SSH config. The remote host has no such file.

Local CI stays green and looks smart. Remote CI dies on a missing path.

That is not a flaky model problem. That is an unstated dependency on the host.

Leaked call, labeled as an example:

{
  "name": "add_label",
  "args": {
    "repo": "acme/widgets",
    "issue_id": 441,
    "label": "bug",
    "ssh_identity": "/Users/you/.ssh/id_ed25519"
  }
}

The extra label looks useful in review. The smuggled key is the entire bug.

Record a fingerprint before any model call

Do not start with graded prose output. Start with a boring host fingerprint file.

Treat that file as a test fixture. Store it beside the tool-call trace.

The script below is a labeled proposal. It is example code, not a benchmark.

#!/usr/bin/env python3
"""Proposal: fingerprint the host before an agent eval."""
import hashlib, json, os, platform, socket, sys
from datetime import datetime, timezone
from pathlib import Path

def fingerprint(cwd: str) -> dict:
    path = Path(cwd)
    return {
        "captured_at": datetime.now(timezone.utc).isoformat(),
        "hostname": socket.gethostname(),
        "platform": platform.platform(),
        "python": sys.version.split()[0],
        "cwd": str(path.resolve()),
        "cwd_exists": path.exists(),
        "user_home_in_cwd": str(Path.home()) in str(path.resolve()),
        "env_keys": sorted(
            k for k in os.environ if k.startswith(("HTTP", "NO_", "PATH", "SSH"))
        ),
        "has_local_dotenv": (path / ".env").exists(),
        "cpu_count": os.cpu_count(),
    }

def digest(doc: dict) -> str:
    blob = json.dumps(doc, sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()[:16]

if __name__ == "__main__":
    doc = fingerprint(".")
    stable = {k: doc[k] for k in doc if k != "captured_at"}
    doc["fingerprint"] = digest(stable)
    print(json.dumps(doc, indent=2))

Run the script on your development laptop. Run the same script on a spare server.

Compare the fingerprints before you trust traces. If later calls need laptop-only keys, fail.

Allowlist the environment, then freeze it

Do not export your whole environment remotely. Export an allowlist, then freeze that list.

# Proposal: freeze env keys before the remote replay.
export AGENT_ALLOWED_ENV="PATH,LANG,TZ"
python fingerprint.py > fp.json
jq -r '.env_keys[]' fp.json > /tmp/seen_env.txt
# Fail the job if SSH_* or *_TOKEN keys appear.
if grep -E 'SSH_|_TOKEN|_SECRET' /tmp/seen_env.txt; then
  echo "env leak in fingerprint" >&2
  exit 1
fi

Extra keys are clues, not convenience features. Delete them before the second host runs.

The contract the second host must honor

Do not ship a copied chat transcript. Ship a tool-shape contract both hosts share.

The second host must see identical tools. It must return the same result shapes.

The fixture below is also a proposal. Keep it in source control with tests.

{
  "contract_id": "issue.triage.v3",
  "tools": [
    {
      "name": "list_open_issues",
      "required_args": ["repo", "limit"],
      "forbidden_args": ["token", "cookie", "ssh_identity"],
      "result_shape": ["id", "title", "state"]
    },
    {
      "name": "add_label",
      "required_args": ["repo", "issue_id", "label"],
      "forbidden_args": ["assignee_email", "ssh_identity"],
      "result_shape": ["ok", "issue_id"]
    }
  ],
  "invariants": [
    "no_absolute_local_paths",
    "no_home_directory_strings",
    "no_hostname_in_tool_args"
  ]
}

You are not grading the model's essays. You are grading the shape of calls.

Compare traces, not vibes

Write a checker that rejects host leakage. Keep the checker boring, strict, and exit-coded.

#!/usr/bin/env python3
"""Proposal: fail traces that smuggle host details into tool calls."""
import json, re, sys
from pathlib import Path

ABS_PATH = re.compile(r"(?:/Users/|/home/|[A-Za-z]:\\)")

def load(p):
    return json.loads(Path(p).read_text())

def leaks(text, fingerprint):
    hay = text.lower()
    host = str(fingerprint.get("hostname", "")).lower()
    cwd = str(fingerprint.get("cwd", "")).lower()
    if host and host in hay:
        return "hostname"
    if cwd and cwd in hay:
        return "cwd"
    if ABS_PATH.search(text):
        return "absolute_path"
    return None

def main(contract_path, trace_path, fp_path):
    contract = load(contract_path)
    trace = load(trace_path)
    fp = load(fp_path)
    errors = []
    allowed = {t["name"] for t in contract["tools"]}
    for i, call in enumerate(trace.get("tool_calls", [])):
        name = call.get("name")
        args = json.dumps(call.get("args", {}), sort_keys=True)
        if name not in allowed:
            errors.append(f"call {i}: unknown tool {name}")
            continue
        spec = next(t for t in contract["tools"] if t["name"] == name)
        missing = [k for k in spec["required_args"] if k not in call.get("args", {})]
        forbidden = [k for k in spec["forbidden_args"] if k in call.get("args", {})]
        if missing:
            errors.append(f"call {i}: missing {missing}")
        if forbidden:
            errors.append(f"call {i}: forbidden {forbidden}")
        leak = leaks(args, fp)
        if leak:
            errors.append(f"call {i}: leaked {leak}")
    if errors:
        print("FAIL")
        print("\n".join(errors))
        sys.exit(1)
    print("PASS")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2], sys.argv[3])

Wire it into CI with plain commands. Do not hide the steps in chat.

python fingerprint.py > /tmp/fp.local.json
# Run your agent here. Write /tmp/trace.local.json.
python check_trace.py contract.json /tmp/trace.local.json /tmp/fp.local.json

Repeat every step on the second host. Diff tool shapes, and ignore extra prose.

jq -S '.tool_calls' trace.local.json > /tmp/calls.local.json
jq -S '.tool_calls' trace.remote.json > /tmp/calls.remote.json
diff -u /tmp/calls.local.json /tmp/calls.remote.json

A wording mismatch is usually just cheap noise. A shape mismatch is the actual bug.

Copy the fixtures with boring scp commands. Do not wrap them in a chat agent.

# Proposal only. Replace the remote entrypoint with yours.
scp contract.json fingerprint.py check_trace.py user@remote:/tmp/eval/
ssh user@remote 'cd /tmp/eval && python3 fingerprint.py > fp.json'

Decision table: ship, quarantine, or delete

Fill this table before you merge anything. Empty cells mean you still lack evidence.

Observation	Meaning	Action
Local pass, remote fail	Host coupling	Quarantine the agent
Both pass, shapes differ	Unstable contract	Pin tools, rerun
Both fail the same way	Honest bug	Fix the tool
Remote pass, local fail	Laptop contamination	Clean local secrets
Fingerprints identical	You did not change hosts	Use a real second machine

If you cannot complete this table, stop. You cannot honestly claim the agent passed.

Where a free remote loop actually helps

You need a second host for this. You do not need a private cluster.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open source coding project. It offers free model access and a free server option.

Use those two things as a second lane. Do not treat them as a personality test.

The free server holds the remote fingerprint. The free model access reruns the same contract.

Keep the contract file identical across both hosts. Change only the host under the runner.

This is not a performance claim at all. It is a clean separation of eval inputs.

You still must pin the tool schemas. You still must archive the raw traces.

A free lane does not replace code review. It only removes your laptop from evidence.

If you lack a spare host, use that free server. That is enough for this host-independence check.

What this does not prove

A matching remote trace is not production safety. It only shows host independence for that contract.

It does not prove the model stays stable. It does not prove the HTTP API is fast.

It does not prove the tool is authorized. It does not prove your token spend is sane.

Free access can change without any notice. Treat it as a moving worker, never a promise.

Do not publish latency numbers from one run. Do not invent model names inside eval reports.

Do not park secrets on a shared server. Redact traces before you copy them anywhere.

Shared hosts also mean you have noisy neighbors. Your eval must tolerate extra latency without cheating.

If the remote box is shared, pin timeouts. Do not loosen shapes to hide slow tools.

Who should not use this approach

Skip this pattern if you ship pure functions. Skip it if the agent exposes no tools.

Skip it if policy forbids shared remote hosts. Skip it if traces still contain customer data.

Skip it if you cannot write a contract file. Vague vibes will not survive a shape diff.

Skip it when the agent must use local GPUs. That constraint belongs on a private runner.

Do not use a free shared server for regulated data. That is the wrong class of host.

The rule to paste in the README

Put the rule in one blunt checklist. Make the merge bot enforce every line.

An agent eval is invalid until:
1. a host fingerprint is stored,
2. a tool-shape contract is stored,
3. a second host replays the contract,
4. the checker finds no host leakage.

You may argue about the exact wording. You should not argue about the underlying need.

One host remains only a polished demo. Two agreeing hosts finally count as a test.

If the second host happens to be free, fine. If the second host is yours, that is also fine.

Just stop merging rumors from a single machine.

If the Model Moved, Burn the Golden Files

Emery Chen — Mon, 21 Sep 2026 13:36:00 +0000

Frozen golden files cannot survive a model upgrade. You should run a living critic on a clean host.

The argument

Your agent tests still freeze last week's answers. Then the serving model changes overnight without warning. The suite fails on cosmetic wording after that bump.

Or it stays green while real behavior drifts. You do not hold a quality signal anymore. You hold a snapshot of a dead model.

Treat those frozen answers as unpaid test debt. A moving model needs a living check. A frozen string is not that check.

The tests did not get weaker

People now ask if AI outgrew our tests. Ask first whether fixtures outlived the recorded model. Those are different failures with different fixes.

A golden string still assumes one correct reply. Agents do not emit one stable reply anymore. They emit tool traces, side effects, and leftover prose.

Wording shifts after every quiet vendor bump. Call order also shifts after a schema tweak. Your diff tool then screams about both events.

Neither scream maps cleanly to user harm. Worse cases stay silent under the same fixtures. The new model skips a required precondition check.

Your frozen paragraph still matches yesterday's run. The build stays green and then ships. That is not coverage. That is nostalgia.

Frozen files punish harmless change far too often. They also miss harmful change far too often. That combination makes a bad test instrument.

Three layers beat one snapshot

Keep three layers in the same job. Do not collapse them into chat scores.

Contracts

Contracts check JSON shape and required tools. Those checks remain deterministic across later model swaps.

Effects

Effects checks inspect the world after the run. Count rows, HTTP codes, and file hashes.

Critique

Critique is a second model reading the trace. It hunts plans that are valid and still dumb.

Contracts catch schema rot early. Effects catch real damage in the system. Critique catches confident nonsense that contracts allow.

Most teams stop at contracts or prose. Few run a critic on every bump. The missing layer is almost always the critic.

Opinion: the critic must stay cheap

Here is the hard claim for this piece. Costly critique will not run on every bump.

You will postpone that job until the customer demo. You will skip it during the actual incident. Skipped critique is how untested agents ship.

The critic should not be the actor model. The actor is expensive and already moving. The critic should be cheaper and pinned in config.

The critic should not share your laptop either. Your laptop holds secrets and dirty state. Your laptop also sleeps while CI expects an answer.

Park redacted traces on a clean remote host. Then ask the critic to lint the trace. Keep production credentials off that host.

Artifact: a living eval harness

This section is a proposed workflow. It is not a vendor benchmark. It claims no accuracy number and no leaderboard.

You capture traces, gate contracts, then call a critic. Keep each step boring enough to rerun. Glamour is how eval suites rot.

Step 1 — Capture traces as JSONL

Write one JSON object per agent run. Keep the capture schema boring, small, and stable.

{"id": "pr-4412-a", "goal": "create invoice draft", "tools": [{"name": "db.query", "args": {"table": "customers", "id": 918}}, {"name": "invoices.create", "args": {"customer_id": 918, "cents": 1299}}], "final": "Draft created for customer 918"}

Redact tokens before anything hits disk. Redact names if your policy requires that step.

python capture_traces.py --task invoices --out traces.jsonl
python redact.py --in traces.jsonl --out traces.clean.jsonl

Step 2 — Fail closed on contracts

Run deterministic checks before any model call. Do not spend critique budget on garbage traces.

# contract_gate.py — labeled proposal, not a shipped product
import json, sys

REQUIRED = {"db.query", "invoices.create"}
FORBIDDEN = ("password", "ssn", "authorization", "api_key")

def errors_for(event):
    problems = []
    names = {step["name"] for step in event.get("tools", [])}
    missing = REQUIRED - names
    if missing:
        problems.append(f"missing tools: {sorted(missing)}")
    blob = json.dumps(event).lower()
    for needle in FORBIDDEN:
        if needle in blob:
            problems.append(f"possible secret field: {needle}")
    if not event.get("goal"):
        problems.append("empty goal")
    return problems

fail = 0
with open(sys.argv[1]) as handle:
    for line in handle:
        event = json.loads(line)
        problems = errors_for(event)
        if problems:
            fail += 1
            print(event.get("id"), problems)
sys.exit(1 if fail else 0)

Call it like any other unit test. A red contract means stop. Critique comes only after this gate.

python contract_gate.py traces.clean.jsonl

Step 3 — Pin the critic prompt in git

The prompt is part of the test suite. Version it and review it like code.

You are a trace linter, not a chat assistant.
Return JSON only with keys risk and reasons.
risk must be low, med, or high.
Flag missing preconditions, extra tools, and unverifiable claims.
Do not rewrite the agent. Do not praise the agent.

If nobody reviews this file, you cannot trust the critic. An unowned prompt is another moving model.

Step 4 — Call a cheap HTTP endpoint

Keep the HTTP client dull on purpose. Do not hide retries inside a heavy SDK.

# critique_run.py — labeled proposal
import json, os, sys, urllib.request

ENDPOINT = os.environ["CRITIC_ENDPOINT"]
TOKEN = os.environ["CRITIC_TOKEN"]
PROMPT = open("critic_prompt.txt").read()

def critique(event):
    body = json.dumps({
        "instructions": PROMPT,
        "trace": event,
    }).encode()
    req = urllib.request.Request(
        ENDPOINT,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {TOKEN}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.loads(resp.read().decode())

high = 0
with open(sys.argv[1]) as handle:
    for line in handle:
        event = json.loads(line)
        result = critique(event)
        risk = result.get("risk", "high")
        print(event["id"], risk, result.get("reasons"))
        if risk == "high":
            high += 1
sys.exit(1 if high else 0)

Fail the job on high risk. Log med for humans, then move on. Never auto-merge on low alone.

Step 5 — Make the ritual one command

clean:
    python redact.py --in traces.jsonl --out traces.clean.jsonl

contracts: clean
    python contract_gate.py traces.clean.jsonl

critique: contracts
    python critique_run.py traces.clean.jsonl

eval: critique
    python report.py traces.clean.jsonl

You should run make eval after every model bump. If that command needs a meeting, the harness is too heavy.

Decision table you can copy

Use this table in design reviews. Do not argue this choice from vibe.

Question	Frozen golden file	Contract test	Cheap remote critic
Does wording drift fail the build?	Yes, noisily	No	Only if risk rises
Does an extra tool get caught?	Sometimes	Yes, if listed	Often
Does a silent no-op get caught?	Rarely	Rarely	More often
Does a schema break get caught?	No	Yes	Weakly
Does it survive a vendor bump?	No	Yes	Yes, if the prompt is pinned
What should it cost per PR?	Near zero	Near zero	Must stay near zero

Freeze bytes that must not change across releases. That means SQL migrations, signed manifests, and PDF hashes. Do not freeze chat.

Why the critic belongs off your laptop

Local critique inherits your clipboard history. Local critique inherits forgotten .env files. Local critique vanishes when the lid closes.

A remote box gives you a repeatable host. You can wipe that box after each run. You can point CI at one hostname.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

You still need a cheap model and a cheap host. MonkeyCode is an open-source project with free model access. It also offers a free server option for remote runs.

Use that pair for the critic host and the critic model. Do not park your production agent there on day one. Traces and a linter belong there first.

This article will not quote quotas or hardware claims. Those numbers go stale faster than your fixtures. Read the current project docs before you pin any limit.

If you already have a spare VM, use it. The harness does not require a particular brand. The method should survive a vendor swap.

Limitations you should not hand-wave

A cheap critic is not an oracle for truth. It can bless fluent errors without shame. It can nag about style instead of harm.

It will miss domain bugs that need a human. It will miss races and partial writes. It will miss authz holes unless you feed it policy.

Remote calls flake under real networks. Set a timeout and retry once. Then fail closed, not open.

Free endpoints can throttle or vanish later. The job must fail loud on empty responses. A silent skip returns you to frozen files.

Do not treat critic JSON as ground truth. Treat it as a linter with opinions. Humans still own merge decisions.

Who should not run this

Do not upload regulated traces to a shared model. Do not critique live PII through this path. Do not score people with this output.

Do not replace effect tests with a critic paragraph. If the agent moves money, test the ledger first. Critique is a third layer, not the first.

If nobody owns the critic prompt, stop now. An unversioned critic is another moving model. Ownership is part of the test.

If your traces still contain customer secrets, stop now. Redaction is a gate, not a nicety. Clean traces or no remote call.

After the next vendor bump

The production model will move again soon. Your golden files will lie again soon. Plan for that lie before it ships.

Keep contracts in the same pipeline. Keep effect checks next to them. Keep a cheap critic on a clean host.

Throw away the frozen answers after each bump. Measure moving systems with living tests instead. Anything cheaper than that stack is theater.

If you need a remote critic box this week, read MonkeyCode's current docs. Run this harness on a handful of traces first.

Your Tool Schema Should Outlive the Model

Emery Chen — Sun, 20 Sep 2026 11:37:16 +0000

If your agent dies after a model swap, it was never finished. You shipped a vendor accent inside a prompt. A green eval on one stack is still a maybe.

Take the position

You should treat model disagreement as a merge blocker. Treat that disagreement as a schema bug. Do not treat that disagreement as model personality.

Two callers must emit the same tool names. They must fill the same argument keys. Divergence means the task text is underspecified.

Paid models hide that hole every day. They guess the missing enums without asking you. They complete file paths you never declared.

Cheap models stay ruder, and that helps. They follow the schema you actually wrote. That rudeness is the cheapest bug report available.

Single-model green is a rumor

A single endpoint will only agree with itself. That kind of agreement is easy and misleading. It cannot falsify vendor lock-in at all.

Teams then write longer judges after each miss. They add extra rubric bullets for comfort. They still query the same vendor stack anyway.

That pattern is how measurement quietly dies. The model drifts and the judge drifts with it. Your repository never held a truly fixed contract.

You need a second caller on purpose. Make that caller cheaper than the primary path. Run it on a machine that is not your laptop.

Compare the contract, not the essay

Diff the tool shapes and skip the essay. Ignore the assistant's closing paragraph every time. Remember that prose is not your API.

Check only these fields:

tool name
argument key set
required keys that arrived
enum membership
call order when order is part of the contract

Drop these fields every time:

chatty summaries
markdown flavor
hedge words
restated code with new comments

The essay is costume for the user. The tool list is the actual product. You should merge only on that product.

Proposed swap harness

The following harness is only a proposal. It is not a published benchmark. Do not cite it as production evidence.

1. Freeze a task fixture

{
  "id": "pin-formatter",
  "goal": "Pin the repo formatter to an exact version and open a PR.",
  "must_include_tools": ["read_file", "edit_file", "run_tests"]
}

Keep the goals short and operational. Keep the allowed tools as a closed set. Never hide control flow inside style notes.

2. Extract a stable shape

# Proposed harness. Unexecuted example. Pin your own clients.
import json, os, sys, urllib.request

def chat(url, key, model, messages, tools):
    payload = {
        "model": model,
        "messages": messages,
        "tools": tools,
        "temperature": 0,
    }
    req = urllib.request.Request(
        url,
        data=json.dumps(payload).encode(),
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + key,
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)

def tool_shape(payload):
    shapes = []
    for choice in payload.get("choices", []):
        message = choice.get("message", {})
        for call in message.get("tool_calls") or []:
            fn = call.get("function", {})
            raw = fn.get("arguments") or "{}"
            args = json.loads(raw) if isinstance(raw, str) else raw
            keys = sorted(args) if isinstance(args, dict) else []
            shapes.append({"name": fn.get("name"), "keys": keys})
    return shapes

def main():
    task = json.load(open(sys.argv[1]))
    tools = json.load(open(sys.argv[2]))
    messages = [{"role": "user", "content": task["goal"]}]
    primary = chat(
        os.environ["PRIMARY_URL"],
        os.environ["PRIMARY_KEY"],
        os.environ["PRIMARY_MODEL"],
        messages,
        tools,
    )
    shadow = chat(
        os.environ["SHADOW_URL"],
        os.environ["SHADOW_KEY"],
        os.environ["SHADOW_MODEL"],
        messages,
        tools,
    )
    left = tool_shape(primary)
    right = tool_shape(shadow)
    json.dump(
        {"id": task["id"], "primary": left, "shadow": right},
        sys.stdout,
        indent=2,
    )
    if left != right:
        sys.stderr.write("tool shape disagreement\n")
        sys.exit(2)

if __name__ == "__main__":
    main()

Keep temperature at zero for this gate. Keep argument keys sorted before compare. Keep tool names literal, never normalized by hand.

3. Fail the job in CI

# Proposed GitHub Actions job. Replace secrets with yours.
name: model-swap
on: [pull_request]
jobs:
  shadow:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python swap_test.py tasks/pin-formatter.json tools.json
        env:
          PRIMARY_URL: ${{ secrets.PRIMARY_URL }}
          PRIMARY_KEY: ${{ secrets.PRIMARY_KEY }}
          PRIMARY_MODEL: ${{ secrets.PRIMARY_MODEL }}
          SHADOW_URL: ${{ secrets.SHADOW_URL }}
          SHADOW_KEY: ${{ secrets.SHADOW_KEY }}
          SHADOW_MODEL: ${{ secrets.SHADOW_MODEL }}

Exit code 2 means you stop the merge. Do not bump the paid model to silence it. Fix the schema that leaked a vendor habit.

Decision table for review

Do not adjudicate disagreement inside chat threads. Put the observation on this table. Then follow the merge column without debate.

Observation	Merge?	Why
Same names, same keys, same order	Yes	Contract survived the swap
Extra optional keys only on paid	Warn	Paid model is padding arguments
Cheap model skips a required tool	No	Prompt relies on vendor inference
Different tool names, same goal	No	Task text is ambiguous
Cheap model loops until timeout	No	Stop condition is unspecified
Both refuse a disallowed tool	Yes	Safety instruction survived
Only paid model hits a private plugin	No	You shipped lock-in

Every "No" row is your bug. It is not the cheap model's personality. It is not a reason to hide the shadow run.

Why the shadow run must be remote and cheap

You need volume for this disagreement check. Isolation still matters more than raw volume. You cannot skip either constraint here.

A shadow model you never invoke is theater. A shadow model on your laptop is contaminated. Local files, caches, and tokens all leak.

You want a remote job you can rerun. You want a cost that does not train skipping. Cheap and remote is the point of the gate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project for this workflow. It offers free model access and a free server option. Park the shadow caller there when you need a remote box. You avoid standing up a second invoice. Pin the model id in the job. Pin the base URL beside that id. Store the tool-shape JSON next to the trace. If that free path is down, fail the job closed.

The merge rule still stands without that project. Any pinned remote shadow can play this role.

Debug disagreement in a fixed order

Do not start by rewriting the system prompt. Start with the tool schema instead. Change only one variable per rerun.

Walk this list in order every time:

Are tool names short verbs a second model can parse?
Are enums closed lists instead of free strings?
Are file paths constraints, not decorative examples?
Did you bury branching rules in prose?
Did you assume one vendor's function-call quirks?

Most so-called quality gaps die at step three. You wrote a suggestion, not a contract. The paid model treated the suggestion like law.

Tighten that one field and stop there. Rerun the pair on the same fixture. Keep both traces next to the schema change.

Limitations

This gate is blunt on purpose. It will punish agents that should improvise. It will also punish research spikes and sketch work.

It assumes both endpoints speak the same tool protocol. It assumes temperature remains zero during the compare. It assumes the task is tool-shaped and pinnable.

Agreement does not mean the edit was right. Two models can share one wrong refactor. This test catches lock-in, not business correctness.

You still need unit tests around the tools. You still need replay for the session. You still need a human on the patch.

Unpinned shadows are noise in CI. Noise does not belong on the merge gate.

Who should not use this

Skip this if you sell a single-vendor assistant on purpose. Skip it for story generation and marketing copy. Skip it for one-off notebooks with no tools.

Skip it if your tools are free text blobs. Diffing essays across models returns vibe scores. Those scores do not travel between vendors at all.

Skip it if you cannot name both models in the job. A floating alias is not a real fixture. Fixtures need identifiers that you can grep.

What to run this week

Pick five tasks from your real tickets. Reject demo prompts that have no tools. Reject poems and other toy summary prompts.

Run the proposed harness on those five. Read only the shape JSON output file. Apply the table without any extra debate.

Look for a hidden vendor guess in the shapes. Put that guess into the tool schema. Rerun the pair until the shapes match.

A model you can swap is an agent. A model you cannot swap is a demo. Keep the invoice off the critical path.

An Agent That Needs Your Laptop Is Not Ready

Emery Chen — Sat, 19 Sep 2026 09:39:43 +0000

An agent that only succeeds on your laptop is not ready. That run borrowed your credentials, caches, and leftover files. Treat local IDE success as a sketch, never as evidence.

Local success is a contaminated sample

Your interactive shell is not a clean room for agents. It already holds SSH keys, cloud tokens, and package caches. It already holds git author data and private registry logins.

You did not measure the agent in isolation at all. You measured the agent plus your entire machine. Those two results are not interchangeable for shipping.

Shipping the laptop result is a category error. The agent never proved it could work elsewhere. It proved it could lean on you.

Name the contaminants before you argue prompts

Stop tuning the prompt while the machine is lying. List what leaked into the last “green” run. Then decide whether the merge still deserves a yes.

SSH agent sockets the model never requested
~/.netrc, ~/.aws, leftover gcloud application-default files
Global npm, pip, and cargo caches with private packages
Editor buffers that never reached git
Docker images you built last Tuesday
Environment variables from a forgotten direnv file
Previous agent patches still sitting unstaged
VPN routes, /etc/hosts hacks, and local DNS lies

If the task “worked,” which item did the heavy lifting? You cannot know from a chat pane. That ignorance is the actual defect.

The opinion, without a hedge

Fail the merge when the agent needs your laptop. A clean remote workspace is the first honest test. A disposable model budget is enough to run that test.

Paid frontier models can wait for later stages. Your identity isolation cannot wait at all. The first production-shaped question is blunt. Did this agent work without your credentials?

Vibe coding in a chat pane is fine. Calling that session engineering is not. Engineering starts when a stranger machine can finish the work.

What the laptop hides on purpose

Localhost hides clock, network, and permission shape. Your VPN makes private APIs look public. Your Docker socket makes “shell” feel like root.

Your git user.email is already configured. Your SSH key already signs commits. Your password manager already filled the .env.

None of that will exist on the on-call host. None of that will exist in CI. None of that will exist for the next hire.

A clean-room gate you can actually run

This section is a proposed workflow, not a vendor score. Swap in the remote box you already control. Keep the assertions even if the runner changes.

You need four pinned inputs, nothing fuzzy.

The git SHA under test
A task file the agent must complete
An allowlist of environment variables
A hard stop on wall clock and steps

If any input is “whatever is on my desk,” stop. You are not testing yet. You are narrating a lucky afternoon.

Provision a workspace that is not `$HOME`

Use a throwaway server. Clone only the SHA. Create a non-root user with no extra keys.

# proposal: clean-room bootstrap on a disposable host
set -euo pipefail
SHA="${SHA:?pin the commit}"
TASK="${TASK:?path to task.md}"
WORKDIR="/opt/agent-gate/${SHA}"

sudo useradd --create-home --shell /bin/bash agentgate || true
sudo mkdir -p "$WORKDIR"
sudo chown agentgate:agentgate "$WORKDIR"

sudo -u agentgate git clone --depth 1 "$REPO_URL" "$WORKDIR/src"
sudo -u agentgate git -C "$WORKDIR/src" fetch --depth 1 origin "$SHA"
sudo -u agentgate git -C "$WORKDIR/src" checkout --detach "$SHA"
sudo -u agentgate cp "$TASK" "$WORKDIR/task.md"

Replace $REPO_URL with your repository. Do not copy your local .env. Do not forward your SSH agent socket.

If the job needs a secret, inject one scoped token. Delete that token in teardown. A long-lived key on a “test” box is still production access.

Strip the environment on purpose

Default-deny the process environment. Pass only what the task truly needs. env -i is the whole technique.

# proposal: default-deny env, then exec the agent runner
ALLOW="PATH,HOME,LANG,TASK_PATH,MODEL_BASE_URL,MODEL_API_KEY"
ENV_FILE="$(mktemp)"

for key in $(echo "$ALLOW" | tr ',' ' '); do
  eval "val=\${$key-}"
  [ -n "$val" ] && printf '%s=%s\n' "$key" "$val" >> "$ENV_FILE"
done

sudo -u agentgate env -i \
  HOME=/home/agentgate \
  PATH=/usr/local/bin:/usr/bin:/bin \
  TASK_PATH="$WORKDIR/task.md" \
  MODEL_BASE_URL="${MODEL_BASE_URL:?}" \
  MODEL_API_KEY="${MODEL_API_KEY:?}" \
  /usr/local/bin/run-agent \
    --workdir "$WORKDIR/src" \
    --task "$WORKDIR/task.md" \
    --trace "$WORKDIR/trace.jsonl"

Label the runner binary as a proposal in your repo. The brand of runner is not the lesson. The empty environment is the lesson.

Assert identity leakage, not writing quality

Do not grade the README tone. Inspect the tree. Inspect the trace for credential tools.

# proposal: fail the gate on identity leakage and noisy traces
from pathlib import Path
import json
import subprocess
import sys

sha = sys.argv[1]
root = Path("/opt/agent-gate") / sha
src = root / "src"
trace = root / "trace.jsonl"

diff = subprocess.check_output(
    ["git", "-C", src, "status", "--porcelain"], text=True
)
if not diff.strip():
    raise SystemExit("agent produced no tree change; task likely skipped")

forbidden = ("id_rsa", ".aws/", ".git-credentials", ".netrc", ".env")
for line in diff.splitlines():
    path = line[3:]
    if any(token in path for token in forbidden):
        raise SystemExit(f"identity file appeared in diff: {path}")

steps = 0
for raw in trace.read_text().splitlines():
    event = json.loads(raw)
    steps += 1
    if event.get("tool") not in {"shell", "exec"}:
        continue
    cmd = str(event.get("args", {}).get("command", ""))
    banned = ("ssh-add", "aws sts", "gcloud auth", "pbcopy", "security dump")
    if any(token in cmd for token in banned):
        raise SystemExit(f"tool reached identity APIs: {cmd!r}")

if steps > 80:
    raise SystemExit(f"too many steps for a cheap gate: {steps}")

print("clean-room gate passed")

Pin your own step budget in team docs. The 80 above is an example threshold. This article does not present it as a measured result.

Tear the box down every time

Leave no home directory behind. Leave no key in shell history. Optional teardown is how secrets rot.

sudo pkill -u agentgate || true
sudo userdel -r agentgate || true
sudo rm -rf "/opt/agent-gate/${SHA}"
unset MODEL_API_KEY

If teardown feels optional in your head, the gate is already compromised. A leftover user is a leftover identity.

Decision table: when local is allowed

Situation	Local IDE	Clean remote workspace
Spike to learn an unfamiliar API	Yes	No
Commit you intend to merge	No	Yes
Stakeholder demo	No	Yes, then keep the trace
Debugging a flaky tool parser	Yes, then promote	Yes before merge
Anything that can touch credentials	Never	Yes, with a scoped token
“It works on my machine” as evidence	Never	Required

If your row is “merge” and your column is “laptop,” you are guessing. Guesses do not belong in the main branch.

Why a free model belongs in this gate

The gate must be cheap enough to fail. Expensive models make teams skip the remote run. Skipping the remote run puts the laptop back in the loop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open source project with free model access and a free server option. Those two facts matter here only because the gate needs a disposable workspace and a budget you can burn. This article does not claim quotas, hardware, latency, or model names. Point MODEL_BASE_URL at the endpoint you actually operate.

If you already run this gate on your own iron, keep doing that. The isolation rule does not require a brand. It requires a box that is not your laptop.

What this gate will not catch

It will not prove the generated code is correct. It will not prove the prompt is optimal. It will not replace unit tests, contract tests, or a human review.

It will not stop a determined agent from doing something stupid inside the sandbox. It only removes your laptop from the evidence. That is a smaller claim, and smaller claims are defensible.

Clock skew, rate limits, and cold starts still exist remotely. Good. Those are production-shaped problems. Hiding them on localhost is how stalled agents reach users.

A passed gate is not a benchmark. A passed gate is not a quality score. A passed gate only says the agent did not need your privileged machine.

Who should not use this approach

Skip this workflow if you do not ship agents. Skip it if the task is a private editor thought. Skip it if you cannot isolate secrets from the model.

Do not use a shared box for customer data. Do not paste production credentials into any remote agent runtime. Do not treat a passed gate as a performance study.

If your compliance model forbids third-party inference, host the model yourself. The script still works. Only the endpoint changes.

Solo spikes can stay local. Merges cannot. That split is the whole policy.

Operating rule you can paste on the wiki

Write four lines. Enforce them on pull requests. Ignore them in chat sketches.

Local chat is a sketch.
A clean remote workspace is the first test.
Tool traces are the artifact you review.
No trace, no merge.

You can delete every product sentence and keep the rule. Agents fail in the environment you refused to measure. Measure a box that is not yours.

Stop Calling It Staging If the Agent Never Leaves Localhost

Emery Chen — Fri, 18 Sep 2026 08:01:22 +0000

Your laptop is not a real staging environment. A fluent local chat still proves almost nothing. Stage the agent remotely, or admit you guessed.

The claim

Local agent runs hide the failures that matter. You skip DNS, cold starts, and queue delay. You also skip boring contract breaks.

A remote canary is the cheapest honest gate. It does not need a larger model. It needs a frozen tool allowlist and a replayable fixture.

If the agent never left localhost, you did not stage it. A green demo is still a draft. Merge gates need exit codes, not screenshots.

What you should measure

Do not score the generated prose. Score the tool surface instead. Reviewers cannot replay a vibe.

A useful canary answers four questions:

Did the agent call a tool you never allowed?
Did a required argument key disappear?
Did a call exceed your step budget?
Did the remote host fail before the model did?

Those are merge questions. They are not demo questions. Chat quality can wait until the contract holds.

Freeze the contract first

Put the allowlist in git. Treat it like a public API. Prompts drift. Files get reviewed.

{
  "version": "2026-09-18",
  "max_steps": 8,
  "allow": [
    {
      "name": "repo_read",
      "required": ["path"],
      "optional": ["start_line", "end_line"]
    },
    {
      "name": "repo_search",
      "required": ["query"],
      "optional": ["globs", "max_hits"]
    },
    {
      "name": "http_get",
      "required": ["url"],
      "optional": ["timeout_ms"]
    }
  ],
  "deny": ["shell_exec", "fs_write", "net_raw"]
}

Label this file as the contract. Do not bury it in a system prompt. If a tool is missing here, the agent must not call it.

Build a fixture, not a vibe

A fixture is a frozen input plus expected shape. It is not a transcript you liked last night.

{
  "id": "canary-pr-summary-01",
  "goal": "Summarize PR 1842 using repo_read only.",
  "context": {
    "pr": 1842,
    "paths": [
      "src/billing/invoice.py",
      "tests/billing/test_invoice.py"
    ]
  },
  "expect": {
    "allowed_tools": ["repo_read"],
    "forbidden_tools": ["http_get", "repo_search"],
    "max_steps": 4,
    "must_read": ["src/billing/invoice.py"]
  }
}

Write a handful of these. You do not need a hundred. Coverage beats theater. Delete any fixture that cannot fail.

Run the canary off the laptop

The runner below is a labeled example. Wire it to your own remote endpoint. Keep tokens in the environment, not in git.

#!/usr/bin/env python3
"""Labeled example: remote agent canary. Not a production SDK."""

from __future__ import annotations

import json
import os
import sys
import urllib.error
import urllib.request
from pathlib import Path

CONTRACT = json.loads(Path("tools.allowlist.json").read_text())
FIXTURE = json.loads(Path("fixtures/canary-pr-summary-01.json").read_text())
BASE = os.environ["CANARY_BASE_URL"].rstrip("/")
TOKEN = os.environ.get("CANARY_TOKEN", "")


def post(path: str, payload: dict) -> dict:
    data = json.dumps(payload).encode()
    req = urllib.request.Request(
        f"{BASE}{path}",
        data=data,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {TOKEN}",
        },
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=60) as resp:
            return json.loads(resp.read().decode())
    except urllib.error.URLError as exc:
        print(f"remote_error {exc}")
        sys.exit(2)


def violations(trace: dict) -> list[str]:
    issues: list[str] = []
    allowed = {item["name"] for item in CONTRACT["allow"]}
    denied = set(CONTRACT["deny"])
    steps = trace.get("tool_calls", [])
    if len(steps) > FIXTURE["expect"]["max_steps"]:
        issues.append("step_budget")
    for call in steps:
        name = call.get("name")
        args = call.get("args") or {}
        if name in denied or name not in allowed:
            issues.append(f"denied_or_unknown:{name}")
            continue
        spec = next(item for item in CONTRACT["allow"] if item["name"] == name)
        missing = [k for k in spec["required"] if k not in args]
        if missing:
            issues.append(f"missing_args:{name}:{','.join(missing)}")
        if name not in FIXTURE["expect"]["allowed_tools"]:
            issues.append(f"fixture_forbid:{name}")
    for path in FIXTURE["expect"]["must_read"]:
        reads = [
            c
            for c in steps
            if c.get("name") == "repo_read"
            and (c.get("args") or {}).get("path") == path
        ]
        if not reads:
            issues.append(f"unread:{path}")
    return issues


def main() -> None:
    trace = post("/v1/agent/run", {"fixture": FIXTURE, "contract": CONTRACT})
    issues = violations(trace)
    report = {
        "fixture": FIXTURE["id"],
        "contract": CONTRACT["version"],
        "steps": len(trace.get("tool_calls", [])),
        "issues": issues,
        "ok": not issues,
    }
    print(json.dumps(report, indent=2))
    sys.exit(0 if report["ok"] else 1)


if __name__ == "__main__":
    main()

Exit codes matter more than summaries.

0 means the contract held.
1 means the agent broke the contract.
2 means the remote host failed first.

That last code is the one local demos never show you. Treat host failure as a blocked merge, not a skip.

Put it in CI, not in a screenshot

# .github/workflows/agent-canary.yml
name: agent-canary
on:
  pull_request:
    paths:
      - "tools.allowlist.json"
      - "fixtures/**"
      - "agents/**"
jobs:
  canary:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: run remote canary
        env:
          CANARY_BASE_URL: ${{ secrets.CANARY_BASE_URL }}
          CANARY_TOKEN: ${{ secrets.CANARY_TOKEN }}
        run: python3 canary.py

You can also run it by hand:

export CANARY_BASE_URL="https://canary.example.internal"
export CANARY_TOKEN="***"
python3 canary.py
echo $?

If you cannot point that command at a host you do not own, you are still on localhost. Rename the ritual. It is not staging.

Decision table

Use this at review time. Do not invent extra scores.

Remote result	Tool contract	Merge action
Host unreachable	Unknown	Block. Fix the canary host.
Exit 2	Untested	Block. You learned nothing.
Exit 1, denied tool	Broken	Block. Tighten the allowlist or the agent.
Exit 1, missing args	Broken	Block. The schema drifted.
Exit 1, unread path	Weak	Block. The fixture was skipped.
Exit 0	Held	Allow human review to continue.

Notice the table never asks if the summary sounded good. Sound is not a gate. Contract shape is.

Why a free remote slot changes the argument

You do not need a paid cluster to learn this. You need a host that is not your laptop. Local RAM will not emulate queueing, shared CPU, or foreign DNS.

MonkeyCode offers free model access and a free server option. That pair is useful here as a remote canary target, not as a demo coupon.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the same contract file. Keep the same fixture. Point CANARY_BASE_URL at the free server, then watch which failures appear only off-laptop.

If that path cannot host your canary, do not pretend a local chat replaced it. Find another remote. The method does not depend on one vendor.

What this will not catch

This canary does not prove correctness. It proves the tool surface stayed inside a box. Green canaries still need humans.

It will miss:

Quiet logic errors inside an allowed tool
Poisoned files the agent was allowed to read
Cost blowups below the step cap
Reviewer fatigue after a green report

It also assumes you can freeze tools. If your agent is a raw shell, stop. An allowlist of shell_exec is not a contract. It is a shrug.

Who should not use this

Skip this workflow if you are:

Pairing on a throwaway spike
Editing copy with no tools
Running a model with no function calls
Unable to send fixtures off-laptop

In those cases a remote canary is overhead. Do not cargo-cult the YAML. A spike can stay local until tools exist.

Limitations you should write down

Remote does not mean representative. A free server may differ from production in queue time and payload limits. This article will not invent numbers for those gaps.

Treat the canary as a bias detector. Localhost is optimistic. Remote is less so. Neither is production.

Pin versions of the contract. Rotate fixtures when the product changes. Delete fixtures that always pass. A canary that cannot fail is decoration.

A short review script

Paste this into the pull request template:

## Agent canary
- [ ] `tools.allowlist.json` version bumped if tools changed
- [ ] Fixture IDs listed
- [ ] Remote canary exit code pasted
- [ ] No new deny-list hits
- [ ] No localhost-only screenshot as evidence

If a reviewer sees a chat screenshot, they should ask for the exit code. If they cannot get one, the patch is not staged. No exit code, no merge.

Close

You already know localhost lies. You still present it as staging. Stop.

Freeze the tool contract. Replay a fixture on a remote host. Merge only when the canary returns zero.

A fluent local agent is a draft. A remote canary is a gate. If you already have a free remote slot, point the runner at it and paste the JSON report.