DEV Community: Emery Chen

If the Model Moved, Burn the Golden Files

Emery Chen — Mon, 21 Sep 2026 13:36:00 +0000

Frozen golden files cannot survive a model upgrade. You should run a living critic on a clean host.

The argument

Your agent tests still freeze last week's answers. Then the serving model changes overnight without warning. The suite fails on cosmetic wording after that bump.

Or it stays green while real behavior drifts. You do not hold a quality signal anymore. You hold a snapshot of a dead model.

Treat those frozen answers as unpaid test debt. A moving model needs a living check. A frozen string is not that check.

The tests did not get weaker

People now ask if AI outgrew our tests. Ask first whether fixtures outlived the recorded model. Those are different failures with different fixes.

A golden string still assumes one correct reply. Agents do not emit one stable reply anymore. They emit tool traces, side effects, and leftover prose.

Wording shifts after every quiet vendor bump. Call order also shifts after a schema tweak. Your diff tool then screams about both events.

Neither scream maps cleanly to user harm. Worse cases stay silent under the same fixtures. The new model skips a required precondition check.

Your frozen paragraph still matches yesterday's run. The build stays green and then ships. That is not coverage. That is nostalgia.

Frozen files punish harmless change far too often. They also miss harmful change far too often. That combination makes a bad test instrument.

Three layers beat one snapshot

Keep three layers in the same job. Do not collapse them into chat scores.

Contracts

Contracts check JSON shape and required tools. Those checks remain deterministic across later model swaps.

Effects

Effects checks inspect the world after the run. Count rows, HTTP codes, and file hashes.

Critique

Critique is a second model reading the trace. It hunts plans that are valid and still dumb.

Contracts catch schema rot early. Effects catch real damage in the system. Critique catches confident nonsense that contracts allow.

Most teams stop at contracts or prose. Few run a critic on every bump. The missing layer is almost always the critic.

Opinion: the critic must stay cheap

Here is the hard claim for this piece. Costly critique will not run on every bump.

You will postpone that job until the customer demo. You will skip it during the actual incident. Skipped critique is how untested agents ship.

The critic should not be the actor model. The actor is expensive and already moving. The critic should be cheaper and pinned in config.

The critic should not share your laptop either. Your laptop holds secrets and dirty state. Your laptop also sleeps while CI expects an answer.

Park redacted traces on a clean remote host. Then ask the critic to lint the trace. Keep production credentials off that host.

Artifact: a living eval harness

This section is a proposed workflow. It is not a vendor benchmark. It claims no accuracy number and no leaderboard.

You capture traces, gate contracts, then call a critic. Keep each step boring enough to rerun. Glamour is how eval suites rot.

Step 1 — Capture traces as JSONL

Write one JSON object per agent run. Keep the capture schema boring, small, and stable.

{"id": "pr-4412-a", "goal": "create invoice draft", "tools": [{"name": "db.query", "args": {"table": "customers", "id": 918}}, {"name": "invoices.create", "args": {"customer_id": 918, "cents": 1299}}], "final": "Draft created for customer 918"}

Redact tokens before anything hits disk. Redact names if your policy requires that step.

python capture_traces.py --task invoices --out traces.jsonl
python redact.py --in traces.jsonl --out traces.clean.jsonl

Step 2 — Fail closed on contracts

Run deterministic checks before any model call. Do not spend critique budget on garbage traces.

# contract_gate.py — labeled proposal, not a shipped product
import json, sys

REQUIRED = {"db.query", "invoices.create"}
FORBIDDEN = ("password", "ssn", "authorization", "api_key")

def errors_for(event):
    problems = []
    names = {step["name"] for step in event.get("tools", [])}
    missing = REQUIRED - names
    if missing:
        problems.append(f"missing tools: {sorted(missing)}")
    blob = json.dumps(event).lower()
    for needle in FORBIDDEN:
        if needle in blob:
            problems.append(f"possible secret field: {needle}")
    if not event.get("goal"):
        problems.append("empty goal")
    return problems

fail = 0
with open(sys.argv[1]) as handle:
    for line in handle:
        event = json.loads(line)
        problems = errors_for(event)
        if problems:
            fail += 1
            print(event.get("id"), problems)
sys.exit(1 if fail else 0)

Call it like any other unit test. A red contract means stop. Critique comes only after this gate.

python contract_gate.py traces.clean.jsonl

Step 3 — Pin the critic prompt in git

The prompt is part of the test suite. Version it and review it like code.

You are a trace linter, not a chat assistant.
Return JSON only with keys risk and reasons.
risk must be low, med, or high.
Flag missing preconditions, extra tools, and unverifiable claims.
Do not rewrite the agent. Do not praise the agent.

If nobody reviews this file, you cannot trust the critic. An unowned prompt is another moving model.

Step 4 — Call a cheap HTTP endpoint

Keep the HTTP client dull on purpose. Do not hide retries inside a heavy SDK.

# critique_run.py — labeled proposal
import json, os, sys, urllib.request

ENDPOINT = os.environ["CRITIC_ENDPOINT"]
TOKEN = os.environ["CRITIC_TOKEN"]
PROMPT = open("critic_prompt.txt").read()

def critique(event):
    body = json.dumps({
        "instructions": PROMPT,
        "trace": event,
    }).encode()
    req = urllib.request.Request(
        ENDPOINT,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {TOKEN}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.loads(resp.read().decode())

high = 0
with open(sys.argv[1]) as handle:
    for line in handle:
        event = json.loads(line)
        result = critique(event)
        risk = result.get("risk", "high")
        print(event["id"], risk, result.get("reasons"))
        if risk == "high":
            high += 1
sys.exit(1 if high else 0)

Fail the job on high risk. Log med for humans, then move on. Never auto-merge on low alone.

Step 5 — Make the ritual one command

clean:
    python redact.py --in traces.jsonl --out traces.clean.jsonl

contracts: clean
    python contract_gate.py traces.clean.jsonl

critique: contracts
    python critique_run.py traces.clean.jsonl

eval: critique
    python report.py traces.clean.jsonl

You should run make eval after every model bump. If that command needs a meeting, the harness is too heavy.

Decision table you can copy

Use this table in design reviews. Do not argue this choice from vibe.

Question	Frozen golden file	Contract test	Cheap remote critic
Does wording drift fail the build?	Yes, noisily	No	Only if risk rises
Does an extra tool get caught?	Sometimes	Yes, if listed	Often
Does a silent no-op get caught?	Rarely	Rarely	More often
Does a schema break get caught?	No	Yes	Weakly
Does it survive a vendor bump?	No	Yes	Yes, if the prompt is pinned
What should it cost per PR?	Near zero	Near zero	Must stay near zero

Freeze bytes that must not change across releases. That means SQL migrations, signed manifests, and PDF hashes. Do not freeze chat.

Why the critic belongs off your laptop

Local critique inherits your clipboard history. Local critique inherits forgotten .env files. Local critique vanishes when the lid closes.

A remote box gives you a repeatable host. You can wipe that box after each run. You can point CI at one hostname.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

You still need a cheap model and a cheap host. MonkeyCode is an open-source project with free model access. It also offers a free server option for remote runs.

Use that pair for the critic host and the critic model. Do not park your production agent there on day one. Traces and a linter belong there first.

This article will not quote quotas or hardware claims. Those numbers go stale faster than your fixtures. Read the current project docs before you pin any limit.

If you already have a spare VM, use it. The harness does not require a particular brand. The method should survive a vendor swap.

Limitations you should not hand-wave

A cheap critic is not an oracle for truth. It can bless fluent errors without shame. It can nag about style instead of harm.

It will miss domain bugs that need a human. It will miss races and partial writes. It will miss authz holes unless you feed it policy.

Remote calls flake under real networks. Set a timeout and retry once. Then fail closed, not open.

Free endpoints can throttle or vanish later. The job must fail loud on empty responses. A silent skip returns you to frozen files.

Do not treat critic JSON as ground truth. Treat it as a linter with opinions. Humans still own merge decisions.

Who should not run this

Do not upload regulated traces to a shared model. Do not critique live PII through this path. Do not score people with this output.

Do not replace effect tests with a critic paragraph. If the agent moves money, test the ledger first. Critique is a third layer, not the first.

If nobody owns the critic prompt, stop now. An unversioned critic is another moving model. Ownership is part of the test.

If your traces still contain customer secrets, stop now. Redaction is a gate, not a nicety. Clean traces or no remote call.

After the next vendor bump

The production model will move again soon. Your golden files will lie again soon. Plan for that lie before it ships.

Keep contracts in the same pipeline. Keep effect checks next to them. Keep a cheap critic on a clean host.

Throw away the frozen answers after each bump. Measure moving systems with living tests instead. Anything cheaper than that stack is theater.

If you need a remote critic box this week, read MonkeyCode's current docs. Run this harness on a handful of traces first.

Your Tool Schema Should Outlive the Model

Emery Chen — Sun, 20 Sep 2026 11:37:16 +0000

If your agent dies after a model swap, it was never finished. You shipped a vendor accent inside a prompt. A green eval on one stack is still a maybe.

Take the position

You should treat model disagreement as a merge blocker. Treat that disagreement as a schema bug. Do not treat that disagreement as model personality.

Two callers must emit the same tool names. They must fill the same argument keys. Divergence means the task text is underspecified.

Paid models hide that hole every day. They guess the missing enums without asking you. They complete file paths you never declared.

Cheap models stay ruder, and that helps. They follow the schema you actually wrote. That rudeness is the cheapest bug report available.

Single-model green is a rumor

A single endpoint will only agree with itself. That kind of agreement is easy and misleading. It cannot falsify vendor lock-in at all.

Teams then write longer judges after each miss. They add extra rubric bullets for comfort. They still query the same vendor stack anyway.

That pattern is how measurement quietly dies. The model drifts and the judge drifts with it. Your repository never held a truly fixed contract.

You need a second caller on purpose. Make that caller cheaper than the primary path. Run it on a machine that is not your laptop.

Compare the contract, not the essay

Diff the tool shapes and skip the essay. Ignore the assistant's closing paragraph every time. Remember that prose is not your API.

Check only these fields:

tool name
argument key set
required keys that arrived
enum membership
call order when order is part of the contract

Drop these fields every time:

chatty summaries
markdown flavor
hedge words
restated code with new comments

The essay is costume for the user. The tool list is the actual product. You should merge only on that product.

Proposed swap harness

The following harness is only a proposal. It is not a published benchmark. Do not cite it as production evidence.

1. Freeze a task fixture

{
  "id": "pin-formatter",
  "goal": "Pin the repo formatter to an exact version and open a PR.",
  "must_include_tools": ["read_file", "edit_file", "run_tests"]
}

Keep the goals short and operational. Keep the allowed tools as a closed set. Never hide control flow inside style notes.

2. Extract a stable shape

# Proposed harness. Unexecuted example. Pin your own clients.
import json, os, sys, urllib.request

def chat(url, key, model, messages, tools):
    payload = {
        "model": model,
        "messages": messages,
        "tools": tools,
        "temperature": 0,
    }
    req = urllib.request.Request(
        url,
        data=json.dumps(payload).encode(),
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + key,
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)

def tool_shape(payload):
    shapes = []
    for choice in payload.get("choices", []):
        message = choice.get("message", {})
        for call in message.get("tool_calls") or []:
            fn = call.get("function", {})
            raw = fn.get("arguments") or "{}"
            args = json.loads(raw) if isinstance(raw, str) else raw
            keys = sorted(args) if isinstance(args, dict) else []
            shapes.append({"name": fn.get("name"), "keys": keys})
    return shapes

def main():
    task = json.load(open(sys.argv[1]))
    tools = json.load(open(sys.argv[2]))
    messages = [{"role": "user", "content": task["goal"]}]
    primary = chat(
        os.environ["PRIMARY_URL"],
        os.environ["PRIMARY_KEY"],
        os.environ["PRIMARY_MODEL"],
        messages,
        tools,
    )
    shadow = chat(
        os.environ["SHADOW_URL"],
        os.environ["SHADOW_KEY"],
        os.environ["SHADOW_MODEL"],
        messages,
        tools,
    )
    left = tool_shape(primary)
    right = tool_shape(shadow)
    json.dump(
        {"id": task["id"], "primary": left, "shadow": right},
        sys.stdout,
        indent=2,
    )
    if left != right:
        sys.stderr.write("tool shape disagreement\n")
        sys.exit(2)

if __name__ == "__main__":
    main()

Keep temperature at zero for this gate. Keep argument keys sorted before compare. Keep tool names literal, never normalized by hand.

3. Fail the job in CI

# Proposed GitHub Actions job. Replace secrets with yours.
name: model-swap
on: [pull_request]
jobs:
  shadow:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python swap_test.py tasks/pin-formatter.json tools.json
        env:
          PRIMARY_URL: ${{ secrets.PRIMARY_URL }}
          PRIMARY_KEY: ${{ secrets.PRIMARY_KEY }}
          PRIMARY_MODEL: ${{ secrets.PRIMARY_MODEL }}
          SHADOW_URL: ${{ secrets.SHADOW_URL }}
          SHADOW_KEY: ${{ secrets.SHADOW_KEY }}
          SHADOW_MODEL: ${{ secrets.SHADOW_MODEL }}

Exit code 2 means you stop the merge. Do not bump the paid model to silence it. Fix the schema that leaked a vendor habit.

Decision table for review

Do not adjudicate disagreement inside chat threads. Put the observation on this table. Then follow the merge column without debate.

Observation	Merge?	Why
Same names, same keys, same order	Yes	Contract survived the swap
Extra optional keys only on paid	Warn	Paid model is padding arguments
Cheap model skips a required tool	No	Prompt relies on vendor inference
Different tool names, same goal	No	Task text is ambiguous
Cheap model loops until timeout	No	Stop condition is unspecified
Both refuse a disallowed tool	Yes	Safety instruction survived
Only paid model hits a private plugin	No	You shipped lock-in

Every "No" row is your bug. It is not the cheap model's personality. It is not a reason to hide the shadow run.

Why the shadow run must be remote and cheap

You need volume for this disagreement check. Isolation still matters more than raw volume. You cannot skip either constraint here.

A shadow model you never invoke is theater. A shadow model on your laptop is contaminated. Local files, caches, and tokens all leak.

You want a remote job you can rerun. You want a cost that does not train skipping. Cheap and remote is the point of the gate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project for this workflow. It offers free model access and a free server option. Park the shadow caller there when you need a remote box. You avoid standing up a second invoice. Pin the model id in the job. Pin the base URL beside that id. Store the tool-shape JSON next to the trace. If that free path is down, fail the job closed.

The merge rule still stands without that project. Any pinned remote shadow can play this role.

Debug disagreement in a fixed order

Do not start by rewriting the system prompt. Start with the tool schema instead. Change only one variable per rerun.

Walk this list in order every time:

Are tool names short verbs a second model can parse?
Are enums closed lists instead of free strings?
Are file paths constraints, not decorative examples?
Did you bury branching rules in prose?
Did you assume one vendor's function-call quirks?

Most so-called quality gaps die at step three. You wrote a suggestion, not a contract. The paid model treated the suggestion like law.

Tighten that one field and stop there. Rerun the pair on the same fixture. Keep both traces next to the schema change.

Limitations

This gate is blunt on purpose. It will punish agents that should improvise. It will also punish research spikes and sketch work.

It assumes both endpoints speak the same tool protocol. It assumes temperature remains zero during the compare. It assumes the task is tool-shaped and pinnable.

Agreement does not mean the edit was right. Two models can share one wrong refactor. This test catches lock-in, not business correctness.

You still need unit tests around the tools. You still need replay for the session. You still need a human on the patch.

Unpinned shadows are noise in CI. Noise does not belong on the merge gate.

Who should not use this

Skip this if you sell a single-vendor assistant on purpose. Skip it for story generation and marketing copy. Skip it for one-off notebooks with no tools.

Skip it if your tools are free text blobs. Diffing essays across models returns vibe scores. Those scores do not travel between vendors at all.

Skip it if you cannot name both models in the job. A floating alias is not a real fixture. Fixtures need identifiers that you can grep.

What to run this week

Pick five tasks from your real tickets. Reject demo prompts that have no tools. Reject poems and other toy summary prompts.

Run the proposed harness on those five. Read only the shape JSON output file. Apply the table without any extra debate.

Look for a hidden vendor guess in the shapes. Put that guess into the tool schema. Rerun the pair until the shapes match.

A model you can swap is an agent. A model you cannot swap is a demo. Keep the invoice off the critical path.

An Agent That Needs Your Laptop Is Not Ready

Emery Chen — Sat, 19 Sep 2026 09:39:43 +0000

An agent that only succeeds on your laptop is not ready. That run borrowed your credentials, caches, and leftover files. Treat local IDE success as a sketch, never as evidence.

Local success is a contaminated sample

Your interactive shell is not a clean room for agents. It already holds SSH keys, cloud tokens, and package caches. It already holds git author data and private registry logins.

You did not measure the agent in isolation at all. You measured the agent plus your entire machine. Those two results are not interchangeable for shipping.

Shipping the laptop result is a category error. The agent never proved it could work elsewhere. It proved it could lean on you.

Name the contaminants before you argue prompts

Stop tuning the prompt while the machine is lying. List what leaked into the last “green” run. Then decide whether the merge still deserves a yes.

SSH agent sockets the model never requested
~/.netrc, ~/.aws, leftover gcloud application-default files
Global npm, pip, and cargo caches with private packages
Editor buffers that never reached git
Docker images you built last Tuesday
Environment variables from a forgotten direnv file
Previous agent patches still sitting unstaged
VPN routes, /etc/hosts hacks, and local DNS lies

If the task “worked,” which item did the heavy lifting? You cannot know from a chat pane. That ignorance is the actual defect.

The opinion, without a hedge

Fail the merge when the agent needs your laptop. A clean remote workspace is the first honest test. A disposable model budget is enough to run that test.

Paid frontier models can wait for later stages. Your identity isolation cannot wait at all. The first production-shaped question is blunt. Did this agent work without your credentials?

Vibe coding in a chat pane is fine. Calling that session engineering is not. Engineering starts when a stranger machine can finish the work.

What the laptop hides on purpose

Localhost hides clock, network, and permission shape. Your VPN makes private APIs look public. Your Docker socket makes “shell” feel like root.

Your git user.email is already configured. Your SSH key already signs commits. Your password manager already filled the .env.

None of that will exist on the on-call host. None of that will exist in CI. None of that will exist for the next hire.

A clean-room gate you can actually run

This section is a proposed workflow, not a vendor score. Swap in the remote box you already control. Keep the assertions even if the runner changes.

You need four pinned inputs, nothing fuzzy.

The git SHA under test
A task file the agent must complete
An allowlist of environment variables
A hard stop on wall clock and steps

If any input is “whatever is on my desk,” stop. You are not testing yet. You are narrating a lucky afternoon.

Provision a workspace that is not `$HOME`

Use a throwaway server. Clone only the SHA. Create a non-root user with no extra keys.

# proposal: clean-room bootstrap on a disposable host
set -euo pipefail
SHA="${SHA:?pin the commit}"
TASK="${TASK:?path to task.md}"
WORKDIR="/opt/agent-gate/${SHA}"

sudo useradd --create-home --shell /bin/bash agentgate || true
sudo mkdir -p "$WORKDIR"
sudo chown agentgate:agentgate "$WORKDIR"

sudo -u agentgate git clone --depth 1 "$REPO_URL" "$WORKDIR/src"
sudo -u agentgate git -C "$WORKDIR/src" fetch --depth 1 origin "$SHA"
sudo -u agentgate git -C "$WORKDIR/src" checkout --detach "$SHA"
sudo -u agentgate cp "$TASK" "$WORKDIR/task.md"

Replace $REPO_URL with your repository. Do not copy your local .env. Do not forward your SSH agent socket.

If the job needs a secret, inject one scoped token. Delete that token in teardown. A long-lived key on a “test” box is still production access.

Strip the environment on purpose

Default-deny the process environment. Pass only what the task truly needs. env -i is the whole technique.

# proposal: default-deny env, then exec the agent runner
ALLOW="PATH,HOME,LANG,TASK_PATH,MODEL_BASE_URL,MODEL_API_KEY"
ENV_FILE="$(mktemp)"

for key in $(echo "$ALLOW" | tr ',' ' '); do
  eval "val=\${$key-}"
  [ -n "$val" ] && printf '%s=%s\n' "$key" "$val" >> "$ENV_FILE"
done

sudo -u agentgate env -i \
  HOME=/home/agentgate \
  PATH=/usr/local/bin:/usr/bin:/bin \
  TASK_PATH="$WORKDIR/task.md" \
  MODEL_BASE_URL="${MODEL_BASE_URL:?}" \
  MODEL_API_KEY="${MODEL_API_KEY:?}" \
  /usr/local/bin/run-agent \
    --workdir "$WORKDIR/src" \
    --task "$WORKDIR/task.md" \
    --trace "$WORKDIR/trace.jsonl"

Label the runner binary as a proposal in your repo. The brand of runner is not the lesson. The empty environment is the lesson.

Assert identity leakage, not writing quality

Do not grade the README tone. Inspect the tree. Inspect the trace for credential tools.

# proposal: fail the gate on identity leakage and noisy traces
from pathlib import Path
import json
import subprocess
import sys

sha = sys.argv[1]
root = Path("/opt/agent-gate") / sha
src = root / "src"
trace = root / "trace.jsonl"

diff = subprocess.check_output(
    ["git", "-C", src, "status", "--porcelain"], text=True
)
if not diff.strip():
    raise SystemExit("agent produced no tree change; task likely skipped")

forbidden = ("id_rsa", ".aws/", ".git-credentials", ".netrc", ".env")
for line in diff.splitlines():
    path = line[3:]
    if any(token in path for token in forbidden):
        raise SystemExit(f"identity file appeared in diff: {path}")

steps = 0
for raw in trace.read_text().splitlines():
    event = json.loads(raw)
    steps += 1
    if event.get("tool") not in {"shell", "exec"}:
        continue
    cmd = str(event.get("args", {}).get("command", ""))
    banned = ("ssh-add", "aws sts", "gcloud auth", "pbcopy", "security dump")
    if any(token in cmd for token in banned):
        raise SystemExit(f"tool reached identity APIs: {cmd!r}")

if steps > 80:
    raise SystemExit(f"too many steps for a cheap gate: {steps}")

print("clean-room gate passed")

Pin your own step budget in team docs. The 80 above is an example threshold. This article does not present it as a measured result.

Tear the box down every time

Leave no home directory behind. Leave no key in shell history. Optional teardown is how secrets rot.

sudo pkill -u agentgate || true
sudo userdel -r agentgate || true
sudo rm -rf "/opt/agent-gate/${SHA}"
unset MODEL_API_KEY

If teardown feels optional in your head, the gate is already compromised. A leftover user is a leftover identity.

Decision table: when local is allowed

Situation	Local IDE	Clean remote workspace
Spike to learn an unfamiliar API	Yes	No
Commit you intend to merge	No	Yes
Stakeholder demo	No	Yes, then keep the trace
Debugging a flaky tool parser	Yes, then promote	Yes before merge
Anything that can touch credentials	Never	Yes, with a scoped token
“It works on my machine” as evidence	Never	Required

If your row is “merge” and your column is “laptop,” you are guessing. Guesses do not belong in the main branch.

Why a free model belongs in this gate

The gate must be cheap enough to fail. Expensive models make teams skip the remote run. Skipping the remote run puts the laptop back in the loop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open source project with free model access and a free server option. Those two facts matter here only because the gate needs a disposable workspace and a budget you can burn. This article does not claim quotas, hardware, latency, or model names. Point MODEL_BASE_URL at the endpoint you actually operate.

If you already run this gate on your own iron, keep doing that. The isolation rule does not require a brand. It requires a box that is not your laptop.

What this gate will not catch

It will not prove the generated code is correct. It will not prove the prompt is optimal. It will not replace unit tests, contract tests, or a human review.

It will not stop a determined agent from doing something stupid inside the sandbox. It only removes your laptop from the evidence. That is a smaller claim, and smaller claims are defensible.

Clock skew, rate limits, and cold starts still exist remotely. Good. Those are production-shaped problems. Hiding them on localhost is how stalled agents reach users.

A passed gate is not a benchmark. A passed gate is not a quality score. A passed gate only says the agent did not need your privileged machine.

Who should not use this approach

Skip this workflow if you do not ship agents. Skip it if the task is a private editor thought. Skip it if you cannot isolate secrets from the model.

Do not use a shared box for customer data. Do not paste production credentials into any remote agent runtime. Do not treat a passed gate as a performance study.

If your compliance model forbids third-party inference, host the model yourself. The script still works. Only the endpoint changes.

Solo spikes can stay local. Merges cannot. That split is the whole policy.

Operating rule you can paste on the wiki

Write four lines. Enforce them on pull requests. Ignore them in chat sketches.

Local chat is a sketch.
A clean remote workspace is the first test.
Tool traces are the artifact you review.
No trace, no merge.

You can delete every product sentence and keep the rule. Agents fail in the environment you refused to measure. Measure a box that is not yours.

Stop Calling It Staging If the Agent Never Leaves Localhost

Emery Chen — Fri, 18 Sep 2026 08:01:22 +0000

Your laptop is not a real staging environment. A fluent local chat still proves almost nothing. Stage the agent remotely, or admit you guessed.

The claim

Local agent runs hide the failures that matter. You skip DNS, cold starts, and queue delay. You also skip boring contract breaks.

A remote canary is the cheapest honest gate. It does not need a larger model. It needs a frozen tool allowlist and a replayable fixture.

If the agent never left localhost, you did not stage it. A green demo is still a draft. Merge gates need exit codes, not screenshots.

What you should measure

Do not score the generated prose. Score the tool surface instead. Reviewers cannot replay a vibe.

A useful canary answers four questions:

Did the agent call a tool you never allowed?
Did a required argument key disappear?
Did a call exceed your step budget?
Did the remote host fail before the model did?

Those are merge questions. They are not demo questions. Chat quality can wait until the contract holds.

Freeze the contract first

Put the allowlist in git. Treat it like a public API. Prompts drift. Files get reviewed.

{
  "version": "2026-09-18",
  "max_steps": 8,
  "allow": [
    {
      "name": "repo_read",
      "required": ["path"],
      "optional": ["start_line", "end_line"]
    },
    {
      "name": "repo_search",
      "required": ["query"],
      "optional": ["globs", "max_hits"]
    },
    {
      "name": "http_get",
      "required": ["url"],
      "optional": ["timeout_ms"]
    }
  ],
  "deny": ["shell_exec", "fs_write", "net_raw"]
}

Label this file as the contract. Do not bury it in a system prompt. If a tool is missing here, the agent must not call it.

Build a fixture, not a vibe

A fixture is a frozen input plus expected shape. It is not a transcript you liked last night.

{
  "id": "canary-pr-summary-01",
  "goal": "Summarize PR 1842 using repo_read only.",
  "context": {
    "pr": 1842,
    "paths": [
      "src/billing/invoice.py",
      "tests/billing/test_invoice.py"
    ]
  },
  "expect": {
    "allowed_tools": ["repo_read"],
    "forbidden_tools": ["http_get", "repo_search"],
    "max_steps": 4,
    "must_read": ["src/billing/invoice.py"]
  }
}

Write a handful of these. You do not need a hundred. Coverage beats theater. Delete any fixture that cannot fail.

Run the canary off the laptop

The runner below is a labeled example. Wire it to your own remote endpoint. Keep tokens in the environment, not in git.

#!/usr/bin/env python3
"""Labeled example: remote agent canary. Not a production SDK."""

from __future__ import annotations

import json
import os
import sys
import urllib.error
import urllib.request
from pathlib import Path

CONTRACT = json.loads(Path("tools.allowlist.json").read_text())
FIXTURE = json.loads(Path("fixtures/canary-pr-summary-01.json").read_text())
BASE = os.environ["CANARY_BASE_URL"].rstrip("/")
TOKEN = os.environ.get("CANARY_TOKEN", "")


def post(path: str, payload: dict) -> dict:
    data = json.dumps(payload).encode()
    req = urllib.request.Request(
        f"{BASE}{path}",
        data=data,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {TOKEN}",
        },
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=60) as resp:
            return json.loads(resp.read().decode())
    except urllib.error.URLError as exc:
        print(f"remote_error {exc}")
        sys.exit(2)


def violations(trace: dict) -> list[str]:
    issues: list[str] = []
    allowed = {item["name"] for item in CONTRACT["allow"]}
    denied = set(CONTRACT["deny"])
    steps = trace.get("tool_calls", [])
    if len(steps) > FIXTURE["expect"]["max_steps"]:
        issues.append("step_budget")
    for call in steps:
        name = call.get("name")
        args = call.get("args") or {}
        if name in denied or name not in allowed:
            issues.append(f"denied_or_unknown:{name}")
            continue
        spec = next(item for item in CONTRACT["allow"] if item["name"] == name)
        missing = [k for k in spec["required"] if k not in args]
        if missing:
            issues.append(f"missing_args:{name}:{','.join(missing)}")
        if name not in FIXTURE["expect"]["allowed_tools"]:
            issues.append(f"fixture_forbid:{name}")
    for path in FIXTURE["expect"]["must_read"]:
        reads = [
            c
            for c in steps
            if c.get("name") == "repo_read"
            and (c.get("args") or {}).get("path") == path
        ]
        if not reads:
            issues.append(f"unread:{path}")
    return issues


def main() -> None:
    trace = post("/v1/agent/run", {"fixture": FIXTURE, "contract": CONTRACT})
    issues = violations(trace)
    report = {
        "fixture": FIXTURE["id"],
        "contract": CONTRACT["version"],
        "steps": len(trace.get("tool_calls", [])),
        "issues": issues,
        "ok": not issues,
    }
    print(json.dumps(report, indent=2))
    sys.exit(0 if report["ok"] else 1)


if __name__ == "__main__":
    main()

Exit codes matter more than summaries.

0 means the contract held.
1 means the agent broke the contract.
2 means the remote host failed first.

That last code is the one local demos never show you. Treat host failure as a blocked merge, not a skip.

Put it in CI, not in a screenshot

# .github/workflows/agent-canary.yml
name: agent-canary
on:
  pull_request:
    paths:
      - "tools.allowlist.json"
      - "fixtures/**"
      - "agents/**"
jobs:
  canary:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: run remote canary
        env:
          CANARY_BASE_URL: ${{ secrets.CANARY_BASE_URL }}
          CANARY_TOKEN: ${{ secrets.CANARY_TOKEN }}
        run: python3 canary.py

You can also run it by hand:

export CANARY_BASE_URL="https://canary.example.internal"
export CANARY_TOKEN="***"
python3 canary.py
echo $?

If you cannot point that command at a host you do not own, you are still on localhost. Rename the ritual. It is not staging.

Decision table

Use this at review time. Do not invent extra scores.

Remote result	Tool contract	Merge action
Host unreachable	Unknown	Block. Fix the canary host.
Exit 2	Untested	Block. You learned nothing.
Exit 1, denied tool	Broken	Block. Tighten the allowlist or the agent.
Exit 1, missing args	Broken	Block. The schema drifted.
Exit 1, unread path	Weak	Block. The fixture was skipped.
Exit 0	Held	Allow human review to continue.

Notice the table never asks if the summary sounded good. Sound is not a gate. Contract shape is.

Why a free remote slot changes the argument

You do not need a paid cluster to learn this. You need a host that is not your laptop. Local RAM will not emulate queueing, shared CPU, or foreign DNS.

MonkeyCode offers free model access and a free server option. That pair is useful here as a remote canary target, not as a demo coupon.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the same contract file. Keep the same fixture. Point CANARY_BASE_URL at the free server, then watch which failures appear only off-laptop.

If that path cannot host your canary, do not pretend a local chat replaced it. Find another remote. The method does not depend on one vendor.

What this will not catch

This canary does not prove correctness. It proves the tool surface stayed inside a box. Green canaries still need humans.

It will miss:

Quiet logic errors inside an allowed tool
Poisoned files the agent was allowed to read
Cost blowups below the step cap
Reviewer fatigue after a green report

It also assumes you can freeze tools. If your agent is a raw shell, stop. An allowlist of shell_exec is not a contract. It is a shrug.

Who should not use this

Skip this workflow if you are:

Pairing on a throwaway spike
Editing copy with no tools
Running a model with no function calls
Unable to send fixtures off-laptop

In those cases a remote canary is overhead. Do not cargo-cult the YAML. A spike can stay local until tools exist.

Limitations you should write down

Remote does not mean representative. A free server may differ from production in queue time and payload limits. This article will not invent numbers for those gaps.

Treat the canary as a bias detector. Localhost is optimistic. Remote is less so. Neither is production.

Pin versions of the contract. Rotate fixtures when the product changes. Delete fixtures that always pass. A canary that cannot fail is decoration.

A short review script

Paste this into the pull request template:

## Agent canary
- [ ] `tools.allowlist.json` version bumped if tools changed
- [ ] Fixture IDs listed
- [ ] Remote canary exit code pasted
- [ ] No new deny-list hits
- [ ] No localhost-only screenshot as evidence

If a reviewer sees a chat screenshot, they should ask for the exit code. If they cannot get one, the patch is not staged. No exit code, no merge.

Close

You already know localhost lies. You still present it as staging. Stop.

Freeze the tool contract. Replay a fixture on a remote host. Merge only when the canary returns zero.

A fluent local agent is a draft. A remote canary is a gate. If you already have a free remote slot, point the runner at it and paste the JSON report.

Token Caps Are Tests. Treat Them That Way.

Emery Chen — Thu, 17 Sep 2026 06:06:38 +0000

You would not merge a job with no timeout. An agent with no token cap is the same risk.

A fluent local demo is not a bound on spend. It is theater that dies when the bill arrives.

The claim

Vibe coding is not the real engineering scandal here. Unbounded tool loops with no fuse are the scandal.

You let a model call tools until it feels done. Treat that loop as an unsigned check, not a prototype.

A cost fuse is the first honest test you can write. It fails closed and it refuses to grade prose. It only stops spend when the cap trips.

What you are pretending

You open a chat and the agent refactors a file. You screenshot the diff and call it a prototype.

You still shipped none of the controls below.

a hard ceiling on tokens for the run
a kill path that the host cannot ignore
a record of which machine actually ran
a fail code your CI already understands

Without those controls you demonstrated taste, not control. Taste does not survive contact with a retry storm.

Define the fuse before the prompt

A fuse is not a usage dashboard you glance at. A fuse is a gate that kills the process.

You set a budget and the runner enforces it. The process dies and the job goes red.

Start with three numbers and nothing fancier. Add more ceilings only after these three hold.

max_prompt_tokens caps the user payload size
max_completion_tokens caps what the model emits
max_tool_rounds caps how often tools may run

If any ceiling trips, you fail the job. You do not retry inside the same CI job. Retries hide leaks and train you to ignore red.

Why three numbers beat one dollar cap

A single dollar cap hides the failure mode. Tokens and tool rounds fail at different layers.

Prompt tokens catch a context dump before the call. Completion tokens catch a model that will not stop. Tool rounds catch a loop that forgot the user.

One money number arrives too late to help. Providers bill after the damage is already done. Your job needs a local veto before the HTTP call.

Decision table

Signal	Pass	Fail closed
Tokens under cap	continue	n/a
Tool rounds under cap	continue	n/a
Cap hit mid-tool	abort and dump a trace	yes
Host is not disposable	do not score the run	yes
Budget missing from config	refuse to start	yes

Read the last row twice before you merge. A missing budget must be a start failure. Generous defaults are how surprise bills get born.

Pick numbers like timeouts, not like hopes

Copy timeout practice from your HTTP clients. You already pick 5s without a research paper.

Start tight, then loosen with evidence from trips. Do not start wide and promise to optimize later.

Use this tight first pass until trips teach you more.

{
  "max_prompt_tokens": 2000,
  "max_completion_tokens": 1500,
  "max_tool_rounds": 4
}

Four tool rounds is already a long leash. Most repair jobs should die before round four. If your task needs twenty rounds, split the task.

Write the reason for each number in the PR. "Seems fine" is not a reason you can audit.

Artifact: a budget gate you can run

Treat the next file as a copyable proposal. It is not a production SDK or a vendor wrapper. It is a fuse you can trip on purpose.

# budget_gate.py — proposal, unexecuted example
from __future__ import annotations

import json
import os
import sys
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path


@dataclass(frozen=True)
class Budget:
    max_prompt_tokens: int
    max_completion_tokens: int
    max_tool_rounds: int


def load_budget(path: Path) -> Budget:
    raw = json.loads(path.read_text())
    required = (
        "max_prompt_tokens",
        "max_completion_tokens",
        "max_tool_rounds",
    )
    missing = [k for k in required if k not in raw]
    if missing:
        raise SystemExit(f"budget missing keys: {missing}")
    if any(int(raw[k]) <= 0 for k in required):
        raise SystemExit("budget values must be positive")
    return Budget(
        max_prompt_tokens=int(raw["max_prompt_tokens"]),
        max_completion_tokens=int(raw["max_completion_tokens"]),
        max_tool_rounds=int(raw["max_tool_rounds"]),
    )


class Fuse:
    def __init__(self, budget: Budget) -> None:
        self.budget = budget
        self.prompt_tokens = 0
        self.completion_tokens = 0
        self.tool_rounds = 0

    def charge_prompt(self, n: int) -> None:
        self.prompt_tokens += n
        if self.prompt_tokens > self.budget.max_prompt_tokens:
            self._trip("prompt")

    def charge_completion(self, n: int) -> None:
        self.completion_tokens += n
        if self.completion_tokens > self.budget.max_completion_tokens:
            self._trip("completion")

    def charge_tool_round(self) -> None:
        self.tool_rounds += 1
        if self.tool_rounds > self.budget.max_tool_rounds:
            self._trip("tool_rounds")

    def snapshot(self) -> dict:
        return {
            "prompt_tokens": self.prompt_tokens,
            "completion_tokens": self.completion_tokens,
            "tool_rounds": self.tool_rounds,
            "hostname": os.uname().nodename,
            "utc": datetime.now(timezone.utc).isoformat(),
        }

    def _trip(self, reason: str) -> None:
        dump = {"tripped": reason, **self.snapshot()}
        Path("fuse-trip.json").write_text(json.dumps(dump, indent=2))
        print(f"FUSE TRIPPED: {reason}", file=sys.stderr)
        raise SystemExit(2)


def estimate_tokens(text: str) -> int:
    # Crude stand-in. Replace with your tokenizer.
    return max(1, len(text) // 4)


def main() -> None:
    budget = load_budget(Path(os.environ.get("BUDGET_FILE", "budget.json")))
    fuse = Fuse(budget)
    prompt = Path("prompt.txt").read_text()
    fuse.charge_prompt(estimate_tokens(prompt))
    # Your agent loop would charge completion and tools here.
    print("budget accepted; runner may start")
    print(json.dumps(fuse.snapshot()))


if __name__ == "__main__":
    main()

Pair the gate with a tiny committed config file. Keep budget.json next to the runner, not in chat history.

Run the gate like a unit test, not a chat.

printf 'summarize src/app.ts in five bullets\n' > prompt.txt
export BUDGET_FILE=./budget.json
python budget_gate.py
echo $?   # 0 means the fuse armed

Trip the fuse on purpose in the same session. A fuse you never trip is only décor.

python - <<'PY'
from pathlib import Path
Path("prompt.txt").write_text("word " * 20000)
PY
python budget_gate.py; echo exit:$?
test -f fuse-trip.json && cat fuse-trip.json

You want exit code 2 and a trip file. Silence from the runner means the gate never loaded.

A second test: tool rounds

Token caps miss a cheap infinite tool loop. Each tool result can be small and still endless.

Extend the runner with an explicit round counter. Fail before the fifth tool call in this example.

# proposal: drop into your loop, unexecuted
for _ in range(99):
    fuse.charge_tool_round()
    # call_tool() would go here

If that loop can run to 99, your fuse is dead. Watch the exit code, not the model apology.

Why the host must not be your laptop

A fuse that lives only in your shell is optional. Optional gates get skipped the moment a demo is due.

Copy the same files onto a disposable server. Keep the budget file and keep the exit codes. If the fuse cannot kill that process, stop scoring.

Your laptop has aliases, extra env, and cached keys. Those extras make a red fuse look green. A clean server removes the accidental safety net.

This is where free model access earns its keep. You rehearse the kill path without a paid key. You still need a machine you can wipe afterward.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open source project with free model access. It also offers a free server option for rehearsal. Use both as a ring you can delete, not an SLA.

Keep the product out of the work's real title. The work is the fuse, not the vendor.

Wire the fuse into the job

A useful CI step should look almost boring. Clever gates get skipped when people are late. Boring gates still run when people are late.

# .github/workflows/agent-fuse.yml — proposal
name: agent-fuse
on:
  pull_request:
    paths:
      - "agent/**"
      - "budget.json"
      - "budget_gate.py"
      - "prompt.txt"
jobs:
  fuse:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Arm budget gate
        run: python budget_gate.py
      - name: Refuse a trip on the happy path
        run: test ! -f fuse-trip.json

Add a second job that expects a trip. Green builds that never see red teach nothing.

# expected-fail.sh — proposal
set -euo pipefail
python -c 'import json,pathlib; p=pathlib.Path("budget.json"); b=json.loads(p.read_text()); b["max_prompt_tokens"]=1; p.write_text(json.dumps(b))'
if python budget_gate.py; then
  echo "fuse failed to trip" >&2
  exit 1
fi
test -f fuse-trip.json

You now have two outcomes worth merging later.

the happy path stays under the committed budget
the angry path dies with exit code 2

Everything else is still just chat with extra steps.

Log the trip like an incident

When the fuse trips, keep a small packet. Do not keep a novel. Keep the indictment.

Write these fields into fuse-trip.json every time:

reason: prompt, completion, or tool_rounds
the three counters at death
git sha of the budget file
hostname of the runner
utc timestamp

Hostname matters because laptop scores do not transfer. If hostname looks like a personal laptop, discard the run. Personal hostnames are a smell, not a platform.

What this does not measure

A token cap is not a quality score at all. It will not catch a wrong refactor in review. It will not catch a secret that leaked into logs.

It will not catch a tool the model should never see. It measures boundedness and almost nothing else.

Do not park a prose scorer next to this fuse. You will start negotiating with the generated essay. Keep the fuse stupid so it stays enforceable.

Who should not use this

Skip this approach when any line below is true.

you ship no agent loop at all
your tools can mutate prod with one call
you need latency SLOs, not rehearsal
you cannot store traces even on a throwaway host
you treat free model access as production capacity

If tools can mutate production, isolate them first. A fuse on a live admin API is theater again. Fix the blast radius, then add the token cap.

Limitations you must name

Token estimates by character count will drift under load. Replace estimate_tokens with your real tokenizer soon. Provider usage fields also lie when streams cut off.

Persist the raw usage object beside fuse-trip.json. Audit the object before you trust any dashboard.

Free model access can be slow, queued, or withdrawn. A free server is not your VPC or your region. Rehearse on the free path. Certify on your own.

Do not advertise a fuse you cannot trip in CI. An untested kill switch is only documentation.

Queueing on a free path can starve the fuse test. A hung job is not a passed budget. Put a wall-clock timeout around the whole job.

Thirty minutes of silence is not a passed bound. Kill the runner, then read the trip file.

The opinion, restated

You are not behind on models this quarter. You are behind on bounds you can enforce.

The argument about whether AI codes better is noise. Better than whom, on which host, under which cap?

If you cannot name the budget, you cannot name the system. If you cannot kill the process, you do not own it.

Put the fuse in the repo before the next prompt. Run it on a box you can burn without regret. Then you can talk about agents without pretending.

You can rehearse the kill path on a throwaway ring. MonkeyCode's free models and free server are enough for that.

Stop Scoring Prose. Start Diffing Tool Shapes.

Emery Chen — Wed, 16 Sep 2026 04:15:33 +0000

Fluent agent answers are a weak merge signal today.
You should gate on tool-call shapes, not prose.
A schema-valid trace beats any polished summary.

Take a side

You are not testing an agent with chat transcripts.
You are testing a narrator that can call tools.
Engineering starts when those calls become contracts.

Vibe coding produces text that looks finished fast.
Calling that text a verified change remains the failure.
The model can lie in English without touching state.

Hot threads still argue about models beating developers.
That debate skips the only artifact you can freeze.
You cannot freeze charm, but you can freeze JSON shapes.

What a shape actually is

A shape is the JSON skeleton of each tool call.
It includes name, argument keys, types, and order.
It ignores free-text values that change every run.

You keep the skeleton and discard the essay around it.
You compare skeletons across runs, models, and branches.
A mismatch means the harness drifted, not the copy.

Store shapes as receipts beside the pull request.
Treat a missing receipt as a build failure immediately.
Do not accept screenshots of the chat as substitutes.

Why answer scores keep lying

Answer scores reward fluency and a confident tone.
Agents game both without doing the requested work.
A skipped tool still yields a soothing paragraph.

Unit tests written after the fact hide that skip.
Reviewers read the summary and then approve the skip.
The production path never sees the missing tool call.

Loop-engineering posts still warn about unbounded retries today.
Retries also mutate shapes while scores stay green.
Shape-diff catches that mutation while score dashboards miss it.

The merge rule you should adopt

This section is a proposal, not a measured case study.

Every agent run must emit a receipt file on disk.
Every receipt must validate against a frozen JSON schema.
CI must shape-diff that receipt against a golden file.
Prose in the PR body stays commentary, never evidence.

If step three fails, you do not merge the change.
You fix the harness or you update the frozen schema.
You never accept nicer wording as a substitute patch.

Pin the schema next to the tool definitions in git.
Require a human review when the schema itself changes.
That review is the real design discussion you wanted.

Freeze a receipt schema

The example below is illustrative and unexecuted here.
Copy it, then tighten types for your real tools.
Do not ship the sample tools into production paths.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "AgentReceipt",
  "type": "object",
  "required": ["schema_version", "run_id", "calls"],
  "properties": {
    "schema_version": { "const": "1" },
    "run_id": { "type": "string", "minLength": 8 },
    "calls": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "required": ["seq", "tool", "arg_keys", "arg_types"],
        "properties": {
          "seq": { "type": "integer", "minimum": 0 },
          "tool": { "type": "string", "pattern": "^[a-z][a-z0-9_]*$" },
          "arg_keys": {
            "type": "array",
            "items": { "type": "string" }
          },
          "arg_types": {
            "type": "array",
            "items": {
              "enum": ["string", "number", "boolean", "object", "array", "null"]
            }
          }
        },
        "additionalProperties": false
      }
    }
  },
  "additionalProperties": false
}

Notice the schema refuses extra keys on each call.
That refusal is the entire product of this gate.
Extra keys mean the harness drifted under your feet.
Pretty summaries cannot sneak a new argument through.

Strip values before you diff

Raw argument values poison the golden file too fast.
Timestamps, ids, and prose will never stay stable.
You strip values and keep names plus types only.

# Example only. Unexecuted in this article.
from __future__ import annotations

import json
from pathlib import Path
from typing import Any

TYPE_MAP = {
    str: "string",
    int: "number",
    float: "number",
    bool: "boolean",
    type(None): "null",
}


def json_type(value: Any) -> str:
    if isinstance(value, list):
        return "array"
    if isinstance(value, dict):
        return "object"
    return TYPE_MAP[type(value)]


def shape_of(call: dict[str, Any]) -> dict[str, Any]:
    args = call.get("arguments") or {}
    keys = sorted(args.keys())
    return {
        "seq": call["seq"],
        "tool": call["tool"],
        "arg_keys": keys,
        "arg_types": [json_type(args[k]) for k in keys],
    }


def receipt_shape(raw: dict[str, Any]) -> dict[str, Any]:
    return {
        "schema_version": "1",
        "run_id": raw["run_id"],
        "calls": [shape_of(c) for c in raw["calls"]],
    }


def dump_shape(src: Path, dest: Path) -> None:
    raw = json.loads(src.read_text())
    dest.write_text(json.dumps(receipt_shape(raw), indent=2) + "\n")

Keep run_id for humans and drop it from equality.
Compare the calls array only as the contract surface.
Sequence plus tool plus typed keys must stay identical.

Fail the build on a shape mismatch

Golden files belong in the same pull request always.
Update them in a dedicated schema-change commit only.
Never bury a shape change inside a feature commit.

# Example only. Unexecuted in this article.
import json
import sys
from pathlib import Path

from jsonschema import Draft202012Validator


def load(path: Path) -> dict:
    return json.loads(path.read_text())


def validate(schema_path: Path, receipt_path: Path) -> None:
    schema = load(schema_path)
    data = load(receipt_path)
    Draft202012Validator(schema).validate(data)
    print("SCHEMA_OK")


def diff_calls(golden_path: Path, actual_path: Path) -> int:
    left = load(golden_path)["calls"]
    right = load(actual_path)["calls"]
    if left != right:
        print("SHAPE_DIFF")
        print("golden:", json.dumps(left, indent=2))
        print("actual:", json.dumps(right, indent=2))
        return 1
    print("SHAPE_OK")
    return 0


if __name__ == "__main__":
    validate(Path("tools/receipt.schema.json"), Path("receipts/actual.shape.json"))
    raise SystemExit(
        diff_calls(Path("receipts/golden.shape.json"), Path("receipts/actual.shape.json"))
    )

Wire it as a boring command, not a dashboard widget.

python3 tools/shape_of.py receipts/actual.json receipts/actual.shape.json
python3 tools/shape_diff.py
test $? -eq 0

A non-zero exit code is the entire merge policy.
Do not parse the model essay after this step.
Do not negotiate with the log output either.

Fixture the tools, not the novel

The fixture below is a stub, not a live agent.
Replace emit_receipt() with your harness hook later.
Keep the tool names identical to production definitions.

# Example only. Unexecuted in this article.
import json
import uuid


def emit_receipt() -> dict:
    return {
        "schema_version": "1",
        "run_id": uuid.uuid4().hex,
        "calls": [
            {
                "seq": 0,
                "tool": "fetch_issue",
                "arguments": {"id": "ISSUE-1"},
            },
            {
                "seq": 1,
                "tool": "post_comment",
                "arguments": {"issue_id": "ISSUE-1", "body": "done"},
            },
        ],
    }


if __name__ == "__main__":
    print(json.dumps(emit_receipt(), indent=2))

Run that fixture until the golden file stops moving.
Then point the same hook at a hosted model runner.
The shape must not change when the host changes.

Run the worker off your laptop

Your laptop shares cache, secrets, and lucky timing.
A dedicated job host removes that shared luck.
Free model access belongs in that job, not demos.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option.
Point the shape-diff worker at that server when hosts are scarce.
Keep product features on your normal models and paid paths.

The free model only has to emit schema-valid receipts.
It does not have to win a style contest against anyone.
If the shape breaks, the model is the wrong job runner.

A minimal job sketch

This YAML is a template and not a vendor file.

# Example only. Unexecuted in this article.
name: shape-diff
on:
  pull_request:
jobs:
  receipt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: emit receipt
        run: python3 tools/run_agent_fixture.py > receipts/actual.json
      - name: strip values
        run: python3 tools/shape_of.py receipts/actual.json receipts/actual.shape.json
      - name: validate and diff
        run: python3 tools/shape_diff.py

The fixture script should use a frozen prompt file.
It should use a frozen tool list and frozen fixtures.
It should not browse the public internet for truth.

Decision table

Use this table before you add another numeric score.

Situation	Trust the prose?	Trust the shape?	Action
You changed copy only	No	Yes, if calls match	Merge the copy
You added a tool argument	No	No, until schema updates	Review the schema
Model skipped a tool	Yes, it will sound fine	No	Block the merge
Model reordered two calls	Maybe	No	Block, then inspect
Values changed, keys did not	No	Yes	Merge after value tests
Schema file changed with no review	No	No	Reject the pull request

Print the table in the team handbook this week.
Do not keep it in a slide deck nobody opens.
The table is the policy and the script is the lock.

What this will not catch

Shape-diff ignores wrong values with the right types.
A string can still be a malicious URL or empty path.
You still need value tests for auth, money, and deletion.

It also ignores missing side effects outside listed tools.
A tool can return 200 and still write the wrong row.
Pair shape-diff with ordinary integration tests after it.

Free model access will not prove production quality here.
It only proves the harness still demands the same skeleton.
Do not publish those runs as public model leaderboards.

Who should not use this

Skip this if your agent has no structured tools.
A single-shot editor chat has no receipt to freeze.
Do not invent fake tools just to satisfy the linter.

Skip this if you cannot pin prompts and tool lists.
A daily prompt tweak without review breaks golden files.
You will spend the week blessing noisy receipt diffs.

Skip this during short exploratory research spikes too.
Exploration needs cheap failure, not a frozen skeleton.
Promote the spike into this gate only after tools stabilize.

Put the schema in CODEOWNERS

Prompt files and tool schemas are production code now.
They deserve owners, review, and a boring textual diff.
Leave the essay to the author and gate the skeleton.

# Example CODEOWNERS fragment
/prompts/ @harness-owners
/tools/*.schema.json @harness-owners
/receipts/golden.shape.json @harness-owners

When the golden file changes, named owners must speak.
Silence is not consent because the shape is the API.
You would not ship an unsigned OpenAPI change today.
Stop doing the same silent drift with agent tools.

Closing position

You do not lack a smarter model for this gate.
You lack a frozen skeleton for every tool call.
Until that skeleton exists, you are still vibing.

Keep the worker dull, local to git, and strict.
If you already emit receipts, run the shape-diff worker on a free server.
Keep the frozen schema in version control beside the tools.

Free Models Belong in Your CI, Not Your Demo

Emery Chen — Mon, 14 Sep 2026 21:48:01 +0000

Free Models Belong in Your CI, Not Your Demo

Most teams use cheap models backwards. They wire them into a chat demo and shrug at the quality. I think the cheapest model you can call should gate every commit instead.

It should not be the thing you show customers. It should be the thing that tells you something moved.

Here is the argument, plus a runner you can copy today.

The real bottleneck is not model quality

You will never learn agent engineering from ten careful calls. You learn it from thousands of disposable runs.

Paid inference punishes that habit. Every run costs money, so you write fewer tests. You trim the suite and stop probing edge cases.

That is the failure mode. Your agent gets fragile because your test budget got tight.

Cheap inference removes the excuse. Not because cheap models are smart, but because they let you run the boring suite every single time.

The front page keeps arguing about whether agents are real. That debate misses the operational question. The question is which tier should run which check.

Treat the cheap model as a canary tier

Three tiers, three jobs

Canary tier: the cheapest model you can reach. Runs the full regression suite on every push.
Workhorse tier: a mid-cost model. Runs only cases the canary failed or never solved.
Review tier: an expensive model. Runs a small, hand-picked adversarial set and anything you plan to merge.

The canary does not judge product quality. It detects change cheaply.

A case flipping from pass to fail is a signal, even on a weak model. A stable pass on a weak model is weak evidence, so label it that way.

A minimal escalation runner

The code below is unexecuted here. Treat it as a sketch to adapt, not a benchmark.

# tier_runner.py — sketch, not a benchmark
import json, os, time, urllib.request, uuid

ENDPOINT = os.environ["MC_BASE_URL"]      # operator-supplied
API_KEY  = os.environ["MC_API_KEY"]
CANARY   = os.environ["MC_MODEL_CANARY"]
WORKER   = os.environ["MC_MODEL_WORKER"]
LOG      = "runs.jsonl"

def call(model, prompt, timeout=60):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(ENDPOINT, data=body, headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {API_KEY}",
    })
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=timeout) as r:
        payload = json.loads(r.read())
    return payload, round(time.time() - t0, 2)

The loop stays boring on purpose.

def run_case(case, model):
    row = {"run_id": str(uuid.uuid4()), "case": case["id"], "model": model}
    try:
        payload, secs = call(model, case["prompt"])
        row.update(ok=check(payload, case),
                   seconds=secs,
                   usage=payload.get("usage", {}))
    except Exception as exc:
        row.update(ok=False, error=type(exc).__name__)
    with open(LOG, "a") as fh:
        fh.write(json.dumps(row) + "\n")
    return row["ok"]

def main(cases):
    escalated = [c for c in cases if not run_case(c, CANARY)]
    for case in escalated:
        run_case(case, WORKER)
    print(f"{len(escalated)}/{len(cases)} escalated")

Two rules that keep this honest

Never compare pass rates across tiers and call it a quality score. You are measuring change, not intelligence.
Never let the canary gate a release. It gates a re-run.

Run it off your laptop

This is where server stability matters more than model choice. A canary tier only helps if it runs in the same place every time.

Three properties you want:

Same image, same env vars, same seed data on every run.
Logs written to a volume you can diff between runs.
A wall-clock budget, so one stuck call cannot eat the night.

Commit the runner and the log schema. Then git diff on runs.jsonl tells you when behavior moved.

I run this pattern on MonkeyCode's free server option, using its free model access as the canary tier. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The project states a free token allowance for that model access. Check the current project page for the exact figure, because quotas and model lists change. Do not paste a number into your README and forget it.

Nothing above depends on that product. Any box with a pinned image works. The hard requirement is that the box is not your daily driver.

What the canary tier actually catches

Cheap models are decent detectors for a specific class of breakage.

Prompt or template drift after a refactor.
Tool schema changes that break argument parsing.
Output regressions, like dropped JSON fields.
Timeouts and retry storms under load.
Latency shifts inside your own wrapper code.

It also produces noise. Weak models fail tasks for reasons unrelated to your change. Budget for that before you trust the log.

Decision table

Question	Canary tier	Workhorse tier	Review tier
Runs on every push	Yes	Only after canary failure	No
Gates a merge	No	No	Yes
Needs a pinned image	Yes	Yes	Yes
Logs diffed across runs	Yes	Yes	Yes
Good for cost tracking	Yes	Yes	Yes
Good for quality claims	No	Rarely	Sometimes

Read the last row twice. A pass on a cheap model is not evidence of correctness. It is evidence that nothing obvious broke.

Who should skip this

Do not use this pattern if any of these are true for you.

You need reproducible quality claims for a customer or an auditor.
Your task needs strict tool-calling fidelity or long multi-step planning.
You cannot pin a model version, so the canary itself changes under you.
Your data cannot leave your own infrastructure.

For those cases, run a fixed, versioned model on hardware you control. A free tier is the wrong tool for an audit trail, and pretending otherwise will burn you in review.

Start with one suite, one box

Pick your ten most annoying agent bugs. Turn each one into a case file. Run them on the cheapest model you can call, on a machine that is not your laptop.

If the suite is cheap, you will actually run it. If you run it, you will see the break on the commit that caused it. That is the whole argument.

If you would rather not provision anything yet, MonkeyCode's free model access and free server option are enough to stand up a canary tier and run the experiment.

If Your Eval Shares Your Laptop, Discard the Score

Emery Chen — Sun, 13 Sep 2026 20:28:47 +0000

You cannot rank an agent from a dirty machine.
Local shells leak secrets, caches, and leftover files.
Treat any laptop score as contaminated until you isolate it.

Laptop evals are not measurements

A real measurement needs a known starting state.
Your laptop is a graveyard of prior experiments.
Your laptop cannot be a starting state.
It is only folklore from prior runs.

You export an API key for just this run.
The next run inherits that key without asking.
Then you praise a model for finding credentials.

You install a CLI during last week's spike.
Today's agent shells out and then discovers it.
You call that autonomy, but it is residue.

Why "same prompt" still lies

People defend laptop evals with a ritual phrase.
They say the prompt file never changed once.
That claim does not rescue a dirty host.

The prompt is one input among many inputs.
The shell, the PATH, and the home directory vote.
Cached wheels and editor swap files vote too.

You would not benchmark a database on leftover WAL.
You would not score a compiler against dirty objects.
Stop giving language models a special exemption here.

What actually contaminates the score

Walk your last eval through this list.
If any item is true, throw the score away.

Environment leaks

PATH still contains last week's tool binaries
~/.config holds tokens from a different vendor
leftover OPENAI_* or ANTHROPIC_* variables
Docker sockets that expose your real containers
SSH agents that can reach production hosts

An agent that just works on your PATH is cheating.
It did not earn the tool and only inherited it.

Filesystem memory

./tmp from a previous loop still exists
node_modules or .venv already solved imports
git hooks rewrite commits behind the model's back
editor swap files contain the target answer
shell history teaches yesterday's exact commands

Green output after a dirty tree is not success.
It is only playback of yesterday's work.

Hidden retries

You rerun the same prompt after a failure.
The model sees a warmer cache or a fixed file.
You log one attempt while the machine lived three.

Retries belong in the trace, not in your head.
If you cannot show them, you did not measure them.

The opinion, without hedging

You should stop ranking models on developer laptops.
The ranking is a story about your machine.
It is not a story about the model.

Isolation is not a luxury for careful teams.
Isolation is the actual fixture for the score.
If you cannot throw the machine away, you cannot trust it.

A public leaderboard built on laptops is theater.
Readers cannot see your PATH or your extra CLIs.
They only see a number that you already liked.

Pin the protocol, not the vibes

Before you open a chat pane, write four pins.

Prompt text, stored as a file, hashed
Tool allowlist, stored as a file, hashed
Budget: max steps, max output bytes
Success predicate: a command that exits 0 or 1

If any pin is missing, you are demoing, not evaluating.
Demos can live on laptops, but evals cannot.

Hash the pins before the first model call.
Print those hashes in the trace header every time.
If a hash moves, you started a different experiment.

Artifact: a throwaway isolation harness

The harness below is a starter, not a product.
Run it on a machine you can destroy.
Label every result untrusted until that is true.

#!/usr/bin/env python3
"""Isolation harness for agent evals. Starter you can run."""
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
import time
from pathlib import Path

ROOT = Path(os.environ.get("EVAL_ROOT", "/tmp/eval-root"))
TRACE = ROOT / "trace.jsonl"
MAX_STEPS = int(os.environ.get("EVAL_MAX_STEPS", "8"))
MAX_BYTES = int(os.environ.get("EVAL_MAX_BYTES", "32768"))


def die(msg: str, code: int = 2) -> None:
    print(msg, file=sys.stderr)
    raise SystemExit(code)


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256(path.read_bytes()).hexdigest()
    return digest[:16]


def require_clean_machine() -> None:
    forbidden = [
        Path.home() / ".ssh",
        Path.home() / ".config",
        Path("/var/run/docker.sock"),
    ]
    for path in forbidden:
        if path.exists():
            die(f"contaminated path present: {path}")

    extra_keys = [
        key for key in os.environ
        if key.endswith("_API_KEY") or key.endswith("_TOKEN")
    ]
    if extra_keys:
        die(f"secret-shaped env vars: {extra_keys}")

    if os.geteuid() == 0 and Path("/home").exists():
        # Root on a throwaway box is fine.
        # Root on a laptop with user homes is not.
        homes = [p for p in Path("/home").iterdir() if p.is_dir()]
        if homes:
            die(f"user homes visible from eval host: {homes}")


def pin_inputs() -> dict:
    prompt = Path("prompt.txt")
    tools = Path("tools.json")
    predicate = Path("success.sh")
    for path in (prompt, tools, predicate):
        if not path.is_file():
            die(f"missing pin file: {path}")
    return {
        "prompt_sha": sha256_file(prompt),
        "tools_sha": sha256_file(tools),
        "predicate_sha": sha256_file(predicate),
        "max_steps": MAX_STEPS,
        "max_bytes": MAX_BYTES,
    }


def record(event: dict) -> None:
    ROOT.mkdir(parents=True, exist_ok=True)
    event["ts"] = time.time()
    with TRACE.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(event) + "\n")


def run_step(step: int, cmd: list[str]) -> dict:
    started = time.time()
    proc = subprocess.run(
        cmd,
        cwd=ROOT,
        capture_output=True,
        text=True,
        timeout=30,
    )
    event = {
        "step": step,
        "cmd": cmd,
        "exit": proc.returncode,
        "stdout": (proc.stdout or "")[:MAX_BYTES],
        "stderr": (proc.stderr or "")[:MAX_BYTES],
        "elapsed_s": round(time.time() - started, 3),
    }
    record(event)
    return event


def main() -> None:
    require_clean_machine()
    pins = pin_inputs()
    record({"type": "start", "pins": pins, "cwd": str(ROOT)})

    # Replace ["true"] with your agent runner.
    # Keep the budget gate outside the model.
    for step in range(1, MAX_STEPS + 1):
        event = run_step(step, ["true"])
        if event["exit"] != 0:
            record({"type": "abort", "reason": "tool_failed", "step": step})
            raise SystemExit(1)

    pred = subprocess.run(["bash", "success.sh"], cwd=ROOT)
    record({"type": "finish", "success": pred.returncode == 0})
    raise SystemExit(0 if pred.returncode == 0 else 1)


if __name__ == "__main__":
    main()

Save it as isolate_eval.py beside the pin files.
The agent never chooses the budget. You do.
That split is the whole point of the fixture.

Companion files

tools.json should be an allowlist, not a wish list.

{
  "allow": ["python3", "rg", "git"],
  "deny": ["ssh", "docker", "curl", "sudo"]
}

success.sh must be boring and binary.

#!/usr/bin/env bash
set -euo pipefail
test -f /tmp/eval-root/REPORT.md
rg -q "STATUS: PASS" /tmp/eval-root/REPORT.md

If the predicate needs a human, it is not a predicate.
It is a review, so log it as a review.

prompt.txt should name the task and the stop rule.
Do not hide extra hints in shell aliases nearby.
If the hint matters, put the hint in the file.

How you actually run it

# On a throwaway host, not your laptop.
install -d /tmp/eval-root
export EVAL_ROOT=/tmp/eval-root
export EVAL_MAX_STEPS=8
export EVAL_MAX_BYTES=32768
python3 isolate_eval.py; echo exit:$?
sha256sum prompt.txt tools.json success.sh
wc -l /tmp/eval-root/trace.jsonl

Copy the traces off the box before you wipe it.
Then destroy the box so residue cannot accumulate.
A kept eval machine becomes a laptop with extra steps.

Decision table: keep the score or burn it

Condition	Verdict	Why
Home directory mounted into the eval	Burn	Hidden files coach the agent
Any `*_API_KEY` in the environment	Burn	Tool use is no longer earned
Prompt edited between retries	Burn	You measured two systems
Budget enforced outside the model	Keep	The loop cannot flatter itself
Success is a command, not a vibe	Keep	Humans drift; scripts do not
Trace is complete and hashed	Keep	Replay is possible later
Host still exists after the run	Burn	Residue will leak into the next score

Print this table next to your CI job.
If a cell says burn, do not publish a leaderboard.

A CI sketch that refuses dirty runners

Laptop habits sneak into CI through shared runners.
Treat the runner like the laptop until proven empty.
Fail the job when the isolation checks fail.

# proposal: isolation gate, not a full agent platform
name: isolated-eval
on: [push]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Refuse a dirty environment
        run: |
          python3 isolate_eval.py
      - name: Upload traces only
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-trace
          path: /tmp/eval-root/trace.jsonl

Do not install your favorite CLIs "to help the agent".
That help is contamination with a nicer name.
If the task needs a tool, pin it in tools.json.

Common objections, answered bluntly

"I know what is on my machine."
You do not, not after six months of tooling.
Memory is not an inventory system.

"Isolation slows iteration."
Good. Ranking should be slower than chatting.
Speed without pins is just a better demo.

"The model should see a realistic developer box."
Then say you measured a developer box, not a model.
Do not mix those claims in the same sentence.

"We already log prompts."
Prompts without host state are incomplete evidence.
Log the hashes, the allowlist, and the budget too.

Where a free remote box fits

You need a machine that starts empty and dies full.
A local VM helps. A remote throwaway host helps more.
The point is disposal, not brand.

MonkeyCode offers free model access and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Use those only as the disposable fixture for this harness.
Do not treat them as a production cluster or a quota promise.

Wire the harness to whatever model endpoint you already trust.
Keep the allowlist and the budget on your side of the cable.
The model proposes actions. Your host accepts or refuses.

If you need a box you can wipe after the trace, that free server option is one place to park the harness.

Limitations you should not skip

This harness does not prove model quality.
It only proves the score was not born on a dirty laptop.
That is a lower bound, not a medal.

It does not sandbox syscalls or kernel objects.
A determined agent can still wreck the throwaway host.
That is acceptable if the host is truly disposable.

It does not pin model weights or provider routing.
If the endpoint drifts, your hashes will not save you.
Log provider headers when the API gives them to you.

Network egress is still a hole in many setups.
Deny curl and ssh in the allowlist, then verify.
An allowlist you never test is documentation.

Shared CI runners can still be dirty in quieter ways.
Cached actions, leftover workspaces, and privileged sockets remain.
Read the runner's filesystem before you trust the first score.

Who should not use this approach

You are pairing, not measuring. Use the laptop.
You lack permission to run untrusted code on a host.
Your agent never shells out. A unit test is enough.
You need guaranteed uptime, GPUs, or a named SLA.
You cannot write a binary success predicate yet.

If you are in that last group, stop ranking models.
Write the predicate first. Then isolate. Then rank.

What you should do on Monday

Pick one agent task you already brag about.
Write prompt.txt, tools.json, and success.sh.
Run the harness on a machine you can delete.

If the score collapses, believe the collapse.
Your laptop was the missing teammate all along.
Ship the isolation fixture before you ship the leaderboard.

If Your Agent Cannot Fail Cheap, Do Not Ship

Emery Chen — Sat, 12 Sep 2026 18:29:32 +0000

An unbounded agent is not a better programmer than you. It is a retry engine with a polite tone. You should refuse to ship any loop that cannot fail cheap.

The public scoreboard is lying

Public threads keep declaring models better than most developers. That claim measures draft speed, not ownership cost. You do not ship drafts into a shared branch.

Faster diffs look like competence during a recorded demo. They hide tool retries, schema thrash, and duplicated calls. You pay for every polite recovery the model invents.

You should score the abort path before the feature path. A loop that cannot stop is not intelligent work. It is a budget leak with syntax highlighting attached.

Cheap failure is a merge rule

Cheap failure means three concrete checks you can automate. The loop names a spend cap before the first tool. The loop dies when that named cap is actually hit.

The log must show the reason for that death. Missing reasons make the next incident undebuggable.

You already demand tests for parsers and database migrations. Demand the same discipline for agent control flow today. A green demo is not a control-flow test at all.

If the agent cannot fail cheap, do not merge it. That sentence is the whole opinion of this piece. Everything below is just an envelope you can copy.

Write the envelope before any prompt

Do not start the week with another prompt rewrite. Start with a budget contract the orchestrator must honor. Keep that contract in the repo beside agent config.

{
  "max_rounds": 4,
  "max_tool_calls": 6,
  "max_input_tokens": 8000,
  "max_output_tokens": 2000,
  "on_breach": "abort",
  "allowed_tools": ["repo.search", "repo.read"],
  "forbidden_tools": ["shell.exec", "net.fetch"]
}

That JSON file is not documentation for future readers. It is a gate your runner must load first.

Treat the snippet as a proposed local contract only. Tune the numbers for your repository and your tools. Do not copy these caps as universal production truth.

What each field actually buys you

max_rounds kills conversational thrash before the bill compounds.
max_tool_calls stops search loops that never actually converge.
max_input_tokens blocks context stuffing copied from earlier failures.
max_output_tokens caps rambling patches and unearned certainty.
on_breach: abort refuses one more try as a product feature.
allowed_tools keeps the model inside a named surface.
forbidden_tools is the real boundary, not the system prompt.

You put this file in version control on purpose. Reviewers then argue numbers instead of demo vibes.

The runner is the test, not the model

You do not trust the model to honor the envelope. You enforce the envelope inside your own process. The model is a guest, and guests do not hold keys.

Here is a small Python gate you can drop beside the agent. Label it as example code, not a vendor SDK.

# example: budget_gate.py — proposed local gate
from dataclasses import dataclass

@dataclass
class Envelope:
    max_rounds: int
    max_tool_calls: int
    max_input_tokens: int
    max_output_tokens: int
    allowed_tools: set
    forbidden_tools: set

@dataclass
class Spend:
    rounds: int = 0
    tool_calls: int = 0
    input_tokens: int = 0
    output_tokens: int = 0

class BudgetBreach(Exception):
    pass

def assert_tool_legal(name: str, env: Envelope) -> None:
    if name in env.forbidden_tools:
        raise BudgetBreach(f"forbidden tool: {name}")
    if name not in env.allowed_tools:
        raise BudgetBreach(f"unnamed tool: {name}")

def assert_spend(spend: Spend, env: Envelope, next_in: int, next_out: int) -> None:
    if spend.rounds > env.max_rounds:
        raise BudgetBreach("round cap")
    if spend.tool_calls > env.max_tool_calls:
        raise BudgetBreach("tool cap")
    if spend.input_tokens + next_in > env.max_input_tokens:
        raise BudgetBreach("input cap")
    if spend.output_tokens + next_out > env.max_output_tokens:
        raise BudgetBreach("output cap")

You call assert_tool_legal before any tool dispatch. You call assert_spend before every model completion. You never ask the model whether the cap still matters.

Wire the gate at the process boundary, not in the prompt. A prompt can be ignored under pressure. A raised exception cannot be ignored without logs.

# example: one round of a guarded loop
def run_round(env, spend, tool_name, prompt_tokens, completion_tokens, dispatch):
    assert_tool_legal(tool_name, env)
    spend.rounds += 1
    spend.tool_calls += 1
    assert_spend(spend, env, prompt_tokens, completion_tokens)
    result = dispatch(tool_name)
    spend.input_tokens += prompt_tokens
    spend.output_tokens += completion_tokens
    return result

If dispatch never runs after a breach, you already won. The cheapest token is the one you did not send.

Prove the abort with a hostile fixture

A happy-path demo is worthless for this merge rule. You need a fixture that wants to run forever. The fixture is the test, and the abort is the assertion.

# example: test_budget_abort.py
import pytest

def test_search_loop_aborts_on_tool_cap(gate):
    env = gate.load("envelope.json")
    spend = gate.Spend()
    with pytest.raises(gate.BudgetBreach, match="tool cap"):
        for _ in range(20):
            gate.assert_tool_legal("repo.search", env)
            spend.tool_calls += 1
            gate.assert_spend(spend, env, next_in=200, next_out=80)

def test_forbidden_shell_never_reaches_dispatch(gate):
    env = gate.load("envelope.json")
    with pytest.raises(gate.BudgetBreach, match="forbidden tool"):
        gate.assert_tool_legal("shell.exec", env)

def test_round_cap_beats_polite_retry(gate):
    env = gate.load("envelope.json")
    spend = gate.Spend()
    with pytest.raises(gate.BudgetBreach, match="round cap"):
        for _ in range(env.max_rounds + 2):
            spend.rounds += 1
            gate.assert_spend(spend, env, next_in=100, next_out=50)

def test_unnamed_tool_is_a_hard_miss(gate):
    env = gate.load("envelope.json")
    with pytest.raises(gate.BudgetBreach, match="unnamed tool"):
        gate.assert_tool_legal("repo.write", env)

Run it like any other unit suite today.

python -m pytest test_budget_abort.py -q

If that suite is red, the agent is not almost ready. It is unshippable, even if the sample task looks clean. You do not negotiate with a missing abort.

Add the suite to required checks on the agent branch. A required check beats a wiki page every time. People skip wikis when a demo is due Friday.

Decision table you can paste into the PR

Signal	Demo culture says	Merge rule says
Green sample task	Ship it	Ignore it
Tool retry on timeout	Try again	Count it as spend
Model asks for shell	Trust the rationale	Forbidden tool, abort
Output looks confident	Accept the patch	Check token caps first
Cap hit mid-task	Raise the cap	Fail the job
Human override just once	Allow it	Treat it as an incident
Unnamed tool appears	Add it quickly	Reject the run
Logs omit the breach reason	Ship anyway	Block merge

You paste the table into the pull request body. Reviewers then argue the numbers, not the vibes. That is the point of an opinionated gate.

Keep a short comment template next to the table. Ask one question only: did the abort fire on purpose? If nobody can answer, the run did not happen.

Rehearse on throwaway capacity

You still need a live loop to find abort bugs. Unit tests cannot see a model that rewrites the tool name. You need a rehearsal host that you can burn without drama.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Rehearse this envelope using MonkeyCode's free model access and free server option. Keep production keys off that throwaway box entirely.

Run only abort fixtures on that isolated host. Do not treat free capacity as proof of quality. Treat it as a place where a runaway loop is allowed to die.

If the abort never fires on the rehearsal host, it will not fire in CI either. Log every breach with the envelope identifier. Keep those logs beside the failing fixture, not in chat.

Do not tune the prompt until the gate is green. Prompt churn hides a missing cap. Caps belong in code, not in vibes.

# example rehearsal loop — proposed workflow, not a vendor CLI
export ENVELOPE_PATH=./envelope.json
python -m pytest test_budget_abort.py -q
python run_hostile_fixture.py --envelope "$ENVELOPE_PATH" --max-seconds 30

Stop the host after the fixture. A rehearsal box that stays up becomes production by accident. Accidental production is how polite retries become invoices.

Limitations you must not hand-wave

This envelope does not measure correctness of the patch. It only measures whether the loop can stop. A cheap abort can still merge a wrong diff if review is skipped.

Token counts are estimates unless your provider returns them. Prefer provider usage fields over local token guesses. Local guesses drift, and drift is a breach, not noise.

The allowed-tool list is not a sandbox by itself. A tool with broad power is still broad. repo.read can leak secrets if your tree is dirty.

Pair this gate with least-privilege credentials on every host. Pair it with replayable logs for every tool call. Pair it with a human review on anything that touches shared branches.

This will annoy agents that need one more search. That annoyance is the feature, not a bug. If your task cannot fit the cap, split the task.

Do not inflate the cap to protect the demo recording. Demo protection is how envelopes become fiction. Fiction is worse than a hard fail in CI.

Who should not use this

Do not use this as a substitute for code review. Do not use this on unsupervised production writes. Do not use this if you cannot replay the run from logs.

Skip it if you only generate one-shot snippets in an editor. Skip it if you have no tool-calling loop at all. Skip it if a hard budget proxy already sits in front of every model call.

This rule is for teams merging agent output into shared branches. If that is not you, the envelope is ceremony. Ceremony without a merge queue is just YAML.

Also skip it if your tools cannot be named in advance. Unnamed tool surfaces cannot be gated honestly. Honesty is the whole point of the abort.

The opinion, again

AI is not better at coding than you. It is better at emitting unfinished work at high speed. You remain the owner of spend, tools, and merge.

Build the abort before you chase the benchmark thread. Make cheap failure a required check. If the agent cannot fail cheap, you do not ship.

If You Cannot Replay the Agent, Do Not Merge

Emery Chen — Fri, 11 Sep 2026 16:01:52 +0000

You should refuse any agent patch you cannot replay. A scrolling demo is not a build artifact. Merge the replay file first, then consider the diff.

Take this position

Unreproducible agents are not production-grade coding assistants. They behave like unrecorded pair programmers with total amnesia. You would not accept that from a human contractor.

You would demand logs from a flaky integration test. Demand the same evidence from every coding-agent session. Otherwise you are shipping a one-time stage performance.

Opinion, not a hedged framework dump: replay or reject. Clever plans cannot replace a missing tool transcript. Your future self cannot debug a vanished chain of tool calls.

What replay actually means

Replay is not running the same prompt and hoping. Replay means stored inputs reconstruct the same side effects. You keep enough evidence to rebuild the job later.

A useful replay file must answer six concrete questions.

You stored the task prompt and every attached file hash.
You stored the tool schemas the process could legally call.
You stored each call, argument blob, and result digest.
You stored the model identifier as a pinned opaque string.
You stored resulting diffs as content-addressed patch files.
You stored the human decision: merge, reject, or isolate.

If any row is missing, you do not have a replay. You have a diary entry with extra punctuation. Diary entries do not belong on the main branch.

Cheap retries make the lie easier

Free or cheap tokens hide the cost of wasted loops. You retry until something compiles, then forget the path. Cheap retries without transcripts create irreproducible success stories.

Teams then paste a green terminal into Slack threads. Nobody can name the tool order that produced files. The next intern cannot reconstruct the change two weeks later.

That is how "the agent just knows the repo" myths start. The agent does not actually know your repository. It sampled a path you failed to record.

Practice the contract in a sandbox

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use that sandbox to practice the replay contract, not to skip it.

Keep the same checker when you later pay for inference. Free sandboxes do not lower the merge bar. They only keep experiments off your invoice.

Hunt these failure modes

Watch for patches that exist only inside a chat pane. Watch for tool traces truncated to look "readable" in demos. Watch for model labels copied from a pricing page.

If the transcript was edited by hand, discard the run. If file hashes drifted after cleanup, discard the run. If the PR description is only vibes, demand the file.

Pin a replay contract

Pin a JSON contract before you write more agent glue. The schema below is an example, not a shipped standard. Adapt field names to your orchestrator without adding poetry.

{
  "schema_version": "1",
  "task_id": "ISSUE-1842",
  "task_hash": "sha256:REPLACE_WITH_PROMPT_FILE_HASH",
  "model_id": "pinned-opaque-string",
  "base_git_sha": "abc1234",
  "allowed_tools": ["read_file", "write_file", "run_tests"],
  "tool_calls": [
    {
      "seq": 1,
      "name": "read_file",
      "args_digest": "sha256:ARGS",
      "result_digest": "sha256:BYTES"
    }
  ],
  "patch_digest": "sha256:GIT_DIFF",
  "human_decision": "review",
  "secrets_redacted": true
}

Every tool_calls item needs a digest, not a novel. Store stdout hashes when the bytes are large. Store raw text only when the payload is tiny and non-secret.

Keep a tight allowlist beside that JSON file. Unknown names fail the run. Broad shell access does not belong in the first draft.

# tools.allow  (example)
read_file
write_file
run_tests

Validate before you argue about the diff

Validate the file in CI before you discuss the patch. The checker below is an example, not a product. Do not treat a passing checker as proof of model quality.

#!/usr/bin/env python3
"""Example only: unexecuted replay-file checker. Not a benchmark."""
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

REQUIRED = {
    "schema_version",
    "task_hash",
    "model_id",
    "base_git_sha",
    "allowed_tools",
    "tool_calls",
    "patch_digest",
    "human_decision",
    "secrets_redacted",
}


def sha256_bytes(data: bytes) -> str:
    return "sha256:" + hashlib.sha256(data).hexdigest()


def fail(message: str) -> None:
    print("REPLAY_FAIL:", message)
    raise SystemExit(2)


def load_allowlist(path: Path) -> set[str]:
    names = set()
    for line in path.read_text(encoding="utf-8").splitlines():
        text = line.strip()
        if text and not text.startswith("#"):
            names.add(text)
    if not names:
        fail("allowlist is empty")
    return names


def main(argv: list[str]) -> None:
    if len(argv) != 3:
        fail("usage: check_replay.py replay.json tools.allow")
    replay_path = Path(argv[1])
    allow_path = Path(argv[2])
    payload = json.loads(replay_path.read_text(encoding="utf-8"))
    missing = REQUIRED - set(payload)
    if missing:
        fail(f"missing keys: {sorted(missing)}")
    if payload.get("secrets_redacted") is not True:
        fail("secrets_redacted must be true")
    if payload.get("human_decision") not in {"review", "reject", "isolate"}:
        fail("human_decision is not an allowed value")
    if not str(payload.get("patch_digest", "")).startswith("sha256:"):
        fail("patch_digest is not a sha256 digest")
    allow = load_allowlist(allow_path)
    declared = set(payload.get("allowed_tools") or [])
    if declared - allow:
        fail(f"tools not in allowlist file: {sorted(declared - allow)}")
    calls = payload.get("tool_calls") or []
    if not isinstance(calls, list) or not calls:
        fail("tool_calls must be a non-empty list")
    for item in calls:
        name = item.get("name")
        if name not in allow:
            fail(f"tool call not allowed: {name}")
        if not str(item.get("args_digest", "")).startswith("sha256:"):
            fail(f"args_digest missing for {name}")
        if not str(item.get("result_digest", "")).startswith("sha256:"):
            fail(f"result_digest missing for {name}")
    print("REPLAY_OK", sha256_bytes(replay_path.read_bytes()))


if __name__ == "__main__":
    main(sys.argv)

Run it on fixtures that should fail. A validator that never fails is only decoration. You want red on missing hashes, unknown tools, and empty diffs.

Merge with a boring table

Use a boring table instead of another agent self-eval. A human still owns merge after the table returns. The table exists to block unreproducible theater.

Signal	Action	Rule you enforce
Replay file missing from the PR	Block	No transcript means no merge
Tool name outside `tools.allow`	Block	The run is already dirty
Digest does not match saved bytes	Block	You cannot replay this job
`human_decision` is empty or `merge`	Block	The agent does not own merge
Tests arrived in the same agent burst	Hold	A human rewrites those tests
Replay valid and tests are human-owned	Review	You still read the diff

Notice merge is absent from allowed decisions. Review, reject, or isolate are the only honest exits. Auto-merge is how unreproducible patches escape.

Wire the gate into CI

Put the replay file next to the patch in the PR. Reject the PR when the checker exits non-zero. Reject the PR when hashes and diffs disagree.

# Example only: local gate before you open the PR.
python3 check_replay.py replay.json tools.allow
git rev-parse HEAD
git diff --binary "$BASE_SHA" > /tmp/agent.patch
python3 - <<'PY'
from pathlib import Path
import hashlib, json, sys
payload = json.loads(Path("replay.json").read_text())
raw = Path("/tmp/agent.patch").read_bytes()
digest = "sha256:" + hashlib.sha256(raw).hexdigest()
if digest != payload["patch_digest"]:
    print("PATCH_DIGEST_MISMATCH", digest)
    sys.exit(2)
print("PATCH_DIGEST_OK")
PY

Do not let the agent write this checker. You write the checker and you own it. The agent may propose patches, never the gate.

A debugging workflow when replay fails

When check_replay.py fails, stop generating more code. You are now in forensics, not in authoring. Follow this order and do not skip steps.

Order of operations

Confirm replay.json parses as UTF-8 JSON without comments.
Confirm task_hash matches a sha256 of the prompt file.
Confirm every tool name exists in the allowlist file.
Confirm each result_digest matches the saved artifact bytes.
Confirm patch hashes match git diff against the base sha.
Only then rerun the agent, with the same pinned model string.

If step six still drifts, you do not have replay yet. You have a non-deterministic tool or an unpinned runtime. Freeze the runtime before you blame the prompt.

What drift usually means

Drift usually means an unpinned tool, not a "creative" model. Network time, random files, and live clocks leak entropy. You stub those before you tune prompts again.

Record the stub, not the live socket. Live search and live HTTP are not replay sources. If a tool must touch the network, isolate that step from merge.

Limitations

This contract does not make a weak model strong. It does not prove the patch is correct, only reconstructable. Correctness still needs tests you wrote without the agent.

It will not capture GPU nondeterminism you refused to pin. It will not redact secrets you stuffed into prompts. It will not help if you never review the diff.

Floating-point tools, network calls, and clocks break naive replay. You must stub those tools or record their exact bytes. If you cannot stub them, isolate that step from merge.

Who should not use this approach

Do not use this as a permit for unattended production writes. Do not use this in regulated systems without a real audit log. Do not use this to rubber-stamp generated tests.

Skip the JSON file if you never invoke tools. A single-shot edit in your own editor needs a review, not theater. This gate is for looped agents with side effects.

If you cannot store transcripts safely, do not start. Prompts include secrets, customer names, and private URLs. You need a redaction step before any shared sandbox.

Close

You do not need a smarter agent to ship safer diffs. You need a replay file that survives a cold restart. If you cannot replay the agent, you do not merge.

If the Model Names the Tool, You Already Lost

Emery Chen — Wed, 09 Sep 2026 16:15:46 +0000

You do not have an agent quality problem. You have a raw tool-binding problem instead. A cheap model can draft a plan. It must never choose the executable tool name.

That split is the whole argument. Keep it even when inference looks free.

The opinion, stated plainly

Most agent stacks parse function names from model text. That pattern is a privilege escalation bug. The model should propose intent, not bind symbols. Your process should map intent to a pinned id.

Free models make the bug cheaper to ignore. They do not make the bug smaller. A free server does not forgive a string-keyed tools[name] call. You still handed the model your syscall table.

Why this fails in real loops

You dump a tool list into the prompt. The model returns run_shell or write_file. Your runtime then does registry[name](**args). One hallucinated string now owns disk and network.

This is not intelligence at work. This is if with extra tokens. Demo videos still sell longer loops and more tools. Your incident log will not thank that design.

Agents do not mainly fail at prose quality. They fail at the permission boundary. If the model can mint a new verb, you already lost.

Bind first. Generate second.

Treat tools like syscalls, not chat replies. Each tool gets a stable integer id. The model may output an intent key only. A local table maps that key to code.

No model string should reach getattr. No JSON name should index a function map. If the key is unknown, you refuse. You do not ask the model to retry the name.

Minimal binder you can paste

Label: this is a local fixture, not production telemetry.

# binder.py — pin tools by id, never by model text
from dataclasses import dataclass
from typing import Any, Callable, Dict, FrozenSet

@dataclass(frozen=True)
class Tool:
    tool_id: int
    intent: str
    fn: Callable[..., dict]
    mutating: bool

INTENTS = {
    "read_repo_file": 1,
    "list_repo_dir": 2,
    "run_unit_tests": 3,
    # write_file and shell are absent on purpose
}

class ToolBinder:
    def __init__(self, tools: Dict[int, Tool], allowed: FrozenSet[int]):
        self._tools = tools
        self._allowed = allowed

    def dispatch(self, intent: str, args: dict) -> dict:
        if intent not in INTENTS:
            return {"ok": False, "error": "unknown_intent"}
        tool_id = INTENTS[intent]
        if tool_id not in self._allowed:
            return {"ok": False, "error": "intent_not_allowed"}
        tool = self._tools[tool_id]
        return tool.fn(**args)

Notice what the model never sees. It never sees fn. It never sees numeric ids. It only sees a short intent vocabulary you wrote.

Reject the usual footgun

# anti_pattern.py — do not ship this
def call_tool(model_name: str, args: dict, registry: dict):
    # BAD: model text selects the callable
    return registry[model_name](**args)

If your framework requires a name field, wrap it. Map the name to an id inside your process. Drop the call when the map misses.

Capability receipts beat model excuses

Every allowed call must write a receipt line. You grep that file after the run. If the receipt set exceeds the allowlist, fail the job. Do not debate the model's self-report.

# receipts.py
import json
from pathlib import Path
from datetime import datetime, timezone

RECEIPT = Path("/tmp/agent_receipts.jsonl")

def write_receipt(intent: str, tool_id: int, mutating: bool) -> None:
    row = {
        "ts": datetime.now(timezone.utc).isoformat(),
        "intent": intent,
        "tool_id": tool_id,
        "mutating": mutating,
    }
    with RECEIPT.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(row) + "\n")

def assert_receipts(allowed_ids: set[int]) -> None:
    if not RECEIPT.exists():
        raise AssertionError("no receipts; loop did not run")
    seen = set()
    for line in RECEIPT.read_text(encoding="utf-8").splitlines():
        row = json.loads(line)
        seen.add(row["tool_id"])
        if row["tool_id"] not in allowed_ids:
            raise AssertionError(f"unbound tool_id={row['tool_id']}")
    extra = seen - allowed_ids
    if extra:
        raise AssertionError(f"extra tools {extra}")

Run the binder, then run assert_receipts. Green text from the model is irrelevant. The receipt file is the only scoreboard.

Decision table you can copy

Use this before you attach any runtime.

Intent key	Tool id	Mutates?	Free-model planner may propose?	Local process may execute?
`read_repo_file`	1	no	yes	yes, path allowlisted
`list_repo_dir`	2	no	yes	yes, path allowlisted
`run_unit_tests`	3	no	yes	yes, pinned command
`write_repo_file`	4	yes	yes	only after human pin
`run_shell`	5	yes	no	never from model text
any other string	—	unknown	no	no, hard refuse

Read the last row twice. Unknown intent is not a retry. Unknown intent is a failed test.

Where a cheap planner actually belongs

You still need a place to iterate on proposals. A planner that cannot bind tools is safer. Keep that planner off the write path. Keep execution on your machine, behind the binder.

MonkeyCode is an open-source coding assistant. It provides free model access and a free server option for that planner lane.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use that lane to draft intent lists and dry runs. Do not move ToolBinder.dispatch onto a shared box. The free server is for proposals you can throw away. Your binder remains local code you review.

If you want a scratch planner, try it there once. Then paste receipts into CI, not chat logs.

Reproducible test plan

Do this on a throwaway repo. Do not point it at production secrets.

Freeze three intents only: read, list, test.
Seed /tmp/agent_receipts.jsonl as an empty file.
Feed the planner a task that begs for run_shell.
Assert the binder returns unknown_intent or intent_not_allowed.
Assert no receipt contains a mutating tool_id.
Change the prompt. Repeat the same asserts.
Only then add write_repo_file behind a human pin.

# test_binder.py
from binder import Tool, ToolBinder, INTENTS

def test_model_cannot_invent_shell():
    tools = {
        1: Tool(1, "read_repo_file", lambda path: {"ok": True, "path": path}, False),
    }
    binder = ToolBinder(tools, allowed=frozenset({1}))
    result = binder.dispatch("run_shell", {"cmd": "rm -rf /"})
    assert result["ok"] is False
    assert result["error"] == "unknown_intent"

def test_allowed_read_still_works():
    tools = {
        1: Tool(1, "read_repo_file", lambda path: {"ok": True, "path": path}, False),
    }
    binder = ToolBinder(tools, allowed=frozenset({1}))
    result = binder.dispatch("read_repo_file", {"path": "README.md"})
    assert result == {"ok": True, "path": "README.md"}

Run it with a boring command. You want the refuse path to be loud.

python -m pytest test_binder.py -q

If that test needs a network model, stop. The binder must fail closed offline. Network is for proposal text, not for permission.

Limitations, without theater

This does not stop a compromised host. It does not replace sandboxing or code review. It does not prove the model understood the task. It only proves the model could not pick a new verb.

Intent keys can still be too broad. run_unit_tests might hide a custom script. Pin the argv. Pin the working directory. Pin the timeout.

Receipts are not cryptography. Anyone with write access can edit the log. Put the file on a tmpfs you control. Ship it to CI as an artifact, then diff.

Free model access will still drift. Prompts will still beg for extra tools. Your table, not the vendor card, is the contract.

Who should not use this approach

Do not use this if you need a fully autonomous write loop. This design will block that fantasy on purpose. Do not use this if your framework hides the tool map. If you cannot intercept dispatch, you cannot bind.

Do not use this on regulated data without your own review. A free server is a shared planning box, not a vault. Do not use this as a substitute for human merge rights. The binder is a seatbelt, not a driver.

Skip it if you only generate comments. You do not need receipts for pure text. Apply it when a tool can touch git, files, or money.

Close the privilege hole first

Bigger models will not save a string-keyed registry. Cheaper models will not excuse one either. You bind tools in process, then you let a planner talk.

If the model still names the tool, you already lost. Fix the binder before you tune the prompt.

Hard-Stop the Loop Before You Touch Free Inference

Emery Chen — Tue, 08 Sep 2026 04:38:32 +0000

You should hard-stop an agent before you touch free inference. The prompt cannot kill a retry storm. A missing price signal is the real bug.

Take a side

This is not an argument against free models. It is an argument against unbounded agent loops. You must put the brake in your own client.

Developers still paste stop rules into system prompts. The model can ignore every one of those rules. Your process will not notice until files change.

You keep calling because nothing in the loop stings. Cheap inference hides the cost of another retry. That hidden cost shows up as a dirty worktree.

Why the failure mode shifted

Paid APIs taught you to fear long loops. A rising bill was a crude circuit breaker. You felt the damage in the same working hour.

Free model access removes that pain on purpose. A free server option removes the pain faster. You will retry because each retry looks free.

Agent posts this week keep celebrating extra autonomy. The hard part is still forcing the loop to halt. Assumption-heavy agents burn steps when nothing is priced.

You do not need another glossary of agent terms. You need a kill switch that lives outside the model.

Cheap generation is not cheap cleanup

Industry talk this week worries about AI-made technical debt. Debt arrives faster when a loop can keep writing. Your write budget is a debt budget in disguise.

You do not need a new architecture sermon. You need fewer unattended writes per session. Architecture opinions cannot outrun a loop with no ceiling.

Three budgets that actually halt a loop

One budget is never enough for an agent. Split the cap across steps, writes, and time. Fail closed on the first ceiling you hit.

Step budget

Count every model call as one spent step. Count tool retries as steps, not as free extras. Eight steps is plenty for a local coding task.

Write budget

Reads can be noisy without wrecking the repo. Writes need a tiny and explicit ceiling. Two writes per task is a sane default.

Guessing can stay in the think path. Mutating the tree on a guess is not allowed.

Wall-clock budget

Tokens are not the only runaway resource. A stuck tool call can pin a free server. Forty-five seconds is enough to prove a local patch.

If the clock expires, you stop without extra pleas. Do not ask the model whether more time would help. The model will almost always say yes.

How retry storms start

The model fails a patch and asks to try again. Your wrapper says yes because the call looks cheap. The same broken tool schema comes back three times.

Nothing in the prompt notices the repeated failure. A step counter notices on the next checkpoint. That is why the counter is not optional.

Map every tool to an action class

Classify each tool before you expose it to the loop. If you cannot classify it, do not register the tool. Unclassified tools are how silent writes sneak in.

think produces a plan and must have no side effects.
read opens files, searches, or runs dry tests only.
write patches files, installs packages, or mutates state.
retry repeats a failed tool with the same intent.

Reads still consume steps, because steps are the loop. Only write increments the write ceiling. retry should get harsher as you near the step cap.

The gate belongs in code

Do not negotiate with the model about stopping. Raise an exception in your process and halt. The snippet below is a local, copy-paste gate.

Treat the helper as a starting module, not a vendor SDK.

# agent_budget.py
from __future__ import annotations

import json
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Literal


class BudgetExceeded(RuntimeError):
    pass


@dataclass
class AgentBudget:
    max_steps: int = 8
    max_writes: int = 2
    max_seconds: float = 45.0
    ledger_path: Path = Path('agent_ledger.jsonl')
    started_at: float = field(default_factory=time.monotonic)
    steps: int = 0
    writes: int = 0

    def checkpoint(
        self,
        action: Literal['think', 'read', 'write', 'retry'],
        detail: str,
    ) -> None:
        elapsed = time.monotonic() - self.started_at
        self.steps += 1
        if action == 'write':
            self.writes += 1
        record = {
            'step': self.steps,
            'action': action,
            'detail': detail,
            'writes': self.writes,
            'elapsed_s': round(elapsed, 3),
        }
        with self.ledger_path.open('a', encoding='utf-8') as fh:
            fh.write(json.dumps(record) + '\n')
        self._enforce(action, elapsed)

    def _enforce(self, action: str, elapsed: float) -> None:
        if self.steps > self.max_steps:
            raise BudgetExceeded(
                f'steps {self.steps} > {self.max_steps}'
            )
        if self.writes > self.max_writes:
            raise BudgetExceeded(
                f'writes {self.writes} > {self.max_writes}'
            )
        if elapsed > self.max_seconds:
            raise BudgetExceeded(
                f'elapsed {elapsed:.1f}s > {self.max_seconds}s'
            )
        if action == 'retry' and self.steps >= self.max_steps - 1:
            raise BudgetExceeded('retry blocked near step ceiling')

Wire the gate around every tool dispatch. Catch BudgetExceeded and refuse the pending side effect. Log the reason and do not ask the model for mercy.

# proposed loop sketch — run only after the tests below pass
from agent_budget import AgentBudget, BudgetExceeded


def run_task(budget: AgentBudget, tools: dict, intent: str) -> None:
    budget.checkpoint('think', intent[:120])
    try:
        result = tools['read'](intent)
        budget.checkpoint('read', 'tool:read')
        if not result.ok:
            budget.checkpoint('retry', 'tool:read')
            result = tools['read'](intent)
        if result.needs_write:
            budget.checkpoint('write', 'tool:write')
            tools['write'](result.patch)
    except BudgetExceeded as exc:
        raise SystemExit(f'fail closed: {exc}') from exc

What the ledger must record

If you cannot see the path, you cannot kill it. Append one JSON line for every client action. Keep the schema boring, stable, and secret-free.

Required fields:

step: client counter, not a model claim
action: think, read, write, or retry
detail: short text with no secrets
writes: write count so far
elapsed_s: seconds since started_at

Do not log raw prompts if they contain secrets. Do not log tokens you did not actually measure. An honest ledger beats a decorative status dashboard.

{"step": 3, "action": "retry", "detail": "tool:read", "writes": 0, "elapsed_s": 4.812}

Tests you run before the first write

These tests are a checklist, not a benchmark. They do not prove a hosted model will behave. They prove your client will refuse to continue.

# test_agent_budget.py
import time
from pathlib import Path

import pytest

from agent_budget import AgentBudget, BudgetExceeded


def test_step_ceiling_fails_closed(tmp_path: Path) -> None:
    budget = AgentBudget(
        max_steps=2,
        max_writes=5,
        max_seconds=30,
        ledger_path=tmp_path / 'l.jsonl',
    )
    budget.checkpoint('read', 'a')
    budget.checkpoint('read', 'b')
    with pytest.raises(BudgetExceeded):
        budget.checkpoint('read', 'c')


def test_write_ceiling_is_stricter_than_reads(tmp_path: Path) -> None:
    budget = AgentBudget(
        max_steps=20,
        max_writes=1,
        max_seconds=30,
        ledger_path=tmp_path / 'l.jsonl',
    )
    budget.checkpoint('read', 'a')
    budget.checkpoint('write', 'b')
    with pytest.raises(BudgetExceeded):
        budget.checkpoint('write', 'c')


def test_retry_near_ceiling_is_blocked(tmp_path: Path) -> None:
    budget = AgentBudget(
        max_steps=3,
        max_writes=5,
        max_seconds=30,
        ledger_path=tmp_path / 'l.jsonl',
    )
    budget.checkpoint('read', 'a')
    budget.checkpoint('read', 'b')
    with pytest.raises(BudgetExceeded):
        budget.checkpoint('retry', 'b-again')


def test_wall_clock_fails_closed(
    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
    budget = AgentBudget(
        max_steps=50,
        max_writes=5,
        max_seconds=0.01,
        ledger_path=tmp_path / 'l.jsonl',
    )
    monkeypatch.setattr(
        time, 'monotonic', lambda: budget.started_at + 1.0
    )
    with pytest.raises(BudgetExceeded):
        budget.checkpoint('think', 'late')

Run the file with pytest before you enable writes. If a test fails, the loop stays in read-only mode. That is the whole point of a client-side brake.

Decision table

Read the table before you wire retries. Die on the first ceiling you actually hit.

Signal	Retry allowed	Client action
Read failed, steps left, no writes yet	Yes, once	Log `retry`, then `read`
Read failed, one step left	No	Raise `BudgetExceeded`
Patch failed after one write	No	Stop and open the ledger
Model asks for one more try	No	Ignore the plea
Wall clock past cap	No	Kill the process
Tool is unclassified	No	Do not register it

You retry only when the budget still has room. You never retry a write that already failed closed. Ambiguous model output is not a reason to spend steps.

Where a free model path fits

You still need somewhere to practice the gate. A local coding agent is useful if the brake is on.

MonkeyCode is an open-source project for coding agents. It offers free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use that path to rehearse the budget, not to skip it. The free access is relevant only as a place to iterate. Your kill switch still lives in the Python process.

Limitations

This gate estimates nothing about true billed tokens. It cannot see rate limits inside a hosted server. It will not protect you from a bad tool implementation.

JSONL logs can leak paths if you are sloppy. Wall-clock caps fight hung subprocesses, not all of them. You still need OS-level timeouts around shell tools.

Do not treat eight steps as a universal constant. Tiny refactors and wide migrations need different ceilings. Change the numbers in code, not in the prompt.

Free model access can change behavior without a local notice. A free server option is not an uptime contract. Pin your tests so a silent change still fails closed.

Who should skip this

Skip this if you already have provider-side hard quotas. Skip this if legal logging rules forbid local JSONL. Skip this for unattended bots that must page a human.

Multi-tenant products need real accounting, not a dataclass. This article is for a single developer loop. It is not a billing system and not a sandbox.

Close

Free inference is a gift that deletes your warning light. You should answer that gift with a harsher client. Pin steps, writes, and time before the first call.

After the tests pass, try MonkeyCode as a practice bench.

DEV Community: Emery Chen

If the Model Moved, Burn the Golden Files

The argument

The tests did not get weaker

Three layers beat one snapshot

Contracts

Effects

Critique

Opinion: the critic must stay cheap

Artifact: a living eval harness

Step 1 — Capture traces as JSONL

Step 2 — Fail closed on contracts

Step 3 — Pin the critic prompt in git

Step 4 — Call a cheap HTTP endpoint

Step 5 — Make the ritual one command

Decision table you can copy

Why the critic belongs off your laptop

Limitations you should not hand-wave

Who should not run this

After the next vendor bump

Your Tool Schema Should Outlive the Model

Take the position

Single-model green is a rumor

Compare the contract, not the essay

Proposed swap harness

1. Freeze a task fixture

2. Extract a stable shape

3. Fail the job in CI

Decision table for review

Why the shadow run must be remote and cheap

Debug disagreement in a fixed order

Limitations

Who should not use this

What to run this week

An Agent That Needs Your Laptop Is Not Ready

Local success is a contaminated sample

Name the contaminants before you argue prompts

The opinion, without a hedge

What the laptop hides on purpose

A clean-room gate you can actually run

Provision a workspace that is not $HOME

Strip the environment on purpose

Assert identity leakage, not writing quality

Tear the box down every time

Decision table: when local is allowed

Why a free model belongs in this gate

What this gate will not catch

Who should not use this approach

Operating rule you can paste on the wiki

Stop Calling It Staging If the Agent Never Leaves Localhost

The claim

What you should measure

Freeze the contract first

Build a fixture, not a vibe

Run the canary off the laptop

Put it in CI, not in a screenshot

Decision table

Why a free remote slot changes the argument

What this will not catch

Who should not use this

Limitations you should write down

A short review script

Close

Token Caps Are Tests. Treat Them That Way.

The claim

What you are pretending

Define the fuse before the prompt

Why three numbers beat one dollar cap

Decision table

Pick numbers like timeouts, not like hopes

Artifact: a budget gate you can run

A second test: tool rounds

Why the host must not be your laptop

Wire the fuse into the job

Log the trip like an incident

What this does not measure

Who should not use this

Limitations you must name

The opinion, restated

Stop Scoring Prose. Start Diffing Tool Shapes.

Provision a workspace that is not `$HOME`