DEV Community: Emery Chen

Your Token Budget Is a Test, Not a Dashboard

Emery Chen — Mon, 26 Oct 2026 11:49:47 +0000

Most teams learn about prompt bloat from an invoice. That is the wrong feedback loop. A token budget should fail a pull request, not a finance review.

My position is blunt. Cost assertions belong in version control, next to the prompt they measure. Dashboards are for exploration. Tests are for enforcement.

Dashboards are a lagging signal

Observability tells you what already shipped. It aggregates across branches, so nobody owns the number. By the time the chart bends, the regression is already on main.

A gate behaves differently. It blocks the merge, names the fixture, prints the delta. One reviewer decides: fix the prompt, or accept the growth on purpose.

What a budget gate asserts

Assembling fixture x costs no more than baseline plus tolerance.
A new fixture arrives as a reviewed commit, not a side effect.
A tokenizer or model change bumps a counter_version field.

Point three matters most. Token counts are not portable across tokenizers. A silent tokenizer upgrade can swing every number at once. Version the counter, and your baseline diff stays honest.

Fixtures, not live traffic

Never call a provider inside the blocking job. Store inputs as files. Record counts in a separate refresh job you can rerun at will.

Suggested layout

budget/
  fixtures/summarise.txt
  fixtures/classify.txt
  baseline.json
  gate.py
  record.py
  test_gate.py

baseline.json is small, reviewable data:

{
  "counter_version": "word-proxy-1",
  "fixtures": {
    "summarise": 120,
    "classify": 40
  }
}

The gate: sixty lines, no network

The gate compares numbers. It never touches a model.

# budget/gate.py
"""Fail a build when prompt assembly grows past a recorded baseline."""
import json
import sys
from pathlib import Path

TOL = 0.05  # allowed growth per fixture


def load(path: str) -> dict:
    return json.loads(Path(path).read_text(encoding="utf-8"))


def compare(baseline: dict, current: dict, tol: float = TOL):
    rows, failures = [], []
    for name, base in sorted(baseline["fixtures"].items()):
        cur = current["fixtures"].get(name)
        if cur is None:
            failures.append((name, "fixture missing from current run"))
            continue
        growth = (cur - base) / base if base else float("inf")
        rows.append((name, base, cur, growth))
        if growth > tol:
            failures.append((name, f"{growth:+.1%} over baseline of {base}"))
    return rows, failures


def main(argv: list[str]) -> int:
    baseline, current = load(argv[1]), load(argv[2])
    tol = float(argv[3]) if len(argv) > 3 else TOL
    if baseline["counter_version"] != current["counter_version"]:
        print("counter_version changed: re-record the baseline in its own commit")
        return 2
    rows, failures = compare(baseline, current, tol)
    for name, base, cur, growth in rows:
        print(f"{name:<24} {base:>7} -> {cur:>7}  {growth:+.1%}")
    if failures:
        print("\nbudget gate failed:")
        for name, why in failures:
            print(f"  - {name}: {why}")
        return 1
    print("\nbudget gate passed")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv))

Run it locally against two JSON files. That is the whole contract.

python budget/gate.py budget/baseline.json budget/current.json 0.05

Prove the gate before you trust it

A gate nobody tested is a gate everyone disables. Keep three offline tests.

# budget/test_gate.py
import unittest
from budget.gate import compare

BASE = {
    "counter_version": "word-proxy-1",
    "fixtures": {"summarise": 120, "classify": 40},
}


def current(summarise: int, classify: int = 40) -> dict:
    return {
        "counter_version": "word-proxy-1",
        "fixtures": {"summarise": summarise, "classify": classify},
    }


class GateTest(unittest.TestCase):
    def test_small_growth_passes(self):
        rows, failures = compare(BASE, current(124), tol=0.05)
        self.assertEqual(failures, [])

    def test_bloat_fails(self):
        rows, failures = compare(BASE, current(180), tol=0.05)
        self.assertEqual(len(failures), 1)

    def test_missing_fixture_fails(self):
        data = current(120)
        data["fixtures"].pop("summarise")
        rows, failures = compare(BASE, data, tol=0.05)
        self.assertEqual(len(failures), 1)


if __name__ == "__main__":
    unittest.main()

Run them with no network and no credentials.

python -m unittest budget.test_gate -v

If these pass, the logic is sound. Any later failure is a real prompt change, not a bug in your harness.

Count with a proxy first, a real tokenizer second

Start with something deterministic and obviously fake.

# budget/count.py
class WordProxy:
    """A stand-in, not a tokenizer. Use it to prove the plumbing works."""
    version = "word-proxy-1"

    def count(self, text: str) -> int:
        return len(text.split())

WordProxy counts words. It is a placeholder, not a billing estimate. Ship it only to validate the pipeline.

Then swap in your provider's tokenizer for real numbers. Bump counter_version in the same commit. Otherwise every fixture looks like a regression at once.

Wire it into CI with a disposable runner

Keep the blocking job cheap and offline.

name: budget-gate
on: [pull_request]
jobs:
  budget:
    runs-on: [self-hosted, budget-runner]
    steps:
      - uses: actions/checkout@v4
      - run: python -m budget.record --offline --out budget/current.json
      - run: python budget/gate.py budget/baseline.json budget/current.json 0.05

A self-hosted runner fits this job well. Re-recording is bursty and boring, and it should never touch your laptop.

This is where a free tier is genuinely useful. MonkeyCode's operator describes free model access and a free server option, with an advertised allowance of 10 million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not benchmarked it, and I am not claiming it replaces a paid quota.

Treat that machine as a runner, not a pet. If it disappears, record.py reruns somewhere else. Nothing in the gate depends on it.

The baseline commit is the weak link

Every cost gate dies the same way. Someone refreshes the baseline to unblock a merge.

Four rules keep it alive:

Baseline edits need a reviewer who is not the author.
Require a one-line reason in the commit body.
Never refresh the baseline inside the gate job.
Pin one counter_version per commit.

Without these, just refresh the baseline becomes the new just rerun the flaky test.

When the gate fails, debug in this order

Read the failing fixture name; ignore the rest.
Diff the prompt template for that fixture only.
Check injected context: retrieval chunks, tool schemas, few-shot examples.
Confirm the counter version before editing any prompt.
Fix the smallest cause. Rerun the gate locally.
If the growth is intentional, bump the baseline in its own commit.

Most failures land on step three. Someone adds a document chunk or a tool schema. Cost climbs and quality does not.

Where this approach breaks

Retrieval fixtures are nondeterministic; snapshot the retrieved chunks instead of the query.
Prompt caching and reasoning tokens can change billed cost without changing prompt text.
Under five prompts, the bookkeeping costs more than it saves.
Prompts edited in a vendor dashboard are invisible to git; this gate cannot see them.
Tokenizers move under you; record the baseline deliberately, never automatically.

Decision table

Situation	Gate in CI?	Reason
5-20 stable prompts in the repo	Yes	Cheap to maintain, catches drift early
Prompts edited in a vendor UI	No, export first	The gate cannot read what git cannot see
Heavy RAG pipeline	Yes, with chunk snapshots	Retrieval is the usual source of bloat
Multi-tenant prompt variants	Yes, one fixture per variant	A single variant can regress alone
Throwaway prototype	No	Tolerance tuning is wasted effort
Cost driven by output length	Different test	Assert on `max_tokens`, not prompt size

Who should skip this

Skip it if your prompts are not files. Skip it if your spend is dominated by long completions. Skip it if nobody will review a baseline diff, because then the gate is decoration.

The takeaway

Cost regressions are code regressions. They deserve the same enforcement. Put the number in the repo, fail the pull request, and let a human decide when the number is allowed to move.

If you need a throwaway runner for the refresh job, MonkeyCode's free server option is worth a look. The gate itself should never depend on it.

Free Compute Is a Lease, Not a Gift

Emery Chen — Tue, 20 Oct 2026 09:54:53 +0000

Free tiers are great for disposable work. They are terrible for load-bearing work. Pick that boundary before you pick a product.

I keep seeing the same failure. A team gets free model access and a free server. Two weeks later, a cron job, a dashboard, and someone's demo all depend on it. Nobody knows when it stops, or what breaks first.

The cost was never the problem. The missing contract was.

The free tier problem is drift, not price

Free resources do not fail politely. They expire, throttle, or vanish while your job is mid-write.

Three failure modes show up again and again:

Time drift. The box outlives your attention. Nobody remembers the expiry date.
Spend drift. Token usage creeps until the ceiling hits during a demo.
State drift. The only copy of a result lives on a machine with no promise of tomorrow.

None of these are billing problems. All three are contract problems.

Write the contract as a lease

My position is simple. Every experiment on free capacity gets a lease file, and nothing runs without one.

A lease has three required fields and one optional one:

Deadline. An absolute UTC timestamp, never "in two hours."
Token ceiling. A hard number, not a vibe.
Sink. A durable path outside the free machine where results land.
Teardown. The command that revokes access and stops work.

Here is the file. Copy it, edit it, keep it in git next to the code it governs.

{
  "id": "exp-2026-10-20-a",
  "deadline_utc": "2026-10-20T18:00:00+00:00",
  "token_ceiling": 200000,
  "usage_file": "usage.json",
  "sink": "results/exp-2026-10-20-a.jsonl",
  "teardown": "./teardown.sh"
}

Two details matter more than they look. The deadline is absolute, so timezone bugs cannot extend it. The sink is a path in your repo, so the free box is never the system of record.

A gate that refuses to start

The point of a gate is refusal. A warning you can ignore is decoration.

This script is plain Python 3.11 with no dependencies. Review it before you run it. It exits non-zero when the lease is invalid.

#!/usr/bin/env python3
"""lease_check.py - refuse to start when the lease is invalid."""
import json
import sys
from datetime import datetime, timezone
from pathlib import Path

LEASE = Path("lease.json")

def main() -> int:
    lease = json.loads(LEASE.read_text())
    deadline = datetime.fromisoformat(lease["deadline_utc"])
    left = (deadline - datetime.now(timezone.utc)).total_seconds()

    usage = Path(lease["usage_file"])
    spent = json.loads(usage.read_text())["tokens"] if usage.exists() else 0
    ceiling = int(lease["token_ceiling"])

    if left <= 0:
        print(f"REFUSE: lease {lease['id']} expired {-left / 60:.0f} min ago")
        return 2
    if spent >= ceiling:
        print(f"REFUSE: {spent}/{ceiling} tokens spent, write the sink")
        return 3

    print(f"OK: {left / 60:.0f} min left, {ceiling - spent} tokens left")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Wire it in front of your runner, not inside it. One line, hard stop.

python3 lease_check.py || exit 1
python3 run_experiment.py \
  --out results/exp-2026-10-20-a.jsonl \
  --max-tokens 200000

The ceiling is enforced twice on purpose. The gate blocks a bad start. The runner caps a long one.

Teardown must be boring

Teardown is where free tiers actually get abandoned. Make it three steps, in this order.

Stop work. Nothing should write after the sink closes.
Flush the sink. Commit before you delete anything.
Revoke access. Credentials die last, so recovery stays possible.

#!/usr/bin/env bash
set -euo pipefail

lease_id="$(jq -r .id lease.json)"

pkill -f "experiment:${lease_id}" || true

test -s "results/${lease_id}.jsonl" || {
  echo "refuse teardown: sink is empty"; exit 1
}
git add "results/${lease_id}.jsonl"
git commit -m "sink: ${lease_id}" || true

rm -f .env.local || true
echo "torn down ${lease_id}"

Run it twice. It should succeed both times. A teardown that only works once is a cleanup task, not a contract.

What belongs on free capacity

Workload	Use free capacity?	Reason
Throwaway benchmark harness	Yes	Replaceable, results already in git
Prompt regression suite	Yes	Fixed inputs, small deterministic outputs
Cron job that pings customers	No	Needs uptime and durable state
The only copy of a dataset	No	Sink rule violated at day zero
Long training run	No	The lease expires mid-run
Integration tests with scoped keys	Yes	Rotatable credentials, no customer data

Read the table as a filter, not a ranking. The question is never "is this free?" The question is "is this disposable?"

What hurts, and who should walk away

This pattern has real costs. Say them out loud before adopting it.

Teardown can fire mid-run and kill a slow experiment. Budget for retries.
Token counting differs between providers. Treat your ceiling as approximate, then subtract a safety margin.
The lease file is a new artifact to review. Small, but not zero.
A ceiling cannot rescue a bad prompt. It only limits the damage.

Do not use this approach if you need an uptime commitment, a compliance audit trail, or regulated data handling. Do not use it if nobody on the team will own the deadline. A lease with no owner is a countdown timer nobody watches.

Where MonkeyCode fits

The pattern above needs cheap, expendable capacity. That is exactly what free tiers are good at, and nothing more.

MonkeyCode is an open-source project that offers free model access and a free server option. The operator indicates a 10M-token allowance as part of that offer. I am not going to dress that up as an architecture. It is capacity, and capacity needs a contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it as the first row of the table. Run a harness, write to a git sink, tear the box down, and keep the results. If you want to test the lease pattern without spending money, that is a reasonable place to start.

My closing position

Measure free resources in minutes and tokens, not in gratitude. A tier you cannot expire is a dependency you did not choose.

Write the lease first. Then pick the platform. The order matters more than the tool.

Opinion: Free Model Access Is Worthless Without a Kill Switch

Emery Chen — Mon, 19 Oct 2026 08:36:17 +0000

Free capacity is not the hard part anymore. Throwing it away safely is.

My position: free model access and a free server only matter if your workload is disposable. If it is not, the free tier just moves the failure later. You get the same outage with worse logs.

MonkeyCode is an open-source coding agent project with free model access and a free server option. Both availability claims come from the operator, not from a benchmark I ran. I have not measured their throughput, limits, or uptime.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The thing everyone skips

Cheap inference is easy to demo. It is hard to operate.

Nobody posts about the run that stalled at task 37. Nobody shows the ledger where token spend silently tripled. That failure mode is boring, and boring is where the money goes.

So the real question is not "how many free tokens do I get". It is "what stops the run".

Three properties of a disposable workload

Before I put anything on free capacity, I check three things.

Stateless. Kill the box mid-run and nothing is lost but time.
Bounded. The run has a hard call cap and a wall-clock cap.
Replayable. Inputs are in version control, so a rerun is one command.

Fail any one and free capacity becomes expensive. You pay in debugging hours instead of dollars.

The artifact: a budget-guarded runner

Here is the smallest kill switch I trust. It works against any OpenAI-style chat completions endpoint.

# budget_runner.py — hard caps first, retries never
import json, os, sys, time, urllib.request

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]
MODEL = os.environ.get("LLM_MODEL", "default")
MAX_CALLS = int(os.environ.get("MAX_CALLS", "20"))
MAX_SECONDS = int(os.environ.get("MAX_SECONDS", "300"))
LEDGER = os.environ.get("LEDGER", "ledger.jsonl")


def call(prompt):
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=body,
        headers={
            "Authorization": f"Bearer {KEY}",
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.loads(r.read())


def main():
    prompts = [line.strip() for line in sys.stdin if line.strip()]
    started = time.time()
    with open(LEDGER, "a") as log:
        for i, prompt in enumerate(prompts):
            if i >= MAX_CALLS or time.time() - started > MAX_SECONDS:
                print(f"STOP: budget hit at task {i}", file=sys.stderr)
                sys.exit(2)
            t0 = time.time()
            try:
                resp = call(prompt)
                rec = {"i": i, "ok": True, "secs": round(time.time() - t0, 2),
                       "tokens": resp.get("usage", {}).get("total_tokens")}
            except Exception as exc:
                rec = {"i": i, "ok": False, "secs": round(time.time() - t0, 2),
                       "err": type(exc).__name__}
            log.write(json.dumps(rec) + "\n")
            log.flush()
            print(json.dumps(rec))


if __name__ == "__main__":
    main()

Run it like this. The caps are the point.

printf '%s\n' "summarise src/a.py" "summarise src/b.py" \
  | MAX_CALLS=5 MAX_SECONDS=120 LLM_MODEL=default \
    LLM_BASE_URL=$BASE LLM_API_KEY=$KEY \
    python budget_runner.py

Three behaviours matter here.

Exit code 2 means the budget stopped you, not the network.
No retries. A failed call is data, not a trigger.
The ledger is append-only. One JSONL file per run, diffable across runs.

I have not executed this against the free server. Treat it as a template you adapt to the documented base URL and auth shape.

Why no retries is a feature

Free capacity blips. That is expected. Silent retries hide it.

A retry loop turns one bad minute into a spend spike and a flaky test suite. Your ledger stops meaning anything. You can no longer answer "did the run fail, or did we paper over it".

Count failures instead. If the failure rate is above your tolerance, the workload is not ready for shared capacity. That is a design answer, not an infrastructure answer.

Decision table

Workload	Free capacity OK?	Reason	Guard you need
Refactors on a scratch clone	Yes	Throwaway branch, no secrets	Call cap + clone deletion
Test generation for review	Yes	Human reviews every diff	Diff size cap
Docstring and changelog drafts	Yes	Low blast radius	Read-only repo access
Long agent loops with tool use	Carefully	One bad loop burns hours	Wall-clock cap, no auto-merge
Anything touching customer data	No	Shared capacity, unclear retention	None exists — use your own
Latency-sensitive endpoints	No	No SLA on free anything	Not a guard problem

Read the table as a filter, not a ranking. Most teams should start at row one.

Debugging workflow when the box dies

It will die. Here is the order I check things in.

Read the ledger tail. Confirm the last task index and elapsed seconds.
Check the exit path. Code 2 means your cap fired. Anything else means the environment.
Rerun with MAX_CALLS=1. If that fails, the problem is config, not load.
Diff two ledgers. Same prompt count, different timing means shared capacity noise.
Only then widen the cap. One variable at a time, always.

Step 4 is the one people skip. Two ledgers from the same inputs tell you more than any dashboard.

Where this breaks

The approach has real limits, and I would rather name them.

Shared capacity is shared. Timing will vary and you cannot schedule around it.
The free server is disposable by design. Do not treat local disk as storage.
No SLA means no on-call story. Route alerts to a human who can switch providers.
You still need a paid path. Free capacity is a ramp, not a floor.
Compliance does not care about free. Data retention rules apply either way.

Who should not use this

Skip this workflow if you run production traffic, handle regulated data, or promise latency numbers. Skip it if your team cannot review generated diffs. Free capacity will not fix an unreviewed merge queue. It just makes the queue longer and cheaper to fill.

If you are prototyping, generation-testing, or teaching an agent a new repo, the economics flip. You want cheap, disposable, and boring.

The position, restated

Stop evaluating free capacity by how much you get. Evaluate it by how fast you can walk away.

A kill switch is ten lines of Python. It is also the difference between a free experiment and an unowned incident.

If you want to try the setup, MonkeyCode's free model access and free server option are the cheapest way I know to run this harness for an afternoon and read your own ledger.

Stop Shopping for Models: Your Harness Is the Only Asset You Keep

Emery Chen — Sun, 18 Oct 2026 07:16:07 +0000

The opinion, up front

Model choice is a commodity decision. The harness that proves a model is good enough is the asset. Most teams have this exactly backwards.

They spend three weeks reading leaderboards. They spend one afternoon writing the test that decides the switch. Then they argue about models again next quarter.

Free tiers make this worse, not better. Cheap access invites endless shopping. My position is blunt: pick a model in an hour, then spend a month hardening the swap path.

Everything below is that swap path.

Portability beats preference

You will change models. Prices move, endpoints deprecate, defaults shift under you. That is not a prediction. That is the normal lifecycle of a dependency.

If your prompts live inside SDK-specific calls, every change is a migration. If they live behind one HTTP shape, every change is an environment variable.

Three things survive a swap:

Your test cases and their expected outputs.
Your output contract: schema, length budget, refusal rules.
Your measurement: pass rate, latency, retry cost.

Three things do not survive:

Prompt strings tuned to one model's quirks.
Provider SDK imports scattered across the repo.
Telemetry you can only read in one vendor's dashboard.

The artifact: a swap test you can run today

This is a single-file harness. It speaks the OpenAI-compatible chat shape. Point it at any endpoint that speaks the same shape. No SDK, no framework, no lock-in.

This is a runnable sketch, not a mature tool. Treat the checkers as stubs you replace with your own rules.

# swap_test.py — same cases, any OpenAI-compatible endpoint.
import json, os, sys, time, urllib.request

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]
MODEL = os.environ["LLM_MODEL"]

def call(prompt, temperature=0.0):
    payload = json.dumps({
        "model": MODEL,
        "temperature": temperature,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=payload,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=90) as r:
        body = json.loads(r.read())
    return body["choices"][0]["message"]["content"], round(time.time() - t0, 2)

# --- checkers: replace these three with your real rules ---
def exact_keys(out, keys):
    try:
        return set(json.loads(out)) == set(keys)
    except Exception:
        return False

def says_unknown(out):
    return "unknown" in out.lower() or "null" in out.lower()

def within_budget(out, max_lines):
    return len([l for l in out.splitlines() if l.strip()]) <= max_lines

CASES = [
    {"id": "schema", "prompt": "Return only JSON with keys id, owner, due.",
     "check": lambda o: exact_keys(o, ["id", "owner", "due"])},
    {"id": "no_guess", "prompt": "Ticket T-19 has no owner field. Report the owner or say unknown.",
     "check": says_unknown},
    {"id": "diff_budget", "prompt": "Rename getUser to fetchUser in 2 lines of diff. No commentary.",
     "check": lambda o: within_budget(o, 4)},
]

results = []
for c in CASES:
    out, secs = call(c["prompt"])
    ok = False
    try:
        ok = bool(c["check"](out))
    except Exception:
        ok = False
    results.append({"id": c["id"], "pass": ok, "secs": secs,
                    "chars": len(out), "output": out[:400]})

report = {"model": MODEL, "base": BASE, "results": results}
json.dump(report, sys.stdout, indent=2)

Run it once per candidate. Keep the JSON.

LLM_BASE_URL=https://your-endpoint/v1 \
LLM_API_KEY=... \
LLM_MODEL=your-candidate \
  python swap_test.py > run-a.json

Then diff two runs by case id, not by prose.

python - <<'PY'
import json
for name in ("run-a.json", "run-b.json"):
    r = json.load(open(name))
    row = {x["id"]: x["pass"] for x in r["results"]}
    print(name, row)
PY

That diff is your decision document. Prose reviews are not.

The failure signal you actually want

A useful candidate fails in boring ways. Watch for these three shapes:

Confident guessing. It fills a missing field instead of reporting unknown. This is the failure that reaches production.
Diff inflation. It passes correctness but writes 40 lines where you budgeted 4. Your review time pays that bill.
Silent narrowing. It drops a schema key you never asserted. Assert the key set, not the vibes.

Pass rate alone hides all three. That is why every case carries its own checker.

Where the free tier fits, and where it does not

MonkeyCode offers free model access and a free server option. Both are supplied by the project, not by me.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

My honest read: two uses are defensible. Three are not.

Good: the free model access is a candidate pool for swap runs. Cheap candidates mean you test more of them.
Good: the free server option is a disposable runner. Boot it, run the harness, throw the machine away.
Good: both together lower the cost of a first, honest evaluation.
Bad: production state on a free box. Treat it as ephemeral by default.
Bad: customer data on any free tier before you read the terms.
Bad: assuming quotas, limits, or availability are permanent. Check the project's current docs, not this post.

I am not claiming the allowance is unlimited or the server is production-grade. I am claiming the cost of the first evaluation should not be the thing that stops you.

Decision table

Signal from your run	Reading	Move
All cases pass, twice in a row	Stable enough to trial	Canary on low-risk tasks
Fails only the ambiguous case	Contract is underspecified	Fix your prompt, not the model
Passes but exceeds diff budget	Review tax incoming	Keep the current model
Passes at temperature 0 only	Fragile	Add a sampling run before trusting
Latency tail exceeds your wait budget	Workflow problem	Queue it, or drop it

Set the thresholds before you run. Post-hoc thresholds are negotiation, not measurement.

Who should not do this

Solo developers with one model and one workflow. The harness costs more than the swap saves.
Teams handling regulated data. Free tiers are the wrong container. Full stop.
Anyone who needs an SLA. Free access is a convenience, not a contract.
Teams that will not maintain the cases. A stale harness lies politely.

If you are in the first group, skip the code and keep the decision table.

The thirty-day version

Week one: write five cases with hard checkers. No prose judging.
Week two: run your current model as the baseline. Save the JSON.
Week three: run two cheaper candidates. Compare case by case.
Week four: add a weekly canary on the winner. Alert on any flipped case.

That canary is the real deliverable. It tells you when a model changes under you, quietly, on a Tuesday.

Close

Shopping for models feels like progress. Building the swap test is progress.

Do the boring thing. Write the checkers, keep the JSON, and let the diff decide.

If you want to try the workflow without touching a cloud bill, start with the free model access and run the harness on the free server option. Then throw the box away, because that was the point.

AI Fatigue Is Review Fatigue: Budget the Diff, Not the Model

Emery Chen — Sat, 17 Oct 2026 06:18:31 +0000

AI Fatigue Is Review Fatigue: Budget the Diff, Not the Model

The model is rarely your bottleneck. Your review queue is.

You switched models twice this quarter. Your review queue did not shrink.
That is a measurement gap, not a model gap.

So stop shopping for models. Start budgeting the diff.

Your fatigue is review fatigue

Faster generation raises output volume. Higher volume raises review cost.

Nothing in that loop improves when you upgrade a model. The queue just fills faster.

Most AI workflow debates argue about the wrong number. They compare tokens out per second, not review hours per merge.

That is why the fatigue persists. You optimized the input. You never measured the output.

Define the unit you actually pay for

Tokens are an input. Merged changes are the result.

The honest unit is tokens per merged change. It answers one question leaderboards cannot. Did this run pay for itself?

Two log files are enough:

calls.jsonl — one record per model call.
merges.jsonl — one record per merged pull request.

{"run_id":"r-1042","tier":"free","in":3120,"out":840}
{"run_id":"r-1042","merged_at":"2026-10-14T09:12:00Z","files":3,"added":48,"removed":11}

Join them with a small script:

# cost_per_merge.py — reference implementation, not benchmarked here
import json
from collections import defaultdict

def load(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]

spend = defaultdict(int)
for c in load("calls.jsonl"):
    spend[c["run_id"]] += c["in"] + c["out"]

merges = defaultdict(int)
for m in load("merges.jsonl"):
    merges[m["run_id"]] += 1

rows = []
for run_id, tokens in spend.items():
    if merges[run_id]:
        rows.append((tokens / merges[run_id], run_id, tokens))

rows.sort(reverse=True)
for per_merge, run_id, tokens in rows[:10]:
    print(f"{run_id}\t{tokens:>9} tok\t{per_merge:>9.0f} tok/merge")

total = sum(spend.values())
merged = sum(merges.values())
print(f"\nportfolio: {total} tok, {merged} merges, {total/max(merged,1):.0f} tok/merge")

Run it every Friday. Then read the top of the list.

A run that burns 400k tokens for one merge is your problem. The model version is not.

If the top row is a free-tier run, that is fine. Cheap tokens spent on abandoned work still cost you review time.

Gate the diff before you review it

The second half is a hard budget on what an agent may hand you.

# diff_budget.py — fail the build when the agent overreaches
import fnmatch, subprocess, sys

MAX_FILES = 6
MAX_LINES = 250
ALLOW = ["src/**", "tests/**", "docs/**"]
DENY = ["**/migrations/**", "**/*.lock", "infra/**"]

def changed(base="origin/main"):
    out = subprocess.run(
        ["git", "diff", "--numstat", f"{base}...HEAD"],
        capture_output=True, text=True, check=True).stdout
    for line in out.splitlines():
        added, removed, path = line.split("\t")
        yield path, int(added) + int(removed)

files = list(changed())
lines = sum(n for _, n in files)
bad = []

if len(files) > MAX_FILES:
    bad.append(f"files={len(files)} > {MAX_FILES}")
if lines > MAX_LINES:
    bad.append(f"lines={lines} > {MAX_LINES}")

for path, _ in files:
    if any(fnmatch.fnmatch(path, p) for p in DENY):
        bad.append(f"denied path: {path}")
    elif not any(fnmatch.fnmatch(path, p) for p in ALLOW):
        bad.append(f"outside allowlist: {path}")

if bad:
    print("\n".join(bad))
    sys.exit(1)

print(f"ok: {len(files)} files, {lines} lines")

Wire it in as one CI step:

python diff_budget.py || { echo "over budget; split the change"; exit 1; }

Why a budget beats a better model

A diff you can hold in your head gets reviewed honestly. A 900-line diff gets approved by fatigue.

Budgets are negotiable. Silent overruns are not. Raise MAX_LINES when the work justifies it, in the open.

Where a free allowance belongs in this loop

Here is the part that needs a caution label.

MonkeyCode offers free model access and a free server option. The operator also describes a free allowance of 10 million tokens; confirm the current figure in the product before you plan around it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use that capacity for work that is high volume and low stakes:

Classify incoming issues into two or three buckets.
Draft commit messages from the gate-approved diff.
Propose test names that a human then rejects most of.
Re-run a draft pass on a branch that will never merge.

This keeps paid spend on the merge path. It also keeps free spend measured, because calls.jsonl does not care which tier answered.

The limits are real. Shared capacity is shared. Do not put a latency-sensitive pipeline there. Do not run a forty-minute agent loop and call it production.

Who should not do this

Teams with no CI that can run one extra script.
Teams that cannot log model calls at all.
One-shot migrations where a review budget has no meaning.
Anyone who needs a vendor SLA on day one.
Anyone hoping the budget replaces a reviewer instead of protecting one.

The position

Free tokens are a subsidy, not an architecture. Review capacity is the constraint you cannot subsidize away.

So make the constraint visible. Log calls. Join them to merges. Gate the diff at six files and 250 lines.

Then argue about models, if you still have the energy. That argument is cheap. Review time is not.

If you want somewhere to run the high-volume, low-stakes half of this loop, the free tier and free server are worth a trial run. Bring your own logs and see what the week says.

Your Staging Box Is Too Precious to Trust

Emery Chen — Fri, 16 Oct 2026 05:19:13 +0000

The position

Keep every environment you can delete with one command.
A server you are afraid to nuke is already lying to you.
That is the whole argument, and most teams still lose it.

Here is the test I apply now.
Can a new hire rebuild this box from the repo in ten minutes?
If the answer is no, it is not staging. It is a pet with root access.

Long-lived staging boxes rot in four predictable ways

Hand-applied hotfixes exist only on that one machine.
Package versions drift away from the lock file in the repo.
Someone restores a snapshot and never records which one.
Ports, env vars, and certificates get edited by hand at 2am.

Each item breaks a different property you depend on.
All four break the same one: reproducibility.

The rehearsal pattern

The workflow has four steps and keeps no shared state.

Provision a fresh host with nothing on it.
Apply your schema dump, then your migration.
Run assertions that fail loudly on bad data.
Destroy the host and confirm nothing survived.

If step four leaves anything behind, the run did not count.
That single rule is the whole discipline, and it is cheap to enforce.

The script

Save this as rehearsal.sh and point it at a throwaway host.
Do not run it against anything you would mourn.

#!/usr/bin/env bash
set -euo pipefail

HOST="${1:?usage: rehearsal.sh user@host}"
DUMP="${DUMP:-./schema.sql}"
ASSERT="${ASSERT:-./assert.sql}"
DB='rehearsal-db'

teardown() {
  ssh "$HOST" "docker rm -f $DB" >/dev/null 2>&1 || true
}
trap teardown EXIT

ssh "$HOST" bash -s <<'REMOTE'
set -euo pipefail
command -v docker >/dev/null || { echo 'docker missing'; exit 3; }
docker rm -f rehearsal-db >/dev/null 2>&1 || true
docker run -d --name rehearsal-db \
  -e POSTGRES_PASSWORD=rehearsal \
  -p 127.0.0.1:55432:5432 postgres:16-alpine >/dev/null
REMOTE

for _ in $(seq 1 30); do
  ssh "$HOST" 'docker exec rehearsal-db pg_isready -U postgres' >/dev/null 2>&1 && break
  sleep 2
done

# Load the dump from a named snapshot, not from production.
ssh "$HOST" 'docker exec -i rehearsal-db psql -U postgres -d postgres -q' < "$DUMP"

# Run the migration, then the same assertions a second time.
# A clean migration is idempotent. A fragile one is not.
ssh "$HOST" 'docker exec -i rehearsal-db psql -U postgres -d postgres -q' < "$ASSERT"
ssh "$HOST" 'docker exec -i rehearsal-db psql -U postgres -d postgres -q' < "$ASSERT"

echo 'rehearsal passed'

The assertions

Put the invariants in one file so the exit code carries the verdict.

\set ON_ERROR_STOP on

DO $$
BEGIN
  IF EXISTS (SELECT 1 FROM users WHERE email IS NULL) THEN
    RAISE EXCEPTION 'null email survived the migration';
  END IF;

  IF (SELECT count(*) FROM orders) <> (SELECT count(*) FROM orders_archive) THEN
    RAISE EXCEPTION 'order rows did not land in the archive';
  END IF;
END $$;

Run the script twice in a row before you trust it.
A second identical run should change nothing and still pass.

Where a free host and free tokens actually fit

I use two cheap resources differently, and only inside this pattern.

The free server is the disposable target.
It runs the dump, the migration, and the assertions.
Then it gets erased, so losing it costs you nothing.

MonkeyCode is the project I have been using for this.
Its operator states that it is open source.
The operator also states a free model allowance and a free server option.
The figure quoted to me is 10 million tokens, but verify that yourself.
Quotas and terms move fast, and any number here can age in a week.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free model allowance writes the mechanical parts around the rehearsal.
In my workflow, it drafts a first version of the assertion file.
It also generates the down-migration when I only wrote the up.
The rehearsal host then decides whether any of it survives.

That division matters. The model proposes, the run judges.
If the run cannot reproduce it, the proposal does not ship.

Two hard limits I hit and you should expect.
Free hosts restart, change shape, or disappear without warning.
Never keep the only copy of anything on one.
Also, remote network latency distorts migration timings, so never benchmark there.

The decision table

Use this to stop arguing about which environment to borrow.

Work item	Long-lived staging	Disposable rehearsal host
Schema migration	Avoid	Use this
One-time data backfill	Avoid	Use this, then destroy
Latency or load benchmark	Wrong tool	Wrong tool
Client demo	Fine	Only if rebuilt daily
Secret rotation drill	Risky	Use this

Checklist before you merge

The rehearsal ran on a host created today.
The dump came from a named, restorable snapshot.
The assertions failed on a deliberately broken dump first.
The down-migration ran without errors.
The host was destroyed and recreated before the final run.

If you cannot tick all five, you have a demo, not a test.

Who should not use this

Teams holding regulated data they cannot legally copy to a free host.
Teams with no restorable snapshot to dump from.
Teams whose migration needs hours and a large machine.
Anyone who needs a demo environment that stays online unattended.

Free tiers are not a substitute for a platform contract.
They are a substitute for the staging box nobody wants to maintain.

The closing position

Deletability is a feature, not just a cost saving.
A box you can rebuild in minutes is a box you can finally believe.
If you keep one scratch host for exactly this kind of work, start there.
Break a repo on purpose, rebuild it twice, and read what your migrations never told you.

A Free Model Allowance Is Not an Architecture

Emery Chen — Thu, 15 Oct 2026 11:26:22 +0000

Free model access is genuinely useful. Building your stack around it is not.

Every free tier ends differently. Some shrink, some change shape, some just get slower. You cannot predict which one you get. You can make the switch boring.

That is the whole argument. Portability is the part of a free tier you keep. Everything else is rented.

The claim, stated plainly

A free token allowance is a resource, not a design decision. It pays for calls. It does not pay for the coupling you build on top.

Here is the test. If moving to a second endpoint takes a sprint, you never had a free tier. You had a discount with a lock.

I am not arguing against free tiers. I am arguing against spending them before you know your exit.

What coupling looks like in practice

Nobody couples on purpose. It arrives through small conveniences.

A vendor SDK type appears in your function signature.
Tool-call schemas get stored in your database, not converted at the edge.
Retry and streaming logic assumes one response envelope.
Prompt templates get tuned to one model's quirks and nothing else.
Your test suite is only ever green against a single host.

Each item saves an hour now. Each one costs a week later.

The artifact: one adapter, two modes

The fix is a single call shape with two implementations. One talks to the network. One reads from disk.

# gateway.py — sketch only; not yet executed against a live endpoint
from __future__ import annotations

import dataclasses
import hashlib
import json
import os
import pathlib
from typing import Protocol

CASSETTES = pathlib.Path(os.environ.get("CASSETTES", "cassettes"))


@dataclasses.dataclass(frozen=True)
class Call:
    prompt: str
    model: str
    temperature: float = 0.0


def fingerprint(call: Call) -> str:
    blob = json.dumps(dataclasses.asdict(call), sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]


class Provider(Protocol):
    def complete(self, call: Call) -> str: ...


class Replay:
    """Reads recorded responses. No network. No credentials."""

    def complete(self, call: Call) -> str:
        path = CASSETTES / f"{fingerprint(call)}.json"
        if not path.exists():
            raise FileNotFoundError(f"missing cassette: {path.name}")
        return json.loads(path.read_text())["text"]


class Http:
    """One POST, plain JSON, no vendor SDK in your type signatures."""

    def __init__(self, base_url: str, api_key: str) -> None:
        self.base_url = base_url.rstrip("/")
        self.api_key = api_key

    def complete(self, call: Call) -> str:
        import urllib.request

        body = json.dumps(dataclasses.asdict(call)).encode()
        req = urllib.request.Request(
            f"{self.base_url}/v1/complete",
            data=body,
            headers={
                "Content-Type": "application/json",
                "Authorization": f"Bearer {self.api_key}",
            },
        )
        with urllib.request.urlopen(req, timeout=60) as resp:
            return json.loads(resp.read())["text"]

Selection stays in one place. That place is a function you can argue about in review.

def build_provider(mode: str) -> Provider:
    if mode == "replay":
        return Replay()
    if mode == "live":
        return Http(os.environ["BASE_URL"], os.environ["API_KEY"])
    raise SystemExit(f"unknown mode: {mode}")

Two tests keep the design honest. Both run offline and cost nothing.

# test_portability.py
import pytest
from gateway import Call, Replay, fingerprint


def test_fingerprint_is_stable():
    a = Call(prompt="summarise README", model="any")
    b = Call(prompt="summarise README", model="any")
    assert fingerprint(a) == fingerprint(b)


def test_replay_never_touches_network():
    with pytest.raises(FileNotFoundError):
        Replay().complete(Call(prompt="never recorded", model="any"))

The second test is the important one. It fails loudly when someone sneaks a network call into a replay path.

The drill: prove you can switch in under an hour

Run this once per quarter. Write the result in your repository README.

Record ten real prompts against your primary endpoint. Store cassettes in git.
Run the suite in replay mode with networking disabled. Everything must pass.
Point the same suite at a second endpoint. Change no test files.
Diff the outputs into a report. Report differences; do not score them yet.
Count the lines you changed to switch. That number is your coupling score.
Time the whole exercise. Anything over an hour is a backlog item.

pip install -e .
CASSETTES=cassettes python -m gateway --record --suite suites/core.yaml
CASSETTES=cassettes python -m gateway --replay --suite suites/core.yaml

One honest note. The --record and --replay flags are the interface I would want, not a CLI I have run. Wire them to your own runner.

A decision table you can argue with

Situation	Free allowance	Free server option	Paid endpoint
Learning an agent loop	Yes	Yes	No
Recording replay cassettes	Yes	Yes	Only if blocked
Per-commit CI runs	Maybe	Yes, as your runner	Depends on budget
Latency-sensitive traffic	No	No	Yes
Regulated data	No	No	Approved vendors only

Read the table as a default, not a law. The point is that "free" appears in one column, never in your architecture.

Where MonkeyCode fits

MonkeyCode is an open-source project offering free model access and a free server option. It is a good second endpoint for the drill, because a second endpoint is exactly what the drill needs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not quoting a quota or a server spec here. Those numbers change, and an article is a bad place to pin them. Check the project page for current terms before you plan around them.

The genuine fit is narrow and useful. Keep your primary endpoint where it is. Point the adapter at MonkeyCode, run the replay suite, and measure the diff and the lines changed. If the switch is boring, the free tier did its job.

The free server option covers a different gap. Run the recording proxy and the cassette store there, so the drill costs you nothing while you decide.

Limitations

Replay proves reproducibility, not quality. A cassette says what happened, not what is correct.
Cassettes bind to exact prompts. Edit a prompt and you re-record.
A second endpoint passing your suite does not mean the endpoints are equivalent.
Free infrastructure is not a benchmark target. Do not publish latency numbers from it.
This drill detects coupling. It does not remove coupling you already shipped.

Who should not use this

Skip the drill if switching providers is contractually prohibited for you. Skip it if you have a single endpoint and no plausible reason to move. Skip it if your prompts are already versioned and tested, because you have solved this problem.

Also skip it if you want a free tier to be a permanent plan. Free tiers are not permanent. That is the entire reason this article exists.

The close

Write the exit plan first, then spend the allowance. Point the adapter at the free endpoint and run the drill once. If the switch takes an afternoon, you learned something cheap.

If the Agent Agrees First, Discard the Review

Emery Chen — Tue, 13 Oct 2026 01:58:38 +0000

If the Agent Agrees First, Discard the Review

A first-pass agreement is a failed agent review.
You heard your own premise read back to you.
Block the merge until a scored dissent log exists.

The claim I want you to hold

A coding agent often agrees because agreement is cheap.
Praise plus one tiny nit is not a review.
That pattern should stop the diff, not bless it.

This piece takes a hard line on purpose.
First-pass agreement is not evidence of design correctness.
Only objections you can falsify may move the patch.

You remain the reviewer of record on this diff.
The agent is a draft critic with a quota of doubt.
If it will not doubt, you discard the run.

Cheap runs invite lazy reviews

Free model access makes another chat feel free.
A free server option makes another run feel free.
Cheap runs are exactly where lazy reviews hide.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode matters here only as a host for those two options.
Spend free model access on a dissent pass, not on applause.

Use the free server option as a pinned, disposable review box.
Availability alone is not a quality claim here.
This article invents no token quota, model name, or hardware spec.

Pin the host and model your operator actually turned on.
Those facts change, and this gate does not need them.
If a dashboard shows a number, record that number yourself.

Do not cite a figure you did not read today.
A screenshot from last month is not a current fact.
Read the live terms, then write down only what you saw.

What a dissent pass must contain

You want objections that can still be wrong.
Each objection needs a file, a claim, and a falsifier.
Drop any item that misses one of those parts.

Require these four outputs before you read tone:

Write one sentence that restates the design premise.
Collect at least three objections that could still fail.
Attach one falsifier command or assertion to each objection.
Stop the run when the agent only agrees with you.

If the agent cannot attack the design, stop the thread.
Do not beg the same chat to try harder now.
Open a clean run with the same frozen contract.

Freeze the contract before you spend a token

You write the contract, and the agent does not edit it.
Store that contract beside the diff, under version control.
Paste the diff and the contract, not the answer key.

# dissent-contract.yaml
# Proposal you can run. Not a measured benchmark.
review_id: "REV-014"
premise: "Replace sync file reads with a bounded async pool."
forbidden:
  - "Do not praise the design."
  - "Do not restate the premise as an objection."
  - "Do not invent files you did not receive."
required_objections: 3
fields:
  - id
  - file
  - claim
  - falsifier
  - severity  # low | med | high
stop_if:
  - "objection_count < 3"
  - "any_claim_missing_file"
  - "any_falsifier_is_vague"

The forbidden list is the point of the exercise.
You are paying for friction, not for a friendlier summary.
A review that cannot violate your idea is not a review.

Gate the log before you trust the prose

This script never calls a model at all.
It only accepts or rejects a dissent log on stdin.
Run the gate locally before you discuss any objection.

#!/usr/bin/env python3
"""Local dissent gate. A proposal until you execute it."""
import json
import sys

REQUIRED = {"id", "file", "claim", "falsifier", "severity"}
SEVERITY = {"low", "med", "high"}

def main() -> None:
    doc = json.load(sys.stdin)
    items = doc.get("objections")
    errors = []
    if not isinstance(items, list) or len(items) < 3:
        errors.append("need at least three objections")
        items = items if isinstance(items, list) else []
    premise = str(doc.get("premise", "")).strip().lower()
    seen = set()
    for item in items:
        if not isinstance(item, dict) or not REQUIRED.issubset(item):
            errors.append("missing required field")
            continue
        if item["severity"] not in SEVERITY:
            errors.append("bad severity")
        claim = str(item["claim"]).strip()
        falsifier = str(item["falsifier"]).strip()
        if len(claim) < 12 or len(falsifier) < 12:
            errors.append("claim or falsifier too thin")
        if claim.lower() == premise:
            errors.append("objection restates the premise")
        key = (item.get("file"), claim)
        if key in seen:
            errors.append("duplicate objection")
        seen.add(key)
    if errors:
        print("REJECT: " + "; ".join(errors))
        sys.exit(1)
    print("ACCEPT: dissent log has structure")

if __name__ == "__main__":
    main()

Feed the gate a JSON log and read the exit code.

python3 dissent_gate.py < dissent-log.json
echo "gate_exit=$?"

Exit zero means the log has the required shape.
That exit code does not mean any objection is true.
You still open each named file and try the falsifier.

Keep a failing fixture next to the gate

You should prove the gate rejects a mirror before you trust it.
Save a bad log that only restates the premise.
Expect a non-zero exit, and fail the setup if it passes.

{
  "premise": "Replace sync file reads with a bounded async pool.",
  "objections": [
    {
      "id": "O1",
      "file": "src/pool.py",
      "claim": "Replace sync file reads with a bounded async pool.",
      "falsifier": "No check supplied for this restatement.",
      "severity": "low"
    }
  ]
}

That fixture should die on count and on the mirrored claim.
A good log needs three distinct attacks, each with a real check.
Do not weaken the gate so a friendly log can sneak through.

Score every row yourself

Use this table after the gate accepts the log.
You fill the last column, and the agent does not.
A fluent sentence with no file is still a fail.

Check	Pass looks like	Fail looks like
File exists	Path is in the diff	Path was invented
Claim is specific	Names a behavior	Vague style worry
Falsifier runs	Command or assertion	Consider edge cases
Not a mirror	Attacks the premise	Repeats your idea
Severity fits	High means loss or auth	High means formatting

A low severity with a real falsifier can still block you.
A high severity with no file cannot block anyone.
Severity is only a hint you may override in writing.

Walk one diff without fake metrics

Consider a small change around local file reads.
The patch wraps sync reads in a bounded async pool.
A weak review says the pool looks cleaner, then stops.

Discard that review, because it agreed with the premise.
A usable objection says the pool hides backpressure from callers.
The falsifier is a test with more waiters than slots.

A second objection says cancellation leaks an open handle.
The falsifier is a timeout that must close the file.
If you cannot build that check, the objection stays open.

A third objection says a retry double-reads a partial file.
The falsifier is a crash between the read and the checksum.
If the diff has no checksum, the objection still stands.

These lines show a shape, not a measured lab result.
This article did not time them or score a live model.
Your repository has to supply the real file paths.

How you post the result

Do not paste the agent praise into the pull request.
Post the three objections, the falsifiers, and your own scores.
A teammate should see your verdict, not the model's mood.

Use a short comment shape, and keep it boring.

State the premise in one frozen sentence at the top.
List each objection id beside the file you opened.
Mark each falsifier as reproduced, blocked, or dropped.
End with merge, revise, or discard, and skip extra adjectives.

If every row is dropped, you still lack a real proof.
You have an empty critic, not a quiet approval.
Silence from the model is not a quiet yes.

Put the run in quarantine

Run the dissent pass on the free server option.
Keep that host pinned to a digest you already checked.
Do not review code on a host you cannot name.

Treat the server as a quarantine for the review transcript.
Do not let that host apply the patch to your tree.
Copy the dissent log out, then end the review run.

If the host digest is missing, spend no token yet.
An unpinned host is a separate failure from a soft review.
Fix the pin first, then ask the model for dissent.

Who should not use this gate

Skip this workflow when a human already owns the review.
A second person beats a forced list of objections.
Use the agent only when the human queue is the bottleneck.

Also skip the gate in the four cases below.

You need a formal security audit, and this gate is not one.
The change is a comment-only typo, so dissent is waste.
You cannot read the named files, so you cannot score them.
The diff requires secrets, so do not paste those secrets.

Skip the gate if policy forbids sending source to a hosted model.
A free server option does not erase your data rules.
Read the current terms before the first token leaves your laptop.

Limits you should keep in the README

The local gate checks structure, not the truth of claims.
A model can still emit three fluent and false objections.
Your scoring table is the only truth step, and it is manual.

Free access can change or vanish without any promise here.
Do not make a release depend on a permanent free tier.
Keep the contract and the gate in git either way.

The length check is a crude proxy for thin text.
A short and sharp claim can still be a valid objection.
Allow a written override, and record who granted that override.

This article does not report a live product benchmark.
Treat the YAML and the Python as a proposal you execute.
Save your own exit codes before you teach the gate.

Hold the line

Do not merge on first-pass agreement from a coding agent.
Demand a dissent log with files, claims, and falsifiers.
Score that log yourself, then decide with the diff open.

If you need a disposable host for this dissent pass, try MonkeyCode.
Its free model access and free server option fit this quarantine use.
Keep the gate in your repo and the verdict in your hands.

A Free Token Does Not Buy a Blind Retry

Emery Chen — Sun, 11 Oct 2026 21:47:38 +0000

A free retry is still a decision you own.
Cheap tokens do not erase a repeated mistake.
You need a stop rule before the first call.

The claim you can test

Do not retry an agent only because the call is free.
A free allowance changes the bill, not the evidence.
Evidence is the only reason a second call exists.

This piece argues from failure classes, not from scores.
It does not rank models, vendors, or hosted runners.
You can reject the opinion after you count your own logs.

Why retry talk is the wrong comfort

Public posts this week keep circling retries and swaps.
Treat those titles as untrusted signals, not as facts.
No headline here is used as a measurement.

Better inference does not dissolve a local lock.
Missing files and wrong paths stay on your machine.
A remote model cannot invent a file you never sent.

What free access tempts you to skip

A paid meter used to force a short pause.
You read the error before you spent again.
A free pool removes that pause unless you rebuild it.

That pause was doing classification work for you.
It made you name the failure in plain words.
Without a name, you resend the same prompt again.

Excuses that should not spend a call

Reject a retry that rests on these excuses.
Write the excuse down, then throw the call away.

The last reply looked almost right to you.
The log was long, so another summary feels due.
The allowance is large, so waste feels harmless.

Each excuse avoids naming even one new fact.
A retry without a new fact is noise.
Noise gets cheaper when the meter reads zero.

Classify the failure before the network

Read the failure on your own machine first.
Put it in one class from the table.
Only then decide whether the network is owed.

Rows you can apply today

Class	What you saw	Retry	Why
Local lock	Dependency or file lock	No	Another call will not release it
Missing input	Empty diff or absent file	No	Fix the fixture, then call once
Tool mismatch	Command is not on PATH	No	Repair the host before any rephrase
Flaky check	Local rerun already passed	Once	Record the flake and cap it
Wrong file	Edit landed outside the brief	No	Narrow the path list first
New compiler fact	Fresh error after a real edit	Yes	The error text is new evidence

Copy this table as a starting proposal only.
It is not a recorded result from production traffic.
Change every row until it matches your repo.

Keep a referee next to the table

Keep the referee on the machine you trust.
Keep the classifier in code you can diff.
The script below is an unexecuted local example.

#!/usr/bin/env python3
"""Local retry referee. Proposal only. Not a recorded run."""

from dataclasses import dataclass

STOP = "stop"
ONCE = "once"
ALLOW = "allow"

@dataclass(frozen=True)
class Verdict:
    action: str
    reason: str

def classify(kind: str, evidence_changed: bool, attempts: int) -> Verdict:
    local = {"lock", "missing_input", "tool_mismatch", "wrong_file"}
    if kind in local:
        return Verdict(STOP, "local fact is unchanged")
    if kind == "flake" and attempts < 1:
        return Verdict(ONCE, "one local rerun, then stop")
    if kind == "new_compiler_fact" and evidence_changed and attempts < 2:
        return Verdict(ALLOW, "new error text justifies one more call")
    return Verdict(STOP, "no new evidence or cap reached")

if __name__ == "__main__":
    samples = [
        ("lock", False, 0),
        ("missing_input", False, 0),
        ("flake", False, 0),
        ("flake", False, 1),
        ("new_compiler_fact", True, 0),
        ("new_compiler_fact", True, 2),
        ("wrong_file", True, 0),
        ("unknown", True, 0),
    ]
    for kind, changed, attempts in samples:
        verdict = classify(kind, changed, attempts)
        print(f"{kind} changed={changed} attempts={attempts} -> {verdict.action}: {verdict.reason}")

Save the file as retry_referee.py beside your repo.
Run the samples before you trust the gate.

python3 retry_referee.py

What those samples should show

Expect stop on locks, missing files, and wrong paths.
Expect one local chance when the class is flake.
Expect allow only when compiler text actually changed.

If the kind is unknown, the referee must stop.
An unknown kind is not a prompt to invent a class.
You add a row, then you run the samples again.

Put the referee in front of the call

Wrap the referee in a thin shell gate.
The gate reads a class you already wrote down.
It refuses the network when the verdict is stop.

#!/usr/bin/env bash
set -euo pipefail

kind="${1:-unknown}"
changed="${2:-false}"
attempts="${3:-0}"

action="$(python3 -c '
import sys
from retry_referee import classify
kind, changed, attempts = sys.argv[1:]
verdict = classify(kind, changed == "true", int(attempts))
print(verdict.action)
' "$kind" "$changed" "$attempts")"

if [[ "$action" != "allow" && "$action" != "once" ]]; then
  echo "refused: ${kind} has no new evidence"
  exit 2
fi

echo "permitted: ${action} for ${kind}"
# Place one real model call below this line.
# Do not hide a retry loop inside the client.

Save that file as retry_gate.sh next to the referee.
Try the gate with a lock before you try a model.

bash retry_gate.sh lock false 0
bash retry_gate.sh new_compiler_fact true 0

Run both commands from the directory that holds the scripts.
The first command should exit with status two.
The second should print a permit and then wait.

This gate is a sketch, not a finished product.
Label it unexecuted until it runs on your tree.
A printed permit is not a passing integration test.

Spend free access only after allow

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project for coding agents.
The outreach brief states free model access for readers.
It also states a free server option you can try.

The outreach brief states a free allowance of ten million tokens.
Confirm the live figure in the current project docs.
Do not budget a quarter from a copied number.

Free access can change without this article updating.
Model names, hardware, and duration are not assumed.
Permanence is not a feature you should plan on.

Use the free model only after the referee allows it.
Use the free server to host the agent loop.
Do not skip local classification because the host is free.

A practical order keeps the pool for new evidence.
Write the class down before you open a remote session.

Classify the failure on your own machine first.
Refuse the call when the class is local.
Send one bounded prompt only when evidence changed.
Re-run the same local checks after the patch.
Stop at two remote attempts even if tokens remain.

Two remote attempts is a hard cap in this opinion.
Leftover tokens are not an argument for a third call.
Stop even when the allowance still looks large.

Shrink the prompt the referee allows

When the referee allows a call, shrink the prompt.
Send the new error, the paths, and the last diff.
Do not resend the repository because the call is free.

Failure class: new_compiler_fact
Attempts so far: 1
Changed evidence: yes
Touched paths:
- src/retry_gate.py
New compiler text:
- src/retry_gate.py:40: NameError: verdict
Constraint:
- Edit only the touched paths.
- Do not add a retry loop.
- Stop after one patch.

If you cannot point at a new line, do not send it.
Abundance is not the same thing as context.
Context means the fact that changed since last time.

The opinion, without a hedge

Cheap inference rewards impatience more than careful work.
You should grow stricter as the meter disappears.
The stop rule matters most when retries feel free.

A free server does not outrank your worktree.
A large token pool does not outrank a missing file.
Another paraphrase does not outrank a held lock.

If you disagree, sample five failures from last week.
Count how many retries carried no new evidence.
Let that count accept or reject this opinion.

Limits that stay attached to the rule

This workflow does not measure model quality at all.
It does not compare hosts, prices, or latency.
It does not prove a free tier will remain available.

The script knows only the classes you pass in.
A wrong class produces a confident wrong stop.
You still have to read the log yourself.

Network flakes and partial writes need their own rows.
Add those rows before a shared branch uses the gate.
Never paste secrets or customer logs into the sample.

This draft was not executed against a live hosted agent.
Treat every command here as a local proposal.
Verify install steps and terms in the project docs.

A free server still needs your auth and your retention rules.
This draft does not describe those controls for you.
Read them before any source leaves your laptop.

Who should leave this workflow alone

Skip this when the edit is a typo you already see.
A referee is wasted on a fix you can type.
Human eyes are enough for that narrow case.

Skip this if you will not maintain the class list.
An abandoned table becomes theater within a week.
An unused table will not protect a shared branch.

Skip this if policy forbids a third-party agent server.
A free option is still a remote service.
Read the terms before you send repository source.

Skip this if you need audited numbers for a decision.
This article offers a rule, not a study.
Do not cite it as a performance result.

One check before you try the pool

Label five recent failures with the table above.
Run the referee against those labels before any call.
Call a free model only if the verdict is allow.

If current project terms still match the brief, use them.
If they do not, keep the referee and skip the host.
The stop rule stays on your machine either way.

Do Not Let the Free Model Grade Its Own Work

Emery Chen — Sat, 10 Oct 2026 18:27:15 +0000

A free model call is a sample, not a verdict.
You should freeze every fail rule before that call.
If the writer model grades itself, you learned nothing.

The position

Free access does not make a weak check valid.
It only makes that weak check cheaper to repeat.
You still need a grader that you wrote yourself.

Keep a hard split between the writer and the judge.
One model may draft the change you asked for.
A separate program must accept or reject that change.

Headlines are a bad oracle

You may see a small model post a clean run.
People then stop looking at the judge behind it.
That is the wrong lesson from a cheap success.

Public model names shift every few product weeks.
Prices and limits shift along with those names.

Your review should not depend on today's brand.
Pin the rubric instead of the weekly headline.

No fresh benchmark table was supplied for this draft.
Treat every quota line and speed line as unverified.

Read the vendor page on the same day you run.
Do not paste old numbers into a release note.

What a free stack may do

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The operator states two availability facts, and nothing more.
MonkeyCode offers free model access for you to try.

MonkeyCode also offers a free server option to try.
This draft invents no quota, hardware, or duration.

Use that free model as a writer, and nothing more.
Use that free server as a clean process boundary.

Keep the grader on a machine that you control.
Do not ask either service to define success for you.

If either offer disappears, this method still stands.
The files below do not require that specific vendor.
Swap the writer endpoint and keep the same judge.

Confirm the project license on the official page yourself.
Do not vendor a client from a memory of last month.

Freeze three files before any token

You need three local files before any network call.
Write each file by hand, then hash the rubric.

The writer model may read those frozen files.
The writer model must not edit the fail rules.

task.md holds the job in plain, testable language.
rubric.json holds fail rules with stable public ids.
fixture/ holds inputs the writer is forbidden to replace.

Record those hashes in freeze.txt before you call out.
If a later hash differs, you throw that run away.
A rubric edited after the call is a new experiment.

Put the verdict in code

A prompt that says be strict is not a verdict.
The model can agree with that line and still drift.
An exit code cannot flatter you and then drift.

Field contract

You should require three fields, and ignore extra keys.
files_touched must be an integer count of edited paths.
stdout must be captured text, not a model essay.

paths must be the list the prefix rule reads.
If a required field is missing, exit non-zero at once.
A partial JSON object is a failed contract, not a pass.

Proposal grader, not a logged run

The script below is a proposal, not an executed log.
Nothing here was run against a live vendor endpoint.
Copy it, then point it at your own fixture tree.

Expect failures until your paths match this layout.
This local proposal targets Python 3.10 or newer.

#!/usr/bin/env python3
"""Local grader. Proposal only. Does not call a network."""

import hashlib
import json
import sys
from pathlib import Path

ROOT = Path(".").resolve()
RUBRIC = ROOT / "rubric.json"
FREEZE = ROOT / "freeze.txt"
OUTPUT = ROOT / "out" / "result.json"
REQUIRED = ("files_touched", "stdout", "paths")


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def load_freeze() -> dict:
    pairs = {}
    for line in FREEZE.read_text(encoding="utf-8").splitlines():
        if not line.strip():
            continue
        name, digest = line.split()
        pairs[name] = digest
    return pairs


def assert_frozen() -> None:
    frozen = load_freeze()
    current = sha256(RUBRIC)
    expected = frozen.get("rubric.json")
    if current != expected:
        sys.exit("rubric changed after freeze; discard this run")


def missing_keys(payload: dict) -> list:
    return [key for key in REQUIRED if key not in payload]


def check(rule: dict, payload: dict) -> str | None:
    kind = rule["kind"]
    if kind == "max_files":
        if int(payload["files_touched"]) > int(rule["limit"]):
            return rule["id"]
    elif kind == "must_include":
        if rule["token"] not in payload["stdout"]:
            return rule["id"]
    elif kind == "forbidden_path":
        for path in payload["paths"]:
            if str(path).startswith(rule["prefix"]):
                return rule["id"]
    else:
        sys.exit(f"unknown rule kind: {kind}")
    return None


def main() -> int:
    assert_frozen()
    rubric = json.loads(RUBRIC.read_text(encoding="utf-8"))
    payload = json.loads(OUTPUT.read_text(encoding="utf-8"))
    absent = missing_keys(payload)
    if absent:
        print(json.dumps({"missing": absent}))
        return 1
    failed = []
    for rule in rubric["rules"]:
        hit = check(rule, payload)
        if hit:
            failed.append(hit)
    report = {"failed": failed, "rule_count": len(rubric["rules"])}
    print(json.dumps(report, indent=2))
    return 1 if failed else 0


if __name__ == "__main__":
    raise SystemExit(main())

A matching rubric can stay this small on purpose.
Three rules are enough to prove the judge can fail.

{
  "rules": [
    {"id": "F1", "kind": "max_files", "limit": 3},
    {"id": "F2", "kind": "must_include", "token": "EXIT=0"},
    {"id": "F3", "kind": "forbidden_path", "prefix": "secrets/"}
  ]
}

Commands that enforce the order

Run the hash step before any writer process starts.
Do not skip that hash, even on a tiny fixture.
Do not let the writer process touch rubric.json at all.

mkdir -p out fixture
python3 - <<'PY'
import hashlib
import pathlib
path = pathlib.Path("rubric.json")
digest = hashlib.sha256(path.read_bytes()).hexdigest()
out = pathlib.Path("freeze.txt")
out.write_text("rubric.json %s\n" % digest, encoding="utf-8")
PY
# Writer step stays outside this page.
# Save structured writer output to out/result.json.
python3 grade.py
echo "grader_exit=$?"

That shell comment is deliberate, not an omission.
This page does not ship a live vendor client.
You already know your endpoint, or you do not.

Add a client only after the grader passes a fake file.
A fake file is a hand-built result, not a model reply.

Known pass, then known fail

Build out/result.json by hand before any model call.
You want a known pass, then a known fail.
That pair proves the judge can say both words.

{
  "files_touched": 2,
  "stdout": "built fixture\nEXIT=0\n",
  "paths": ["src/app.py", "fixture/input.txt"]
}

Change files_touched to 9 and run the grader again.
You should see rule F1 and a non-zero exit.
If that edit still exits zero, your judge is blind.

Point one path at secrets/keys and expect rule F3.
A missing fail on that path means the prefix check is dead.

Two fixtures, one judge

You do not need another model to challenge the writer.
You need two frozen inputs and one local diff.
If the outputs disagree, keep both and merge neither.

Run the writer on fixture A and save the output.
Run the writer on fixture B with the same rubric.
Grade each output file with that same frozen rubric.
Diff the two grader reports, not the two chat stories.

A disagreement is a result, not a reason to relax rules.
You inspect both diffs, then you pick neither automatically.

How you should read the exit

A clean exit only clears your frozen rule list.
It does not close design review or threat review.
You still read every changed path with your own eyes.

Result you see	Move you make	Reason to move
Exit 0 and the freeze still matches	Keep the diff for human review	The sample met rules you wrote
Exit 1 with known rule ids	Repair the diff and leave the rules	The writer missed a frozen check
Freeze hash does not match the file	Discard the run and start again	The experiment changed mid-flight
`out/result.json` is missing entirely	Fix the writer output contract	Silence must never count as a pass
Grader prints an unknown rule kind	Stop and repair the rubric first	A vague judge is not a judge

Refuse the self-graded demo

You should refuse any demo that grades itself.
A self-score from the writer is only a story.
Those stories can stay inside a chat window.

They are not evidence inside a pull request.
Free tokens make that story easier to produce.
That ease is the danger, not the gift itself.

Spend the free writer on alternate drafts only.
Spend your own code on the verdict every time.

A free server helps only as a process boundary.
It can run the writer away from your laptop.
It should not hold the only copy of the rubric.

If you cannot re-run the grader offline, you stop.
Offline means the rubric and the grader stay local.
A server copy alone is not a review record.

What this check will not catch

This grader checks output shape, not writer intent.
A bad rule will happily pass a bad patch.

You can freeze a mistake just as easily as a truth.
Review the rubric text before you trust any exit.

Free access can change without notice or your consent.
Do not plan a release around an unnamed quota.

Do not put production secrets inside the fixture tree.
Do not send customer rows to a trial host.

The script ignores concurrency, retries, and clock skew.
It will not catch a writer that flakes between calls.

Run the same frozen input two times in a row.
If those two outputs differ, you have no result yet.

Who should not use this

Skip this flow if you need a contracted uptime promise.
Skip it when data cannot leave your private network.
Skip it when nobody on the team can read the rubric.
Skip it when you wanted a score, not a fail list.

Also skip it for medical, legal, or payment logic.
Those domains need stronger oracles than token rules.

A string check will not save a bad payment path.
Build a real suite before you spend any model call.

One next step, then stop

Read the current MonkeyCode access terms yourself today.
Confirm the free model offer and the free server offer.
Then run this grader against a fake result file first.
Spend a writer call only after that local pass holds.

Spend the Free Allowance on Hostile Fixtures

Emery Chen — Fri, 09 Oct 2026 14:53:06 +0000

Spend the free allowance on hostile fixtures only. Do not spend it on flattering model summaries. A summary that praises your diff is cheap comfort.

A fixture that breaks your spec is the actual work. That is the position, not a soft suggestion. If your free-tier log is mostly recap text, you wasted the pool.

Say the rule before the first call

You get a free model path and a free server. You also get a strong urge to request a nice summary. Refuse that urge before you write the first prompt.

Ask for a contradiction inside the spec instead. Ask for a missing field or a broken boundary. Then store the failure beside the claim you named.

A kind paragraph does not stress any contract. A hostile fixture does stress the contract directly. You should point at the broken case during review.

Why summaries feel like progress

Summaries are fast, and they read like finished judgment. You paste them into the pull request body. Reviewers see fluent prose and relax their questions.

The spec never met a single counterexample there. Free access makes this failure cheaper to repeat often. There is no invoice to embarrass the tenth recap.

You can generate twenty summaries and learn nothing new. That pattern is decoration, not real technical exploration. Hostile fixtures burn tokens because each call falsifies one claim.

If the claim survives, you learned one small thing. If the claim fails, you learned something you can file. Either result beats another flattering recap of the diff.

What hostile means here

Hostile does not mean illegal, abusive, or aimed outward. It means unfriendly to your own written specification. You are attacking the contract, not a person or company.

Use three fixture families and keep each family boring. Boring fixtures are easier to rerun after a model move. Clever prompts are harder to defend in review later.

Three families, nothing cleverer

Contradiction means two rules in the spec cannot both hold.
Omission means a required field, state, or error path is missing.
Boundary means an empty input, duplicate event, or hard limit appears.

Do not ask the model to invent crimes or exploit steps. Stay inside your repository and your documented rules only. If a fixture needs a real secret, stop and use a fake.

A harness you can copy

The script below is an unexecuted local example only. It does not call a model or open a socket. It checks that your log spent calls on hostile classes.

You run it locally before anyone cites the session. Adapt the filename if your repo uses another path. Do not treat a green exit as proof of product truth.

#!/usr/bin/env python3
"""Reject a free-tier log that is mostly summaries. Unexecuted example."""
import json
import sys
from collections import Counter
from pathlib import Path

ALLOWED = {"contradiction", "omission", "boundary"}
BANNED = {"summary", "praise", "rewrite"}

def load(path: Path):
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        if line.strip():
            rows.append(json.loads(line))
    return rows

def main() -> int:
    if len(sys.argv) != 2:
        print("usage: hostile_gate.py calls.jsonl")
        return 2
    rows = load(Path(sys.argv[1]))
    if len(rows) < 3:
        print("FAIL: need at least three hostile attempts")
        return 1
    kinds = Counter()
    for i, row in enumerate(rows, start=1):
        kind = str(row.get("fixture", "")).strip().lower()
        claim = str(row.get("claim_id", "")).strip()
        if not claim:
            print(f"FAIL: row {i} has no claim_id")
            return 1
        if kind in BANNED:
            print(f"FAIL: row {i} is a {kind}, not a fixture")
            return 1
        if kind not in ALLOWED:
            print(f"FAIL: row {i} fixture {kind!r} is not allowed")
            return 1
        kinds[kind] += 1
    missing = ALLOWED - set(kinds)
    if missing:
        print("FAIL: missing families: " + ", ".join(sorted(missing)))
        return 1
    print("OK " + " ".join(f"{k}={kinds[k]}" for k in sorted(kinds)))
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

python3 hostile_gate.py calls.jsonl
echo "exit=$?"

A synthetic log you can run

Create a synthetic log before you point the gate at real calls. These three lines are fake fixtures, not live model output. They exist so you can see a passing exit on your machine.

mkdir -p spec-lab && cd spec-lab
cat > calls.jsonl <<'EOF'
{"claim_id":"retry-once","fixture":"contradiction","input":"spec says at-least-once and exactly-once","survived":false,"note":"both sentences are present"}
{"claim_id":"retry-once","fixture":"omission","input":"no rule for a null receipt","survived":false,"note":"null path is unspecified"}
{"claim_id":"retry-once","fixture":"boundary","input":"duplicate delivery","survived":false,"note":"handler applied the side effect twice"}
EOF
python3 hostile_gate.py calls.jsonl

What a passing log must contain

Three logged calls are a minimum, not a quality score. You still need one contradiction, one omission, and one boundary. A log of fifty summaries fails this gate immediately.

That immediate failure is the feature you wanted. It stops you from mistaking volume for coverage. Add more fixtures after the three families already exist.

The line format

Keep each attempt on one single JSON line. Name the claim you tried to break in that call. Name the fixture family and whether the spec survived.

{"claim_id":"retry-once","fixture":"boundary","input":"duplicate delivery","survived":false,"note":"handler applied the side effect twice"}

{"claim_id":"retry-once","fixture":"contradiction","input":"spec says at-least-once and exactly-once","survived":false,"note":"both sentences are present"}

{"claim_id":"retry-once","fixture":"omission","input":"no rule for a null receipt","survived":false,"note":"null path is unspecified"}

Quote the spec line in the note when you need a pointer. Do not store customer payloads or live credentials there. Do not store prompts copied from other people's private chats.

Check the file before you cite it

Check that every line parses before you trust the mix. Then run the gate and keep the printed exit code. Neither command proves your product behavior is correct.

They prove you spent the session trying to break a claim. That is a lower bar than truth, and it is still useful. A session without that bar is just generated prose.

python3 -c 'import json,sys; [json.loads(line) for line in sys.stdin if line.strip()]' < calls.jsonl
python3 hostile_gate.py calls.jsonl
echo "exit=$?"

Decision table for the allowance

Use this table before you send another free call. If the row says stop, you stop without bargaining. Do not bargain with a fluent model answer.

You are about to ask for	Spend a free call?	Why
A short summary of the diff	No	Decoration, not a fixture
Praise for the approach	No	Flattery hides missing cases
A contradiction inside the spec	Yes	One family you must cover
A missing error path	Yes	Omission is a real bug class
A boundary input against one claim	Yes	This is the useful burn
A rewrite of the whole module	No	That is a new project, not a test
Exploit steps against a live company	No	Out of scope and harmful
A score that ranks models	No	This harness is not a benchmark

Read the yes rows as your actual token budget. Your free calls should cluster on those rows. If they do not, the allowance taught you nothing.

How a free server fits the habit

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is described here as an open-source coding project. The brief states free model access and a free server option.

The same brief states a free allowance of ten million tokens. Treat that figure as a brief claim, not a measurement. Nobody measured the live quota, model list, or hardware for this piece.

Check the current terms before you depend on any number. Point the free server at this harness, not at release prose. Hostile fixtures consume calls quickly, so a free pool helps.

Keep production secrets off that server even when the price is zero. A zero price does not change your data handling duties. Synthetic inputs belong in the log, not customer records.

Do not invent a model winner from one free session. Model names move, and published limits move with them. Your calls file is the artifact you can still defend.

If you try this, read MonkeyCode's current free-server terms first. Then run the gate against a fresh hostile log. Use the allowance to break one claim already in your repo.

A workflow for one claim

Pick one claim and run the loop below without shortcuts. The order matters because the label comes before the call. If you label the run after you like the answer, you will cheat.

Seven steps, in order

Pick one sentence from your spec and assign a claim id.
Write a contradiction prompt that quotes only that sentence.
Write an omission prompt for a missing field or error path.
Write a boundary prompt using an empty, duplicate, or maximum input.
Send those three prompts to the free server and save every reply.
Mark survived yourself, and do not let the model mark it.
Run the gate and commit the log beside the spec change.

Step six matters more than the wording of the model answer. The model can suggest a possible break in the spec. You decide whether that spec sentence actually failed.

If you skip that human decision, you outsourced judgment. A scratch host should not own your release standard. Keep the decision in the note field, in your own words.

Limits of this habit

This gate counts fixture families, not real-world truth. A silly contradiction can still pass the script cleanly. You must read the note before you trust the OK line.

Three calls can easily miss the dangerous case. The minimum exists to stop summary spam, not to certify coverage. Add a fixture when a bug escapes, and keep the old ones.

Free tiers change without asking for your permission first. A workflow that needs today's pool next month will stall. Store the prompts in git so any later host can rerun them.

The script trusts the fixture label you typed yourself. You can lie in the log and still receive OK. That lie fools a script, not a reviewer who reads the input.

What this is not

This piece reports no latency, accuracy, or savings figure. None of those quantities were measured for this workflow. If a quote adds a number, that number was invented later.

Who should not use it

Skip this habit if your change has no written claim. Hostile fixtures need a sentence you can actually falsify. Pure visual polish often has no such sentence yet.

Skip this if the diff holds secrets or personal data. Do not upload that material to a free server to save money. Rewrite the fixture with synthetic values before any call.

Skip this if counsel has not approved third-party model hosts. A free server is still an external processor until policy allows it. A zero invoice is not a legal argument for uploading code.

Skip this if you wanted a public model ranking. The decision table already forbids that use of the gate. Measure on a set you control, and publish the method separately.

Hold the line in review

Ask one question when a teammate pastes a model summary. Which claim did this call try to break today? If they cannot name the claim id, the call was entertainment.

Ask to see all three families inside the saved log. If a family is missing, send them back to the free bench. Do not block the merge on the model's stated confidence.

Block the merge on a missing fixture you already required. Free tokens are a research stipend for your own spec. Spend them on contradictions, omissions, and hard boundaries.

Leave the flattering summary unsent, unquoted, and out of the review. A free run that only comforts you did not earn a citation. Write the broken case down, or admit you did not look.

Pin the Tool Host Before You Spend a Token

Emery Chen — Thu, 08 Oct 2026 10:27:24 +0000

Stop spending model quota on chats you cannot rerun. A free model call is not an environment. Pin the tool host first, then spend the tokens.

You are not short of demos this month. You are short of proof that another person can rebuild. A transcript from one warm process is a story.

The claim

Here is the position, stated with no hedge. An agent run counts only when both sides are disposable. The model call must be logged as its own record.

The tool host must restart without your personal shell. If either side is "my machine that day," discard the run. Do not cite that run in a pull request.

Do not paste it into a contest writeup as proof. This is not a plea for a bigger cluster. This is a plea for a named host and a frozen contract.

You can hold that bar on a small server. You cannot hold it with a warm laptop session. A session dies, and the proof dies with it.

Why the split matters

Developers keep collapsing two products into one sentence. They say the model itself built the feature. The model emitted text, and a host may have acted.

Those two failures are not the same failure. A fluent model can still call a missing binary. A healthy host can still receive a broken call.

If you grade them as one blob, you learn nothing. You only learn that one afternoon was kind to you. Kind afternoons do not survive a clean reboot.

Agent contests reward a screenshot more than a boot log. Vibe-coded side projects reward that same bad habit. The missing artifact is a host you can boot again.

What pinned means

Pinned does not mean large, costly, or permanent. Pinned means a stranger can rebuild the tool side. They should not need your shell history to do it.

Use this checklist before you spend any quota.

Give the host a name that is not your laptop.
Keep the tool list in a versioned file.
Give each tool a timeout and a real error path.
Keep secrets in the host environment, not the transcript.
Write a manifest before you write any verdict.
Reboot the host and confirm the contract still holds.

Miss one item and the run is only a rehearsal. Rehearsals are useful while you design the loop. Rehearsals are not evidence you can cite later.

Score the host before the model

Score the host before you argue about model quality. If the host column fails, stop spending tokens. Model taste cannot repair a missing boot path.

Question	Pass	Fail
Can you name the host image?	Image id is in the manifest	"My dev box"
Are tools declared outside the prompt?	Schema file, versioned	Tools described in chat
Does a tool crash return structured error?	Exit code plus stderr field	Empty string or hang
Can a second person boot the host?	Documented start command	Needs your dotfiles
Is the model call logged separately?	Request id, token count, latency	Only the final answer
Did you rerun after a host restart?	Same schema, new process	One warm process only

Two failed rows mean you do not have a result. You have a demo that happened to look finished. Demos can guide the next hour of work.

They should not guide a merge decision alone. A green answer on a dead host is still a miss. Write the miss down, then fix the host.

The order you should follow

Follow this order, and do not swap the last steps forward. Quota spent before a restart is a sunk demo. The model cannot refund that kind of confusion.

Put the tool schema in the repo and review it like code.
Boot the tool host on a named machine you can restart.
Run the harness and demand a manifest on disk.
Restart the host and demand a second manifest.
Attach model access and append one logged call.
Score the table, then decide whether the answer matters.

Three failures this rule catches

Warm process

The warm-process failure looks like a clean success. Your tools stay imported, so the second call is cheap. A restart would have shown the missing package.

Secret in the prompt

Putting a secret in the prompt looks like convenience. The model repeats a token it should never have seen. The host should have read that secret from its own env.

Schema only in chat

Keeping the schema in chat looks like speed. You change a tool argument and forget the old transcript. The file in git would have made the break obvious.

A harness you can copy

The samples below are only a proposed workflow. This article does not report an executed benchmark. Copy them, then run them on a host you control.

Refuse early

The harness does three jobs and no more. It refuses to start without a host base URL. It writes a manifest before any model call.

It treats tool output as data, not as a vibe. It also refuses a laptop loopback as a cited host. A loopback probe can exist, but it cannot be the proof.

#!/usr/bin/env python3
"""Proposed harness. Not executed for this article."""

import json
import os
import sys
import time
import urllib.request
from pathlib import Path

ROOT = Path(__file__).resolve().parent
MANIFEST = ROOT / "run-manifest.json"
SCHEMA = ROOT / "tools.json"

def die(msg: str) -> None:
    print(msg, file=sys.stderr)
    raise SystemExit(2)

def require_env(name: str) -> str:
    value = os.environ.get(name, "").strip()
    if not value:
        die(f"missing {name}")
    return value

def load_schema() -> dict:
    if not SCHEMA.is_file():
        die("tools.json missing; refuse to invent tools in a prompt")
    data = json.loads(SCHEMA.read_text())
    if not data.get("tools"):
        die("tools.json has no tools")
    return data

def probe_host(base: str) -> dict:
    url = base.rstrip("/") + "/health"
    req = urllib.request.Request(url, method="GET")
    try:
        with urllib.request.urlopen(req, timeout=5) as resp:
            body = resp.read(2048).decode("utf-8", "replace")
            return {"ok": resp.status == 200, "status": resp.status, "body": body}
    except Exception as exc:
        die(f"host probe failed: {exc}")

def write_manifest(host: str, schema: dict, probe: dict) -> None:
    doc = {
        "host": host,
        "schema_tools": [t.get("name") for t in schema["tools"]],
        "probe": probe,
        "started_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "model_calls": [],
        "tool_results": [],
        "verdict": "incomplete",
    }
    MANIFEST.write_text(json.dumps(doc, indent=2) + "\n")

def main() -> None:
    host = require_env("TOOL_HOST")
    lowered = host.lower()
    if "127.0.0.1" in lowered or "localhost" in lowered:
        die("refusing loopback; pin a named host")
    schema = load_schema()
    probe = probe_host(host)
    write_manifest(host, schema, probe)
    print(f"manifest written: {MANIFEST}")
    print("next: call the model only after this file exists")

if __name__ == "__main__":
    main()

Contract file

Pair the harness with a tool contract in git. Do not hide that contract inside a system prompt. A system prompt is not a versioned interface.

{
  "tools": [
    {
      "name": "run_check",
      "timeout_sec": 20,
      "input": {},
      "output": {"exit_code": "int", "stdout": "string", "stderr": "string"}
    }
  ]
}

An incomplete verdict is the only honest status before a restart. Fill model calls only after the second boot. Leave the list empty rather than inventing a call.

Fixed check

The host process can stay boring on purpose. Boring is a feature when you need a rerun. A free server is enough if the process can restart.

You do not need a clever runtime to prove the split. The sample runs one fixed check, not a shell string. Clever runtimes hide the reboot you are trying to test.

#!/usr/bin/env python3
"""Proposed tool host. Not executed for this article."""

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
import subprocess

HOST = os.environ.get("BIND_HOST", "127.0.0.1")
PORT = int(os.environ.get("BIND_PORT", "8080"))
CHECK = ["python3", "-m", "pytest", "-q"]

class Handler(BaseHTTPRequestHandler):
    def _send(self, code: int, payload: dict) -> None:
        raw = json.dumps(payload).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def do_GET(self) -> None:
        if self.path != "/health":
            self._send(404, {"ok": False})
            return
        self._send(200, {"ok": True, "check": CHECK})

    def do_POST(self) -> None:
        if self.path != "/tools/run_check":
            self._send(404, {"ok": False})
            return
        try:
            proc = subprocess.run(
                CHECK,
                capture_output=True,
                text=True,
                timeout=20,
                check=False,
            )
        except subprocess.TimeoutExpired:
            self._send(200, {"exit_code": 124, "stdout": "", "stderr": "timeout"})
            return
        self._send(
            200,
            {
                "exit_code": proc.returncode,
                "stdout": proc.stdout[-2000:],
                "stderr": proc.stderr[-2000:],
            },
        )

if __name__ == "__main__":
    ThreadingHTTPServer((HOST, PORT), Handler).serve_forever()

Do not expose this sample to the public internet. It is a lab listener, not a production gateway. Bind it on a host you can turn off.

Local bind is only for a private smoke test. A cited run still needs a named host URL. Do not launder a laptop loopback through a tunnel and call it pinned.

Commands that should fail closed

Run the negative cases before you celebrate a green file. A harness that only passes on the happy path is theater. Expect exit code 2 when the pin is missing.

python3 pin_host.py
# expect exit 2: missing TOOL_HOST

TOOL_HOST=http://localhost:8080 python3 pin_host.py
# expect exit 2: loopback refused

TOOL_HOST=http://tool-host.example:8080 python3 pin_host.py
# expect run-manifest.json only after /health returns 200

Then restart the host process and probe again. The new manifest needs a new start timestamp. Same schema, new process, or the pin failed.

Only after that file exists should you attach model access. Append one model record with an id and a latency. If you cannot fill those fields, do not score the answer.

Where a free option fits

You can do this split without buying hardware first. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant here only as a place to try the split.

The operator reports free model access and a free server option. This article does not name models, quotas, or hardware. Those details move, and stale numbers waste your week.

Read the current project page before you depend on either offer. You bring the host process, or you adapt the probe. Nothing here assumes a vendor route named health.

Use the free server as the named tool host if it fits. Use the free model access only after the manifest exists. If the live terms are tighter than you hoped, shrink the loop.

Do not invent capacity the page does not list. Do not treat a free tier as a permanent contract. Confirm the live limits, then throw the run away and rerun.

Who should skip this

Skip this approach if the repo holds regulated data. Skip it if customer secrets would enter the tool host. A free shared option is a lab, not a vault.

Skip it if you need a contract, a region, or a retention promise. Free availability is not an SLA, and this article does not claim one. If your policy team needs paper, get paper elsewhere.

Skip it if you want a leaderboard or a quality score. This workflow checks disposability, not the answer's goodness. A pinned host can still emit a wrong patch.

Skip it if you only needed a single chat reply. Standing up a host for one prompt is wasted motion. Use a chat box, and do not pretend it was an eval.

Limits you should say out loud

The code samples were not run for this writeup. A health check does not prove the tool did useful work. You still need a task oracle you trust outside the model.

A free server can throttle, sleep, or change under you. Treat that as a known limit, not as a surprise betrayal. Pin what you can, and record when the pin breaks.

The localhost refusal is a harness policy, not a firewall. Anyone can delete that check and lie to themselves. The value is the habit, not the ten lines that enforce it.

No benchmark numbers appear here, because none were measured. Do not cite this article as speed, cost, or quality proof. Cite your own manifest, or cite nothing at all.

Hold the line

Keep the screenshots if they help you remember the idea. Do not call those screenshots results in a review. Spend free tokens after the host survives a restart.

If the host cannot restart, the model was never the point. A free call does not repair a host you cannot name. Pin the host, or do not spend the quota.