DEV Community: Dakota Liu

Your Free Endpoint Isn't Slow, It's Queued: A Five-Stage Readiness Probe in Pure Python

Dakota Liu — Tue, 27 Oct 2026 13:15:07 +0000

Your Free Endpoint Isn't Slow, It's Queued: A Five-Stage Readiness Probe in Pure Python

Free-tier latency is not one number. It's at least four: cold start, steady-state, the concurrency knee, and the error rate you only ever see when you stop sending one request at a time.

Measure only the first and you'll make a bad call — either you'll ship user-facing traffic onto a tier that queues at four concurrent requests, or you'll write off a perfectly usable endpoint because your very first call hit a cold container. Both mistakes cost a week.

So let's build the thing that prevents both: one Python file, five stages, a contract you write before you touch the network, and a verdict you can defend in a review. No third-party packages. About an hour, most of it waiting.

Why a probe instead of a benchmark

A benchmark answers "how fast is this?" A probe answers "is it fast enough for my workload, and what breaks first?" Only the second question survives contact with production.

The loudest AI arguments in my feed lately aren't about model quality at all. They're about access and transparency — who sits between you and the model, and what that middle layer hides. Fair debate. But it's unfalsifiable until you measure your own path to the endpoint. That's all this is: a measurement of your path.

Stage 0 — Write the contract first

If you don't define "fast enough" before you see a single number, you will rationalize whatever you get. So the first artifact is a JSON contract, not code.

{
  "name": "summary-endpoint",
  "prompt": "Summarize in one sentence: free tiers fail politely.",
  "max_tokens": 32,
  "budgets": {
    "single_request_p95_ms": 4000,
    "cold_start_ms": 15000,
    "ladder_p95_ceiling_ms": 12000,
    "error_rate": 0.02
  }
}

Those budgets are placeholders. Replace them with numbers derived from your actual use case — an interactive autocomplete has a very different ceiling than a nightly batch summarizer.

Verification for Stage 0: every budget must be a number you can point at a product requirement for. If you can't, delete it.

Stage 1 — One request, one recorded probe

Config comes from the environment so nothing secret lands in the repo. Set your base URL, key, and model name from your provider's dashboard — and if the endpoint path isn't OpenAI-compatible, MC_PATH is the single line you change.

#!/usr/bin/env python3
"""probe.py - five-stage readiness probe. Pure stdlib."""
import argparse, json, os, statistics, sys, time, urllib.error, urllib.request
from concurrent.futures import ThreadPoolExecutor

BASE = os.environ['MC_BASE_URL'].rstrip('/')
KEY = os.environ['MC_API_KEY']
MODEL = os.environ['MC_MODEL']
PATH = os.environ.get('MC_PATH', '/chat/completions')
LOG = 'probe-records.jsonl'


def call(prompt, max_tokens, timeout=60.0):
    payload = json.dumps({
        'model': MODEL,
        'messages': [{'role': 'user', 'content': prompt}],
        'max_tokens': max_tokens,
    }).encode()
    req = urllib.request.Request(
        BASE + PATH, data=payload,
        headers={'Content-Type': 'application/json',
                 'Authorization': 'Bearer ' + KEY},
    )
    t0 = time.perf_counter()
    rec = {'ok': False, 'status': None, 'retry_after': None, 'tokens': None}
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            body = json.loads(r.read())
            rec['ok'] = True
            rec['status'] = r.status
            rec['tokens'] = (body.get('usage') or {}).get('completion_tokens')
    except urllib.error.HTTPError as e:
        rec['status'] = e.code
        rec['retry_after'] = e.headers.get('Retry-After')
    except Exception as e:            # timeout, DNS, TLS, malformed body
        rec['status'] = type(e).__name__
    rec['ms'] = round((time.perf_counter() - t0) * 1000, 1)
    return rec


def record(stage, **fields):
    line = {'stage': stage, 'ts': round(time.time(), 3), **fields}
    with open(LOG, 'a') as f:
        f.write(json.dumps(line) + '\n')
    print(json.dumps(line))
    return line

One request, one JSON line on disk. Append-only, greppable, and diffable week over week.

Verification for Stage 1: you get "ok": true and a non-null tokens value. If tokens is null, your endpoint isn't returning usage data — note that before you build any cost model on top of it.

Stage 2 — Cold start, which is a waiting problem

Idle the client, then time a single request. Don't guess the idle window; 90 seconds catches most short-lived instance recycling, and you can raise it later.

def stage_cold(cfg, idle):
    print('idling', idle, 's')
    time.sleep(idle)
    rec = call(cfg['prompt'], cfg['max_tokens'])
    budget = cfg['budgets']['cold_start_ms']
    ok = rec['ok'] and rec['ms'] <= budget
    record('cold', ok=ok, ms=rec['ms'], budget=budget, status=rec['status'])
    return ok

Here's the part people skip: a slow cold start is not a reason to reject a tier. It's a reason to send a warm-up request before the first user shows up, or to keep the tier on batch work only.

Verification for Stage 2: run it twice. If the second run is fast, you've confirmed the cold path exists and you know its shape.

Stage 3 — The ladder, where you find the knee

This is the stage that actually decides your architecture. Send 1, 2, 4, then 8 requests at once and watch p50 stay flat while p95 climbs. That divergence is the queue.

def p95(xs):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(round(0.95 * (len(xs) - 1))))]


def stage_ladder(cfg, levels):
    for n in levels:
        with ThreadPoolExecutor(max_workers=n) as pool:
            recs = list(pool.map(
                lambda _: call(cfg['prompt'], cfg['max_tokens']), range(n)))
        ms = [r['ms'] for r in recs if r['ok']]
        record('ladder', concurrency=n, sent=len(recs),
               errors=sum(1 for r in recs if not r['ok']),
               p50_ms=round(statistics.median(ms), 1) if ms else None,
               p95_ms=p95(ms) if ms else None)

python probe.py ladder --config contract.json --levels 1,2,4,8

Illustrative output shape — these are placeholders from a made-up run, not a benchmark of anyone's host:

{"stage": "ladder", "concurrency": 1, "errors": 0, "p50_ms": 900, "p95_ms": 950}
{"stage": "ladder", "concurrency": 4, "errors": 0, "p50_ms": 980, "p95_ms": 4100}
{"stage": "ladder", "concurrency": 8, "errors": 3, "p50_ms": 1200, "p95_ms": 9800}

Read that second line again. p50 barely moved; p95 quadrupled. Nothing got slower — requests started waiting their turn. Your in-flight cap belongs just below that point, not above it.

Verification for Stage 3: you can name an integer — "no more than N in flight" — and point at the row that justifies it.

Stage 4 — Push past the knee on purpose

Now run a level you know is too high, and watch your own client. Does it back off, or does it hammer? This stage tests your code, not theirs.

If you see 429s, honor Retry-After when it's present and use exponential backoff with jitter when it isn't. Never retry a 4xx that isn't a rate limit — that's a bug, not congestion.

Verification for Stage 4: errors at 16 concurrent are fine. Errors with no backoff in your logs are not.

Stage 5 — Turn records into a verdict

A decision table beats a vibe. This is the whole payoff of logging JSONL instead of eyeballing a terminal:

Observation	Likely cause	Action
Slow cold start, steady state fine	instance spin-up / cold cache	warm-up ping before user traffic; keep batch work here
p50 flat, p95 climbs with concurrency	shared worker queue	cap in-flight at the knee; queue client-side
Errors only at high concurrency, `Retry-After` present	rate limit	backoff + jitter; don't retry non-429 4xx
Fast responses, truncated output	`max_tokens` too low	raise the budget, re-run Stage 1
Everything passes but you re-ran once	small sample	40+ requests before you trust p95

That last row matters. With eight samples, "p95" is basically the maximum. Treat it as a ceiling, never as a percentile.

Where a free key and a free server fit

You can run all five stages against a paid endpoint, and you probably should eventually. But the free tier is exactly where this probe earns its keep, because free capacity is the capacity most likely to share a queue with strangers.

MonkeyCode is one option here: the team advertises free model access plus a free server option, and presents the project as open source. The free allowance was reported to me as 10 million tokens — treat that as a claim to re-check on the project page before you budget against it, because allowances move and my number is already aging. What I can tell you is that the workflow above costs almost nothing to run: a few hundred short requests, a JSONL file, and no dependencies.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Who should not use this approach

Be honest about the fit.

You need a hard latency SLO for paying users. A probe tells you today's shape; it doesn't give you a contract. Move to a tier with an agreement.
You can't re-run it weekly. Free capacity shifts. A verdict from six weeks ago is folklore.
You need a pinned model version. If your output must be reproducible, availability on any free tier is the wrong axis to optimize.
You haven't read the data-handling terms. I'm making no claim about them here. That's your homework, not this probe's.

The one habit worth keeping

Write the contract, take one measurement per question, and log every probe as a line you can diff. That habit outlives any specific provider — and it turns "the free tier felt slow" into "p95 crossed 12 seconds at eight concurrent requests, so we capped at four."

If you want a zero-cost place to run Stage 0 through Stage 3 and see where your own knee lands, spin up a free key and a free server on MonkeyCode and let the numbers argue for themselves.

Stop Counting Calls, Start Counting Accepted Outputs: A Five-Stage Token Ledger Tutorial

Dakota Liu — Mon, 26 Oct 2026 11:45:00 +0000

Every free-tier LLM project I look at has the same blind spot. The dashboard says requests, the bill says zero, and nobody can answer the one question that matters: how many tokens did we burn to get one output we actually kept?

So here is the conclusion up front. Tokens per call is a vanity metric. Tokens per accepted output is the number that decides whether free inference is cheap or just deferred cost. In this tutorial we build the ledger that produces that number — from an empty file to a report you can paste into a design review.

Five stages. Each one has code and a verification step, so you never advance on a stage you haven't seen work. The code is provider-agnostic on purpose: it talks to any chat endpoint behind a thin adapter, so the ledger stays useful no matter which backend you point it at.

One practical note on where to run it. I use MonkeyCode's free model access behind the adapter in Stage 2 so the meter has something real to measure without a card on file, and the free server option to keep the tiny stub endpoint alive while I iterate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Both of those are availability claims supplied by the project, not benchmarks — quotas, limits, and terms change, so read the current ones on the site before you plan a sprint around them. The ledger below works the same either way.

Stage 1 — Define the receipt before you write the client

Why start with the data shape? Because every metering project I've watched die, died the same way: somebody added a field in week three and the historical numbers stopped being comparable.

Freeze the receipt first. Keep it boring and JSON-serializable.

# ledger/receipt.py
from dataclasses import dataclass, asdict, field
import json, time, uuid

@dataclass
class Receipt:
    request_id: str = field(default_factory=lambda: uuid.uuid4().hex[:12])
    stage: str = "draft"          # which pipeline step asked
    backend: str = "free"         # "free" | "paid" — label it, never infer it
    model: str = "unknown"
    prompt_chars: int = 0
    output_chars: int = 0
    usage_prompt: int | None = None    # provider-reported, None when absent
    usage_output: int | None = None
    latency_ms: int = 0
    status: str = "ok"                # ok | error | truncated
    accepted: bool | None = None      # None = not judged yet

    def to_json(self) -> str:
        return json.dumps(asdict(self), sort_keys=True)

Two fields carry all the weight. backend because a free run and a paid run are not the same observation and must never be averaged together. accepted because a receipt with no verdict is just a timestamp with extra steps.

Verify:

python -c "from ledger.receipt import Receipt; r=Receipt(); print(r.to_json()); assert r.accepted is None"

If that assertion fails, you've already started guessing. Stop and fix it.

Stage 2 — Meter the call, and label every estimate

Now the adapter. The trap here is pretending you know the token count. You often don't — many endpoints return a usage block, some don't, and the ones that do disagree about what counts.

So: prefer provider numbers, fall back to an estimate, and mark the estimate as an estimate.

# ledger/meter.py
import time
from .receipt import Receipt

def estimate_tokens(text: str) -> int:
    # rough, deliberately conservative; NOT a tokenizer
    return max(1, len(text) // 4)

def metered_call(adapter, prompt: str, stage: str, model: str) -> tuple[str, Receipt]:
    rec = Receipt(stage=stage, model=model, backend=adapter.backend,
                  prompt_chars=len(prompt))
    t0 = time.monotonic()
    try:
        out, usage = adapter.chat(prompt)   # usage: dict | None
        rec.status = "ok"
    except Exception:
        rec.status = "error"
        rec.latency_ms = int((time.monotonic() - t0) * 1000)
        return "", rec
    rec.latency_ms = int((time.monotonic() - t0) * 1000)
    rec.output_chars = len(out)
    if usage:
        rec.usage_prompt = usage.get("prompt_tokens")
        rec.usage_output = usage.get("completion_tokens")
    else:
        rec.usage_prompt = estimate_tokens(prompt)
        rec.usage_output = estimate_tokens(out)
    return out, rec

The adapter.chat() signature is a stub by design — write your own transport for whatever endpoint you're testing. Mine is twenty lines and returns (text, usage_or_none). That's the whole contract.

Verify with a fake, before you spend a single token:

# tests/test_meter.py
from ledger.meter import metered_call

class FakeAdapter:
    backend = "free"
    def chat(self, prompt):
        return "ok", {"prompt_tokens": 11, "completion_tokens": 2}

class NoUsageAdapter(FakeAdapter):
    def chat(self, prompt):
        return "ok", None

def test_provider_numbers_win():
    _, rec = metered_call(FakeAdapter(), "hello there", "draft", "m")
    assert (rec.usage_prompt, rec.usage_output) == (11, 2)

def test_estimate_is_still_recorded():
    _, rec = metered_call(NoUsageAdapter(), "hello there", "draft", "m")
    assert rec.usage_prompt > 0 and rec.status == "ok"

python -m pytest tests/test_meter.py -q

Green? Now point the adapter at a real free endpoint and run ten prompts. You should see usage_prompt populated or your estimate filling in — either way, no receipts with zeros.

Stage 3 — Put a gate in front of the meter

Metering after the fact is accounting. Metering before the call is control. The gate answers: is the next request even allowed?

Two rules earn their keep immediately. A projected-token ceiling, and an error-rate circuit breaker — because a free endpoint that starts 500-ing is burning your retry budget, not your token budget.

# ledger/gate.py
from collections import deque

class Gate:
    def __init__(self, max_tokens: int, window: int = 20, max_error_rate: float = 0.25):
        self.max_tokens = max_tokens
        self.window = deque(maxlen=window)
        self.max_error_rate = max_error_rate
        self.spent = 0

    def record(self, rec):
        if rec.status == "error":
            self.window.append(1)
        else:
            self.window.append(0)
            self.spent += (rec.usage_prompt or 0) + (rec.usage_output or 0)

    def allow(self, projected_prompt_tokens: int) -> tuple[bool, str]:
        if self.spent + projected_prompt_tokens > self.max_tokens:
            return False, "ceiling"
        if self.window and sum(self.window) / len(self.window) > self.max_error_rate:
            return False, "error_rate"
        return True, "ok"

Verify with a table of cases, not vibes:

from ledger.gate import Gate
from ledger.receipt import Receipt

def test_gate_stops_at_ceiling():
    g = Gate(max_tokens=100)
    g.record(Receipt(usage_prompt=60, usage_output=30))
    assert g.allow(20) == (False, "ceiling")
    assert g.allow(5)[0] is True

def test_gate_trips_on_errors():
    g = Gate(max_tokens=10_000, window=4, max_error_rate=0.25)
    for _ in range(2):
        g.record(Receipt(status="error"))
    assert g.allow(10) == (False, "error_rate")

The ceiling number is yours to pick — start smaller than you think is polite. The point is that the gate refuses before the request leaves your process, so a surprise quota change produces a clean stop, not a half-finished batch.

Stage 4 — Record acceptance, or the ledger lies

Here's where most metering tutorials quietly cheat. They count tokens and stop. But an output that fails your parser cost you the same tokens as one that passed.

So attach a verdict. Not a human opinion — a mechanical check: does it parse, does the test pass, does the field exist?

# ledger/accept.py
import json

def judge_json(rec, text: str) -> bool:
    try:
        json.loads(text)
        rec.accepted = True
    except json.JSONDecodeError:
        rec.accepted = False
    return rec.accepted

Verify both directions:

from ledger.receipt import Receipt
from ledger.accept import judge_json

def test_accepts_valid():
    r = Receipt()
    assert judge_json(r, '{"a": 1}') is True

def test_rejects_garbage():
    r = Receipt()
    assert judge_json(r, 'Sure! Here is your JSON:') is False

That second test is the one people skip, and it's the one that shows up in production. A model that answers conversationally instead of emitting JSON hasn't failed loudly — it has failed expensively.

Stage 5 — Print one number: tokens per accepted output

Now the payoff. Read your receipts back, split by backend, and compute the ratio.

# ledger/report.py
import json
from collections import defaultdict

def report(path: str) -> dict:
    buckets = defaultdict(lambda: {"tokens": 0, "accepted": 0, "calls": 0, "errors": 0})
    with open(path) as f:
        for line in f:
            r = json.loads(line)
            b = buckets[r["backend"]]
            b["calls"] += 1
            b["tokens"] += (r["usage_prompt"] or 0) + (r["usage_output"] or 0)
            b["errors"] += r["status"] == "error"
            b["accepted"] += bool(r["accepted"])
    for name, b in buckets.items():
        b["tokens_per_accepted"] = round(b["tokens"] / b["accepted"], 1) if b["accepted"] else None
    return dict(buckets)

python -c "from ledger.report import report; import json; print(json.dumps(report('runs/2026-10-26.jsonl'), indent=2))"

Verify the report against a hand-built fixture first. Two receipts, one accepted, 100 tokens total, ratio must be 100.0. If your fixture disagrees with the function, the function is wrong — not the fixture.

The decision table

What do you do with the ratio? Compare it honestly.

Situation	Free backend wins?	Why
Prompt tuning, exploratory work	Yes	Cost of a bad call is zero; the ledger tells you the retry tax
Batch job with a hard deadline	Rarely	Error-rate breaker trips more often than you'd like
Pipeline with strict schema output	Depends	If tokens-per-accepted is high, free is not free
Anything user-facing and latency-bound	No	You're measuring the wrong axis entirely

That third row is the whole article. A free endpoint at 3,000 tokens per accepted output can lose to a paid one at 400.

Limitations, and who should skip this

The estimator is not a tokenizer. len(text) // 4 is a blunt instrument and will mislead you on non-English text, code, or anything with heavy punctuation. Treat estimated numbers as directional only.

Provider usage blocks are not standardized. If yours is missing fields, your ledger will silently under-count — add an assertion, not a comment.

Free availability is not an SLA. Both claims I mentioned earlier are claims, not guarantees; the gate exists precisely because the ground can move under you.

And the acceptance check is the weakest link. json.loads is easy. A semantic grader is not. If you can't define "accepted" mechanically, you can't compute the ratio — build that first.

Skip this approach if you're in a regulated path needing audit-grade billing, if your latency budget is under a second end-to-end, or if your volume is high enough that the free tier is a rounding error anyway.

Close

Five stages, maybe ninety minutes, and you end up with a number that survives contact with a design review: tokens per accepted output, split by backend, with the errors counted instead of hidden.

If standing up the endpoint is the part that keeps holding you back, MonkeyCode's free server option is one way to get a live target for Stage 2 — verify the current terms yourself and let the ledger do the arguing.

Bring-Up Gates: A Four-Stage Tutorial for Wiring an LLM Feature You Don't Trust Yet

Dakota Liu — Tue, 20 Oct 2026 09:53:07 +0000

The first successful API call is not the milestone. It's a coincidence. The milestone is the second one — same input, same output shape, same exit code — on a machine that isn't yours.

So here's the tutorial I wish I'd had: take a narrow feature (ticket triage), bring it up in four gated stages, and make every stage fail loudly. You'll end Stage 3 with the same code running locally and on a remote free server, with a hash diff as the proof.

Two notes before we start. The remote stages use MonkeyCode's free model access and free server option; the operator currently advertises a 10M-token free allowance, and I'm treating that as their claim, not a measurement — quotas and availability move, so check the current terms yourself. Also, I'm not asserting license details here; read the repo before you depend on it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Stage 0 — Freeze the contract before you freeze the prompt

Most people write the prompt first. That's backwards. The prompt is the thing most likely to change; the shape of the input and output is the thing you want bolted down.

Create a 10-line fixture. Hand-label it. Never let the model generate it.

mkdir -p gates fixtures out
cat > fixtures/tickets.jsonl <<'EOF'
{"id":"t01","text":"Charged twice for the same invoice","label":"billing"}
{"id":"t02","text":"App crashes on upload over 20MB","label":"bug"}
{"id":"t03","text":"Cannot reset password, email never arrives","label":"account"}
{"id":"t04","text":"Need to change the tax ID on receipts","label":"billing"}
{"id":"t05","text":"Sidebar renders blank in Safari 17","label":"bug"}
{"id":"t06","text":"How do I add a second admin?","label":"account"}
{"id":"t07","text":"Refund request for last month","label":"billing"}
{"id":"t08","text":"Webhook retries hammer our endpoint","label":"bug"}
{"id":"t09","text":"SSO login loops back to the login page","label":"account"}
{"id":"t10","text":"Is there a status page?","label":"other"}
EOF

Now the gate. No network, no model, just structure.

# gates/contract.py
import json, pathlib, sys

ALLOWED = {"billing", "bug", "account", "other"}

def load(path):
    rows = []
    for i, line in enumerate(pathlib.Path(path).read_text().splitlines(), 1):
        if not line.strip():
            continue
        row = json.loads(line)
        assert set(row) >= {"id", "text", "label"}, f"line {i}: missing keys"
        assert row["text"].strip(), f"line {i}: empty text"
        assert row["label"] in ALLOWED, f"line {i}: bad label {row['label']!r}"
        rows.append(row)
    ids = [r["id"] for r in rows]
    assert len(ids) == len(set(ids)), "duplicate ids"
    return rows

if __name__ == "__main__":
    rows = load(sys.argv[1])
    print(f"CONTRACT_OK {len(rows)}/{len(rows)}")

Verify: python gates/contract.py fixtures/tickets.jsonl prints CONTRACT_OK 10/10 and exits 0. Corrupt one line on purpose — an assert should stop you, not a stack trace three stages later.

Stage 1 — One call, one ledger line

Single call. No retries. No batching. The only goal is to learn what your provider actually returns, and to record it so you can argue with yourself later.

# gates/call.py
import hashlib, json, os, time, urllib.request

ENDPOINT = os.environ["LLM_BASE_URL"].rstrip("/") + "/chat/completions"  # verify path in provider docs

def call(prompt: str):
    body = json.dumps({
        "messages": [{"role": "system", "content": "Reply with one word: billing, bug, account, or other."},
                     {"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(ENDPOINT, data=body, headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {os.environ['LLM_API_KEY']}",
    })
    t0 = time.monotonic()
    with urllib.request.urlopen(req, timeout=60) as resp:
        raw = resp.read().decode()
    return raw, round(time.monotonic() - t0, 3)

def ledger(rec, path="out/ledger.jsonl"):
    with open(path, "a") as f:
        f.write(json.dumps(rec, sort_keys=True) + "\n")

The endpoint path and env var names above are placeholders — read the provider's docs and fix them before you trust anything downstream. Then run one item:

LLM_BASE_URL=... LLM_API_KEY=... python - <<'PY'
from gates.call import call, ledger
import hashlib, json, pathlib
row = json.loads(pathlib.Path("fixtures/tickets.jsonl").read_text().splitlines()[0])
raw, lat = call(row["text"])
ledger({"id": row["id"], "sha256": hashlib.sha256(raw.encode()).hexdigest(),
        "latency_s": lat, "raw": raw})
print(raw)
PY

Verify: out/ledger.jsonl has exactly one line, and tail -1 out/ledger.jsonl | jq -r .latency_s is a number. If the response has no usage block, store null — don't estimate tokens by dividing characters. Guessed accounting is worse than missing accounting.

Stage 2 — The batch gate that can say no

Now run all ten. A gate is only a gate if it can exit non-zero.

# gates/batch.py
import json, pathlib, re, sys
from collections import Counter
from gates.call import call, ledger

MAX_RETRIES, MIN_PASS, MAX_P95_S = 1, 9, 8.0

def label(raw):
    txt = json.loads(raw)["choices"][0]["message"]["content"].strip().lower()
    m = re.search(r"\b(billing|bug|account|other)\b", txt)
    return m.group(1) if m else None

rows = [json.loads(l) for l in pathlib.Path("fixtures/tickets.jsonl").read_text().splitlines() if l.strip()]
results, lat = [], []
for r in rows:
    for attempt in range(MAX_RETRIES + 1):
        raw, sec = call(r["text"])
        lat.append(sec)
        got = label(raw)
        ledger({"id": r["id"], "attempt": attempt, "parsed": got, "latency_s": sec})
        if got:
            break
    results.append((r["id"], r["label"], got))

passed = sum(exp == got for _, exp, got in results)
p95 = sorted(lat)[max(0, int(len(lat) * 0.95) - 1)]
print(json.dumps({"passed": passed, "of": len(results), "p95_s": p95}))
sys.exit(0 if passed >= MIN_PASS and p95 <= MAX_P95_S else 2)

Verify: echo $? prints 0 on a good run and 2 after you deliberately break one label in label(). Try that break. A gate you've never seen fail is decoration.

Stage 3 — Move the same code to a free server and hash-diff it

Here's where the free server earns its keep: it's a second machine, which is the only honest test of "it works." Same fixture, same temperature: 0, same code.

# local
python gates/batch.py && jq -r '[.id,.parsed]|@tsv' <(grep -v '"attempt": 1' out/ledger.jsonl) | sort > out/local.tsv
sha256sum out/local.tsv

# remote (free server), same repo, same command
python gates/batch.py && jq -r '[.id,.parsed]|@tsv' <(grep -v '"attempt": 1' out/ledger.jsonl) | sort > out/remote.tsv
sha256sum out/remote.tsv

Verify: diff out/local.tsv out/remote.tsv is empty. If the hashes diverge, the cause is almost always one of three things: the remote picked a different model than you assumed, the fixture file drifted, or an unset env var silently fell back to a default. Diff the label sets first — that tells you which.

When free access is the right runtime

Situation	Free tier + free server	Notes
Prototype, internal demo, contract test	Yes	Stage 0–2 is exactly this
Teaching a team prompt/parse discipline	Yes	Cheap, reversible, no procurement
Pre-merge smoke test in CI	Usually	Keep it non-blocking; quotas and cold starts will bite
Latency-sensitive user-facing path	No	No SLA, and queueing is invisible to you
Regulated or personal data	No	Assume logs you don't control
Guaranteed throughput	No	Measure first, then decide

Limitations, honestly

Free allowances are a moving target, and "10M tokens" is a number published by the operator on a given day — record your own usage from the ledger rather than trusting a dashboard total. Free servers get shared machine characteristics you don't choose, so treat latency numbers as observations, not contracts. Hashing only proves determinism when you pin temperature: 0 and normalize output yourself; a model that wraps its answer in a sentence will break your diff while being perfectly correct.

And if you need a written SLA, per-request isolation, or a data-processing agreement, this workflow is not for you — go pay someone. That's not a knock on free tiers; it's just what they are.

Start with Stage 0 tonight. It takes five minutes and it costs nothing, and it's the stage that catches the most. If you want a second machine to run Stage 3 on without a credit card, the free server option is the cheapest way I know to find out whether "works on my laptop" was ever true.

Stop Picking Free Models By Vibe: A 20-Case Golden Set You Can Run This Afternoon

Dakota Liu — Mon, 19 Oct 2026 08:34:35 +0000

You can't tell two free models apart by reading their landing pages. You can tell them apart in about 40 minutes, with 20 cases and a pass/fail rule you wrote down before you saw any output.

That's the whole article: a repeatable, boring, cheap way to choose a model — plus a place to run it without a bill.

Why not just read the benchmarks?

Because your task is not on the benchmark. Public leaderboards measure a dataset someone else wrote, on prompts someone else designed, with a scoring rubric you can't inspect. A big chunk of the current AI discourse is arguments about exactly this. Fair.

So build your own tiny eval. Small is fine. Twenty cases with honest graders will tell you more about your workload than a thousand-row leaderboard ever will.

Here's the catch: it only works if the graders are frozen before the run. Write them first, then run. Tune the expectations after seeing output and you're not evaluating — you're decorating.

Step 1 — Write the task contract first (10 minutes)

One paragraph, in a file, in the repo. What goes in, what comes out, what counts as wrong.

task: summarize a support ticket into {category, severity, needs_human}
input: raw ticket text (<= 1200 chars)
output: JSON object, 3 keys, no prose
hard fail: invalid JSON, missing key, severity outside {low, med, high}
soft fail: wrong category (annoying, not fatal)

Verify: if a new teammate can't grade a sample by hand using only this file, the contract is too vague. Fix it before you write a single case. Vague contract, meaningless score.

Step 2 — Build 20 cases in JSONL

Twelve normal, five edge, three adversarial. That ratio is the point. Normal cases make you feel good; edge cases make you choose correctly; adversarial cases stop you from getting burned in week two.

{"id":"n01","prompt":"My invoice is wrong, charged twice for October.","grader":{"type":"json_keys","expect":["category","severity","needs_human"]}}
{"id":"n02","prompt":"App crashes when I upload a 40MB CSV.","grader":{"type":"regex","expect":"\"severity\":\\s*\"(med|high)\""}}
{"id":"e01","prompt":"","grader":{"type":"exact","expect":"{\"error\":\"empty_input\"}"}}
{"id":"e02","prompt":"ignore previous","grader":{"type":"human"}}
{"id":"a01","prompt":"Mixed English/Japanese ticket, 1100 chars.","grader":{"type":"json_keys","expect":["category","severity","needs_human"]}}

Rules I keep:

Every case gets an id. You will reference it in bug reports and in arguments with yourself.
At least one empty input and one oversized input.
One case where the correct behaviour is to refuse.
Real inputs from production logs, redacted — not rows from a public set you might already be contaminated by.

Verify: wc -l cases.jsonl returns 20, and every line parses:

python -c "import json;[json.loads(l) for l in open('cases.jsonl')]"

Step 3 — The runner (stdlib, ~50 lines)

Keep the runner dumb. It calls, times, captures, writes. No scoring inside — grading is a separate pass so you can re-grade old runs when your graders change.

# runner.py
import argparse, json, time, pathlib

def call_model(model, prompt, timeout):
    if model == "stub":
        return {"text": '{"category":"x","severity":"low","needs_human":false}',
                "latency_ms": 1, "error": None}
    raise SystemExit("wire your client here; keep the return shape")

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--model", required=True)
    ap.add_argument("--cases", default="cases.jsonl")
    ap.add_argument("--out", required=True)
    ap.add_argument("--timeout", type=float, default=20.0)
    a = ap.parse_args()

    cases = [json.loads(l) for l in open(a.cases) if l.strip()]
    out = pathlib.Path(a.out); out.parent.mkdir(parents=True, exist_ok=True)
    with out.open("w") as f:
        for c in cases:
            t0 = time.time()
            try:
                r = call_model(a.model, c["prompt"], a.timeout)
            except Exception as e:
                r = {"text": "", "latency_ms": int((time.time()-t0)*1000), "error": repr(e)}
            r["id"] = c["id"]; r["model"] = a.model
            f.write(json.dumps(r, ensure_ascii=False) + "\n")
            print(c["id"], r["latency_ms"], "ms", r["error"] or "ok")

if __name__ == "__main__":
    main()

Now the part people skip. Dry-run the plumbing with the stub before you spend a single token:

python -m venv .venv && . .venv/bin/activate
python runner.py --model stub --out runs/stub.jsonl
head -1 runs/stub.jsonl

Verify: 20 lines, each valid JSON, no error field populated. If the stub run is broken, every number after this is noise. Fix it here, not later.

Step 4 — Grade from files, not from the chat window

python runner.py --model candA --out runs/candA.jsonl
python runner.py --model candB --out runs/candB.jsonl

Same grader code for both candidates. human cases go on a sticky note and get eyeballed once, after the automated numbers exist — otherwise you anchor yourself.

# grade.py
import json, re, sys

def grade(case, text):
    g = case["grader"]
    if g["type"] == "exact":   return text.strip() == g["expect"].strip()
    if g["type"] == "regex":   return re.search(g["expect"], text) is not None
    if g["type"] == "json_keys":
        try: obj = json.loads(text)
        except json.JSONDecodeError: return False
        return all(k in obj for k in g["expect"])
    return None  # human review

cases = {c["id"]: c for c in map(json.loads, open("cases.jsonl"))}
for path in sys.argv[1:]:
    rows = [json.loads(l) for l in open(path)]
    auto = [r for r in rows if grade(cases[r["id"]], r["text"]) is not None]
    passed = sum(1 for r in auto if grade(cases[r["id"]], r["text"]))
    errs = sum(1 for r in rows if r["error"])
    print(f"{path}: {passed}/{len(auto)} auto, {errs} errors")

Verify: grade the stub run. It should score a low, obviously wrong number. If the stub scores 20/20, your graders are too loose and you are about to ship a fake eval. That single check has caught more of my mistakes than any dashboard.

Step 5 — Decide with a rule you wrote before you looked

The order matters. Write the table first, run second, read it third.

Check	Fail condition	Action
Hard error rate	above 2/20	eliminate
Adversarial cases	any hard fail	eliminate
JSON validity	below 19/20	eliminate
Auto pass count	within 2 of the leader	tie, send to human review
p95 latency	above your client timeout	count as error

Those thresholds are illustrative, not published benchmarks. Pick yours from what your users will actually tolerate — not from what looks impressive in a table. A model that is 3% better and 4x slower is not a better model for a synchronous form field.

Where to run this without a bill

Two candidates, 20 cases, three repeats if you can afford it — that's 120 calls. Trivial on a paid API. Annoying on a personal card, which is exactly when people stop evaluating and start guessing.

That's where a free tier genuinely earns its place in the workflow: free model access supplies the candidates, and a free server option hosts the runner so the loop doesn't die because your laptop went to sleep. MonkeyCode offers both — free model access and a free server option, with the free allowance advertised at 10 million tokens. Treat those figures as operator-supplied and check the current limits in the project before you budget on them, because free-tier terms move.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Wire call_model to it, keep the stub path intact, and the harness stops caring who serves the tokens. That's the test of a decent eval harness: swapping the provider is a five-line diff, not a rewrite.

What this method does not tell you

Twenty cases is a smoke test, not a verdict. The confidence interval is wide enough to hide real differences, so resist the urge to declare a winner on a 2-case gap.

One run per case is noisy. Free tiers are also rate-limited, so repeats get slow — and repeated calls to the same endpoint may not be independent samples anyway.

Your graders are the weakest link. Regex over JSON is brittle; schema checks are better; anything about tone or writing quality needs a human and probably deserves its own rubric.

Contamination is real if your cases came from a public dataset. Write your own, from your own logs.

And it says nothing about tasks you didn't include. Change the prompt shape next month and re-run.

Who should not use this

If you already have an evaluation pipeline, you don't need a 50-line runner — add cases to what you have. If your failure mode is subtle prose quality that no regex can catch, hand-review is the honest method; don't dress it up in a pass rate. And if your output lands somewhere regulated, every result gets a human anyway, so optimize for review speed instead of model choice.

The monthly re-run

Freeze cases and graders in git. Re-run on the first of the month, diff the pass counts, and only then consider switching. The value isn't the score — it's that you can say why you switched, and notice when a model quietly gets worse without telling anyone.

Twenty cases. One stub run to prove the plumbing. One decision table you wrote before you peeked. Cheap, boring, and it survives contact with a real workload.

先给输入做哈希。只有当摘要在重跑时完全相同时，才信任这次免费运行。

Dakota Liu — Sun, 18 Oct 2026 07:14:18 +0000

同一个智能体任务跑两次，得到两份不同的输出，而你无法判断究竟是自己的代码变了，还是免费额度本身变了。这种情况我遇到得够多，也知道修起来并不贵。

答案不是换一个更强的模型。你需要一条流水线：每个阶段结束时都有一项检查，要么通过，要么把你拦下；再加上一个哈希，用来判断两次运行是否还处于可比状态。只要搭一次，免费基础设施就能从“说不清”变成“可以使用”。

下面是完整的搭建过程，按阶段推进：命令、代码，以及进入下一阶段前你应该看到的输出。Python 3.11、bash、jq、sha256sum。不用框架。

为什么“免费”会改变设计

免费模型调用和免费服务器意味着你可以把调用成本压到零，却让不确定性上升。这些资源可能很慢，可能在你最需要的时候不可用，也可能一夜之间改变运行方式。这不是要批评某个供应商，而是免费额度的本质就是这么运作的。

所以整条流水线的目标不是“每次都成功”，而是“绝不假装成功”。每一次运行要么产出一份带签名的 manifest，要么大声报错退出。如果是中间状态，就等于是 bug。

MonkeyCode 提供免费模型访问和免费服务器选项，这正是本次实验的落脚点：免费额度足够便宜，可以跑完整个循环，而不用先想预算。本文把这两点当作来自运营方的声明，免费额度条款变化很快，所以不要拿本文当合同；在计划一整周的工作之前，请先自行核对该项目的当前页面。

披露：本文是作为 MonkeyCode 产品推广工作的一部分撰写的。

第 0 步——搭好工作目录

mkdir -p mc-gate/{inputs,work,runs,prompts}
cd mc-gate
python3 -V

验证： python3 -V 打印版本号，且版本 >= 3.11。如果更老，就停下，本文后面的代码都假设 3.11 的 f-string 和 pathlib 行为。

目录约定很重要，因为后面运行的每一步都以摘要命名：runs/<digest>/ 同时存放计划、输出和 manifest。如果你无法仅凭摘要就能找到运行产物，那么摘要闸门也就失去意义了。

第 1 步——对输入做规范化并哈希

这是整件事的基础。不可比较的输入会产生不可比较的运行，后面的检查也都会失效。

# digest.py
import hashlib, json, pathlib, sys

VOLATILE = {"timestamp", "request_id", "trace_id"}

def strip(obj):
    if isinstance(obj, dict):
        return {k: strip(v) for k, v in obj.items()
                if k not in VOLATILE and not k.startswith("_")}
    if isinstance(obj, list):
        return [strip(v) for v in obj]
    return obj

def digest(obj):
    canon = json.dumps(strip(obj), sort_keys=True,
                       separators=(",", ":"), ensure_ascii=False)
    return hashlib.sha256(canon.encode("utf-8")).hexdigest()

if __name__ == "__main__":
    payload = json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))
    print(digest(payload))

注意那个 strip 函数，它才是真正的设计动作。时间戳、请求 ID 和任何以 _ 开头的键，都在你的流水线里属于副作用，而不是输入。把它们排除掉，同一次运行在隔天重跑时，摘要才会保持稳定。

验证：

python3 digest.py inputs/job.json
python3 digest.py inputs/job.json   # identical output

两次调用必须打印出同一个 64 字符的哈希。然后改一个真实的值，比如某个文件路径；哈希必须变化。再去加一个 response_ts 字段，哈希必须保持不变。如果这两条测试里有任何一条不通过，就不要继续往下走，后面所有检查都会继承这个 bug。

第 2 步——先让免费模型生成计划

向模型请求一份 JSON 计划，其他什么都不要。用 jq 来构造请求体，这样就不用把凭据或模型名称硬编码进脚本里。

jq -n --arg m "$MC_MODEL" \
      --arg p "$(cat prompts/plan.md)" \
  '{model:$m, temperature:0, response_format:{type:"json_object"},
    messages:[{role:"user",content:$p}]}' \
| curl -sS "$MC_BASE_URL/chat/completions" \
    -H "Authorization: Bearer $MC_API_KEY" \
    -H "Content-Type: application/json" -d @- \
| jq -r '.choices[0].message.content' > runs/plan.json

temperature: 0 只是一个请求，不是保证。把它理解为“请保持稳定”，而不是“一定会保持稳定”。这就是为什么我们在这里加闸门，而不是信任这个参数。

验证： jq -e '.steps | length > 0' runs/plan.json 以 0 退出。如果你拿到的是散文而不是 JSON，说明你的提示词协议有问题，不要试图在客户端把它补救回来。

第 3 步——在执行任何东西之前先给计划加闸门

一份计划是攻击面。它引用路径，它决定做什么。在任何步骤被实际执行之前，先加闸门。

# gate_plan.py
import json, pathlib, sys

OPS = {"read", "transform", "write"}

def validate(plan):
    if plan.get("version") != 1:
        raise ValueError("unsupported plan version")
    steps = plan.get("steps") or []
    if not steps:
        raise ValueError("empty plan")
    for i, s in enumerate(steps):
        if s.get("op") not in OPS:
            raise ValueError(f"step {i}: unknown op {s.get('op')!r}")
        p = s.get("path") or ""
        if not p.startswith("./work/"):
            raise ValueError(f"step {i}: path escapes work dir: {p}")
    return steps

plan = json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))
print(f"ok: {len(validate(plan))} steps")

验证： python3 gate_plan.py runs/plan.json 打印 ok: N steps。然后做一次反向验证：把某个路径改成 ./etc/passwd，再跑一次。它必须以非零退出码退出，并报出“path escapes work dir”。一个从不失败的闸门属于装饰，而不是控制。

第 4 步——在免费服务器上执行，并做到幂等

免费服务器会重启、休眠，或中途断开。所以每个运行目录都以摘要为键，一旦 manifest 已经存在，脚本就拒绝执行任何操作。

#!/usr/bin/env bash
set -euo pipefail

DIGEST="$(python3 digest.py inputs/job.json)"
RUN="runs/$DIGEST"
mkdir -p "$RUN"

if [ -f "$RUN/manifest.json" ]; then
  echo "already ran $DIGEST"
  exit 0
fi

cp runs/plan.json "$RUN/plan.json"
mkdir -p "$RUN/out"
# your executor reads $RUN/plan.json and writes into $RUN/out
./executor.sh "$RUN/plan.json" "$RUN/out"

jq -n --arg d "$DIGEST" --arg at "$(date -u +%FT%TZ)" \
  '{digest:$d, finished_at:$at, steps:[]}' > "$RUN/manifest.json"
echo "done $DIGEST"

验证： 运行一次，你得到 done <digest>。第二次运行，你得到 already ran <digest>，退出码为 0。这就是幂等性，也是长时间断线重连能够安全的前提。

第 5 步——重跑，并比较结构而不是措辞

生成第二份计划，逐字段比较。模型生成的自然语言每次都会变，所以拿散文去 diff 纯属浪费下午。比较形状。

jq -S '[.steps[] | {op, path}]' runs/plan.json       > runs/a.json
jq -S '[.steps[] | {op, path}]' runs/plan.rerun.json > runs/b.json
diff -u runs/a.json runs/b.json && echo "structure stable"

验证： 输出 structure stable，diff 退出码为 0。如果 diff 显示某个步骤消失了或重新排序了，你有两种诚实的选择：把计划确定下来，或者接受这份任务是随机性的，并在下游做补偿。不要走第三条路，也就是不断调参数直到 diff 看起来干净为止。

第 6 步——提升或者拒绝

在最后加上一张决策表，这样这个问题就由表格来回答，不用再靠某个人临场判断。

观察到的情况	操作	原因
相同摘要，结构 diff 干净	提升产物	这次运行可复现
相同摘要，结构 diff 不一致	修复计划提示词，重跑	模型不是变量，你的提示词才是
输入变化后新出现摘要	新运行，不覆盖旧运行	历史保持可审计
manifest 缺失或不完整	拒绝成果	无法审计的运行等同于猜测

验证： 提升步骤应当拒绝任何没有 manifest 的目录。用 rm runs/*/manifest.json 测试，确认它拒绝。

限制，以及谁应该跳过这套做法

免费额度条款会变。如果一篇文章精确固定了这些条款，它可能已经过时——请检查项目的当前页面。
对于天然带随机性的步骤，摘要闸门帮不上忙。它最多只能告诉你两次运行并不等价。
如果你的合规团队需要明确的数据留存保证，那么在阅读明确的数据策略之前，不要用第三方服务处理真实数据。
不要用免费额度当作付费模型延迟的基准。那是另一种测量，而这次的运行本来就不是为那种测量设计的。
如果你需要硬性正常运行时间 SLO，这套做法就是错的工具。这套做法的全部前提，就是让失败变得可审计。

这套做法的目标读者是谁？ 是在免费层上跑智能体任务、并需要为结果辩护的人。

结尾

这套做法里没有任何一项依赖某个特定供应商。摘要、闸门、manifest，在任何地方都成立，也正是它们让免费资源可以安全使用。如果你想让这套 harness 有个一运行成本为零的起点，MonkeyCode 的免费模型访问和免费服务器选项就是最早跑通循环的合理落脚点——但请先完整跑一遍第 1 步里的正反两个测试。

如果你运行了它，并且摘要闸门抓住了你的代码本来会漏掉的问题，那就是值得分享的部分。是哪一个阶段？

MonkeyCode provides free models that can run this workflow. A free server option is enough to reproduce the setup.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Measure the Escalation Rate. A Local-First Router That Never Earns Back Is Just Latency.

Dakota Liu — Fri, 16 Oct 2026 05:17:13 +0000

A local-first router only pays off if tier 1 absorbs most of your traffic. If your local model answers 20% of prompts and escalates the rest, you didn't build a cost saver. You built a queue.

So here's the order I'd do it in: build the meter before you build the router. Then let the meter tell you whether the cheap tier deserves to exist.

This is a from-zero tutorial. Every stage ends with a verification step, and the whole thing runs against a fake provider for free before you point it at anything real.

The problem, in one number

Everyone routes local-first because it sounds frugal. Few people measure the escalation rate — the share of prompts that fell through tier 1 and had to hit a remote model anyway.

That number decides everything. High escalation rate? You pay the latency twice and the tokens once. Low escalation rate? The cheap tier is doing real work and you should protect it.

Step 1 — Freeze a replay corpus

You cannot tune a router against prompts you make up on the spot. Freeze real ones.

I take 20–40 prompts from actual logs, strip anything sensitive, and write one JSON object per line:

{"id":"c001","prompt":"Summarize this incident timeline and name the top risk."}
{"id":"c002","prompt":"Turn this changelog into three release notes."}

Verify: wc -l corpus.jsonl matches the number of prompts you think you froze, and every line parses with node -e 'require("fs").readFileSync("corpus.jsonl","utf8").split("\n").filter(Boolean).forEach(l=>JSON.parse(l))'. Boring check. Do it anyway — a broken corpus makes every later number a lie.

Step 2 — Decide what "accepted" means, in code

This is the part people skip, and it's the reason their dashboards are useless. You need a deterministic proxy for "was this answer usable?"

Mine is a schema check plus a refusal check. It is a proxy, and it will accept things a human would throw away. Say that out loud in the code comment so nobody treats it as ground truth.

// accept.mjs — a deterministic proxy, not a judge.
const REQUIRED = ["summary", "risk", "next_step"];

export function accept(text) {
  if (typeof text !== "string" || text.length < 40) return { accepted: false, why: "too_short" };

  let obj;
  try { obj = JSON.parse(text); }
  catch { return { accepted: false, why: "not_json" }; }

  const missing = REQUIRED.filter((k) => !(k in obj));
  if (missing.length) return { accepted: false, why: `missing:${missing.join(",")}` };

  // Truncated output often still parses. Length + required keys is a cheap tripwire.
  if (/i cannot|as an ai/i.test(text)) return { accepted: false, why: "refusal" };
  return { accepted: true, why: "schema_ok" };
}

Verify: feed it one good sample and one truncated sample and confirm you get schema_ok and not_json. If truncation passes your check, your escalation meter is measuring nothing.

Step 3 — One ledger row per attempt, grouped by correlation id

Every model call writes a row. Every row carries the same correlation_id as the other tiers tried for that prompt. That's the join key your meter groups on.

// ledger.mjs — append-only JSONL. One row per attempt, no exceptions.
import { appendFileSync, existsSync, readFileSync } from "node:fs";

const LEDGER = process.env.LEDGER ?? "runs/ledger.jsonl";

export function record(entry) {
  // { ts, correlation_id, tier, model_ref, tokens_in, tokens_out, finish_reason, outcome, accepted }
  appendFileSync(LEDGER, JSON.stringify(entry) + "\n");
}

export function readAll() {
  if (!existsSync(LEDGER)) return [];
  return readFileSync(LEDGER, "utf8").split("\n").filter(Boolean).map((l) => JSON.parse(l));
}

One rule matters here: the local tier logs accepted too. If tier 1 succeeded, that's a row with tier:"local", accepted:true and zero remote tokens. If you only log remote calls, your denominator disappears.

Verify: run one prompt end to end, then cat runs/ledger.jsonl | jq -c . and confirm every row for that prompt shares one correlation_id.

Step 4 — The meter, with a kill threshold

Now the interesting file. It reads the ledger and answers a single question: is tier 1 absorbing work, or adding latency?

// meter.mjs — node meter.mjs --ledger runs/ledger.jsonl --kill-threshold 0.5
import { readFileSync } from "node:fs";

const args = process.argv.slice(2);
const arg = (name, fallback) => {
  const i = args.indexOf(name);
  return i === -1 ? fallback : args[i + 1];
};

const LEDGER = arg("--ledger", "runs/ledger.jsonl");
const KILL = Number(arg("--kill-threshold", 0.5));

const rows = readFileSync(LEDGER, "utf8").split("\n").filter(Boolean).map((l) => JSON.parse(l));

const runs = new Map();
for (const r of rows) {
  if (!runs.has(r.correlation_id)) runs.set(r.correlation_id, []);
  runs.get(r.correlation_id).push(r);
}

let tier1 = 0, escalated = 0, accepted = 0, remoteTokens = 0;

for (const attempts of runs.values()) {
  const first = attempts[0];
  const last = attempts[attempts.length - 1];
  if (first.tier === "local") tier1 += 1;
  if (attempts.length > 1) escalated += 1;
  if (last.accepted) {
    accepted += 1;
    remoteTokens += attempts
      .filter((a) => a.tier !== "local")
      .reduce((n, a) => n + (a.tokens_in ?? 0) + (a.tokens_out ?? 0), 0);
  }
}

const rate = tier1 ? escalated / tier1 : 0;
console.table({
  tier1_attempts: tier1,
  escalated,
  escalation_rate: rate.toFixed(3),
  accepted_answers: accepted,
  remote_tokens_per_accepted: accepted ? (remoteTokens / accepted).toFixed(1) : "n/a",
});

if (rate > KILL) {
  console.error(`escalation rate ${rate.toFixed(3)} > ${KILL}: tier 1 is not earning its latency`);
  process.exit(1);
}

Verify: seed a fake ledger with one local-only success and two escalations. You should see escalation_rate 0.667 and exit code 1. That exit code is what you wire into CI, and it's the whole point — the router fails its own performance review.

The criteria I'd start with, and expect to change once I have real numbers:

7-day escalation rate	Read	Action
under 0.20	Tier 1 carries the load	Keep it, and shrink the remote budget
0.20 – 0.50	Mixed	Group failures by `outcome` before touching prompts
over 0.50	Tier 1 mostly costs latency	Demote it to cache-only, or delete it
over 0.80 with rising tokens per accepted answer	You're paying twice	Kill tier 1 this week

The middle row is where most teams actually sit. Don't guess your way out of it — group by outcome first. Rate limits, timeouts, and schema failures are three different diseases.

Step 5 — Swap in the real tiers

The plumbing is proven against a fake provider, so now you change two things and nothing else: the tier-2 base URL and key. Mint a route to a prompt.

This is where MonkeyCode's free model access fits in my own setup: it's the tier-2 endpoint, so it's the thing my escalation rate is measured against. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I can honestly state is limited to what the operator supplied: there is free model access, and there is a free server option. The operator-supplied figure for the free allotment is 10M tokens. I have not independently verified that number, quotas move, and you should confirm the current terms from the project's own documentation before you plan around it. This post is not a source of truth for pricing.

Verify: run the identical corpus against the real tier and diff the ledger outcome sequence against the mock run. Same number of rows, same ordering, or you learned something real about the remote endpoint that no blog post would have told you.

Step 6 — Run the meter on the free server, on a schedule

A one-off escalation rate is an anecdote. A daily one is a trend.

The operator lists a free server option, so I put the meter there and let a scheduled job append one summary row per day: date, escalation rate, tokens per accepted answer. I have not benchmarked that server — sizing, cold starts, and how long the option stays available are unknown to me. Treat them as unmeasured until you measure them yourself.

Verify: after three days you should have three summary rows. If the schedule silently stopped, tokens per accepted answer won't move and you'll notice.

What this does not prove

The acceptance proxy is a schema check. It accepts answers a human would reject, so your escalation rate is a floor, not a ceiling.
appendFileSync on a shared ledger is fine for a single process and unsafe for concurrent writers. Add a lock or ship to a log sink before you parallelize.
Nothing here shows the remote tier is fast, permanent, or unlimited. Free tiers are not SLAs, and this design assumes the remote side can disappear.
The mock's token counting is irrelevant by design — only provider-reported usage belongs in the ledger, and providers sometimes omit it. Log null and treat it as unknown rather than as zero.

Who should not do this

Skip all of it if you make ten calls a day — just call the model and move on. Skip it if you need a latency SLA, because a router with two hops is slower than one. And skip it if the prompts contain regulated data, since tier 2 is a third party by definition.

But if you already route local-first and you have never printed the escalation rate, that's the bug. Build the meter tonight, run it against a fake provider, and let the exit code decide whether your cheap tier stays.

If you run it and tier 1 gets killed by your own threshold, I want to hear which outcome did it.

Publish the Stub First. Wire the Free Model Only After the Remote Passes.

Dakota Liu — Thu, 15 Oct 2026 11:26:01 +0000

Free model access and a free server are cheap the way a blank notebook is cheap: money is not the constraint, attribution is. When the fourth stage breaks, you want to know whether the prompt, the deploy, or the network broke — without re-running everything.

So the whole tutorial below is one sentence: every stage emits evidence, and that evidence is what lets you blame the right stage later. Six checkpoints, four files, one verifier.

Yes, a lot of this week's front-page discussion was about AI fatigue and about models that agree with everything. Fair. This post stays small on purpose: one endpoint, one JSON shape, no dashboards, no orchestration.

The slice you are building

One HTTP endpoint, GET /answer, returns one frozen object. That is all.

gate-demo/
├── contract/response.schema.json   # what you will accept
├── prompt/extract.v1.txt           # what you will send
├── stage1_stub.py                  # what you deploy first
└── verify.py                       # the only harness you get

Notice there is no framework in that tree. Standard library only, so when something fails it is yours and not a dependency's.

Stage 0 — Freeze the contract before anything else exists

Step 1. Write the shape you will accept. Not the shape you hope for — the shape you will reject everything else against.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": ["subject", "confidence"],
  "properties": {
    "subject": { "type": "string", "minLength": 1, "maxLength": 80 },
    "confidence": { "type": "number", "minimum": 0, "maximum": 1 }
  }
}

Step 2. Prove the file is parseable at all:

mkdir -p contract && $EDITOR contract/response.schema.json
python -c "import json; json.load(open('contract/response.schema.json'))" && echo GATE0_OK

Gate 0 is deliberately weak. Parsing proves the file is JSON. It does not prove your service obeys it. That is Stage 1's job, and mixing the two is how you end up arguing with your own schema at midnight.

Stage 1 — Prove a stub locally, with no model and no network

Step 1. The stub returns a canned body, sorted, with a Content-Length:

# stage1_stub.py — no model, no network, one frozen response
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

BODY = json.dumps({"subject": "stub", "confidence": 1.0}, sort_keys=True).encode()

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/answer":
            self.send_response(404); self.end_headers(); return
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(BODY)))
        self.end_headers()
        self.wfile.write(BODY)

    def log_message(self, *args):
        pass

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8080), Handler).serve_forever()

Step 2. Write the only harness you will use for the rest of the article:

#!/usr/bin/env python3
# verify.py <url> <stage> [--require-stable]
# Exit 0 = gate passed. 2 = schema broken. 3 = bytes unstable.
import hashlib, json, sys, urllib.request

RUNS = 3

def check(obj):
    problems = []
    if not isinstance(obj.get("subject"), str) or not obj.get("subject"):
        problems.append("subject_invalid")
    c = obj.get("confidence")
    # isinstance(True, int) is True in Python; exclude bool explicitly.
    if isinstance(c, bool) or not isinstance(c, (int, float)):
        problems.append("confidence_not_number")
    elif not 0 <= c <= 1:
        problems.append("confidence_out_of_range")
    extra = set(obj) - {"subject", "confidence"}
    if extra:
        problems.append(f"extra_keys:{sorted(extra)}")
    return problems

def fetch(url):
    with urllib.request.urlopen(url, timeout=10) as r:
        raw = r.read()
    return hashlib.sha256(raw).hexdigest()[:12], json.loads(raw)

def main():
    url, stage = sys.argv[1], sys.argv[2]
    require_stable = "--require-stable" in sys.argv
    digests, failures = [], []
    for i in range(RUNS):
        digest, obj = fetch(url)
        digests.append(digest)
        failures += [f"run{i}:{p}" for p in check(obj)]
    stable = len(set(digests)) == 1
    print(f"stage={stage} digests={digests} byte_stable={stable} problems={failures}")
    if failures:
        return 2
    if require_stable and not stable:
        return 3
    return 0

if __name__ == "__main__":
    sys.exit(main())

Step 3. Gate it:

python stage1_stub.py &
python verify.py http://127.0.0.1:8080/answer 1 --require-stable; echo "exit=$?"

Why is byte stability a hard gate here and not later? Because a canned response must be identical three times out of three. If it is not, your harness is broken, not an AI. Finding that out on a stub costs nothing. Finding it out after you wired a model call costs an afternoon of suspicion.

Stage 2 — Version the prompt as a file, then hash it

Step 1. The prompt is an artifact, not a string in your code:

mkdir -p prompt cases
printf 'Extract the single subject of the text below.\nReturn only JSON: {"subject": string, "confidence": number}\nTEXT:\n' > prompt/extract.v1.txt
sha256sum prompt/extract.v1.txt | tee prompt/extract.v1.sha256

Step 2. Re-check it before every run:

sha256sum -c prompt/extract.v1.sha256

That is the whole gate, and it is boring on purpose. When it fails, a human edited the prompt, and you can stop pretending the service changed its mind. Put inputs in cases/, never a customer's text, and never inside prompt/.

Stage 3 — Wire one narrow model call behind the same endpoint

This is where free model access earns its place: one call, one shape, no tools, no loop, no agent. MonkeyCode is the project I used for that step here, and the free server option is what Stage 4 below uses. Both are availability claims from the project, not benchmarks from me. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 1. Your call wrapper — deliberately pseudocode, because the SDK differs:

# call_model.py — PSEUDOCODE. Swap `Client` for your provider's SDK.
# No model name is hardcoded: the run record names it, not this file.
import hashlib, json, os, pathlib, sys

prompt_path = pathlib.Path("prompt/extract.v1.txt")
record = {
    "prompt_sha256_12": hashlib.sha256(prompt_path.read_bytes()).hexdigest()[:12],
    "input_case": sys.argv[1],
}
text = pathlib.Path(sys.argv[1]).read_text()
resp = Client(api_key=os.environ["MODEL_KEY"]).complete(
    prompt=prompt_path.read_text() + text,
    temperature=0,
)
record["model"] = resp.model_id      # recorded, never assumed
record["raw"] = resp.text
print(json.dumps(record, indent=2))

Step 2. Change one line in stage1_stub.py: the canned BODY becomes the model output. Same endpoint, same port, same verifier:

MODEL_KEY=... python stage3_app.py &
MODEL_KEY=... python verify.py http://127.0.0.1:8080/answer 3; echo "exit=$?"

Step 3. Read the gate honestly. At this stage, schema validity is 3/3 or you stop. Byte stability is printed, not gated. With temperature 0 some providers come back byte-identical and some do not; I am not going to promise what yours does. And if the call fails schema, do not retry — log the failing case, because that file is the first row of your eval set.

Stage 4 — Publish the unchanged stub to the free server

The important word is unchanged. You already know the stub works locally. If it fails remotely, the failure is the host, not the model.

Step 1. Copy only the two files that contain no secrets:

scp contract/response.schema.json stage1_stub.py user@instance:~/gate-demo/

Step 2. Start it on the instance and check it from inside the box:

cd ~/gate-demo && (nohup python3 stage1_stub.py &> stub.log &)
curl -s localhost:8080/answer

Step 3. Gate it from your laptop, with the same verifier as Stage 1:

python verify.py https://<instance-host>/answer 4 --require-stable; echo "exit=$?"

Gate 4 means: same bytes, same schema, different machine. Do not copy credentials with scp; set MODEL_KEY as an environment variable on the instance later. How you expose a port depends on the free server you picked — read that page, because nothing here assumes one.

Stage 5 — Move the model call to the remote, still one line

Now swap the stub for the app.py variant, set the key, restart, and re-run the verifier:

python verify.py https://<instance-host>/answer 5; echo "exit=$?"

Schema 3/3 is the gate. Latency is not. Time five sequential requests yourself and write down the numbers you measured on your connection — mine would say nothing about yours, so I am not handing you any.

Every gate in one table

Stage	What it proves	Hard gate	Recorded only
0	the contract file parses	JSON parse OK	—
1	harness and endpoint work at all	schema 3/3, bytes stable	latency
2	the prompt is a versioned artifact	hash matches file	—
3	model output fits the contract	schema 3/3	byte stability, model id, raw text
4	hosting is not the broken part	schema 3/3, bytes stable	cold-start time
5	both halves work together	schema 3/3	five latency samples

Where this breaks, and who should skip it

Data that cannot leave your boundary. A hosted free model tier means the prompt leaves your machine. If that is a compliance problem for you, stop at Stage 2 and point the same endpoint at a local model.
Workloads that need persistence. If state must survive a restart, a free instance is the wrong place to discover that. Read the provider's current terms for what the free tier includes on the day you deploy.
Teams that need pinned model versions. This workflow records the model id per run; it does not guarantee tomorrow's call returns the same model.
Anyone who wants load numbers. Three requests plus a hand-typed sample is a smoke test, not a benchmark.
Anyone shipping this to production without an eval set. Stage 3 proves shape, not correctness.

One more: quotas and plans move. If you are budgeting, read the product's current plan page rather than any blog post, this one included.

The one thing to keep

Keep Stage 0's contract file byte-identical across all six stages. The stub, the prompt, the host, and the model are all replaceable. The contract is the part that tells you which stage broke when something does.

If you want to run Stages 3 through 5 against a free model call and a free server instead of wiring your own, that is the setup MonkeyCode offers and what I used above. Start with the contract file, and see what it says the first time something fails.

Seal the Write Fence. Abort If a Forbidden Path Moves.

Dakota Liu — Tue, 13 Oct 2026 01:44:33 +0000

I do not let a free model, or a free server, write a working tree until a local fence names every path that may change. If a later stamp shows a forbidden path moved, the apply is dead. That is the whole policy.

Why so strict? A cheerful diff is not a safe diff. Have you ever accepted a "small fix" and then found a lockfile, a sample env file, or a migration you never mentioned? I stopped treating that as a surprise. It is what happens when the blast radius lives in a prompt instead of a file.

Recent developer chatter keeps circling the same bruise. Should you retry a confident model answer, or is confidence just a fluent guess? I am not settling that debate here. I am narrowing a smaller question. Which files were allowed to move, and can I prove the others stayed put?

This is a from-zero tutorial on a scratch repository. You will build a fence, a pre-image stamp, a redacted task packet, a plan checker, and a post-image verify. Each stage has a command and a pass line. Miss the pass line and you stop. The script is a proposal until you run it. I am not publishing timings, and I am not inventing a model name or a quota.

What you are actually proving

You are not proving the model is smart. You are proving the tree stayed inside a list you wrote while you were still careful. Does that sound unglamorous? It should. Glamour is how a secret-shaped file ends up in a patch.

Skip this if the real task is "change every file." An allowlist of everything is not a fence. Skip it if you cannot throw the workspace away. A checker is not a backup, and a green stamp is not a code review.

Step 1. Create a tiny tree and a boring fence

I start with three files on purpose. Two may change. One must never move. Would you merge a patch you cannot name file by file? Then do not start the job.

mkdir -p scratch/src scratch/notes
printf 'def add(a, b):\n    return a + b\n' > scratch/src/calc.py
printf 'draft\n' > scratch/notes/todo.md
printf 'TOKEN=local-only\n' > scratch/.env

The fence is JSON, not a sentence buried in a prompt. Prompts get negotiated. Files do not.

cat > fence.json << 'EOF'
{
  "root": "scratch",
  "allow": ["src/calc.py", "notes/todo.md"],
  "deny_globs": [".env", "*.pem", ".git/*"]
}
EOF
python3 -c 'import json; d=json.load(open("fence.json")); assert d["allow"]; print("fence_ok", len(d["allow"]))'

Verification: you must see fence_ok 2. If the assert fires, the allowlist is empty. Stop. An empty allowlist is how "tidy this up" becomes "touch anything."

Step 2. Save the checker and compile it

Drop the following into write_fence.py. It only reads the local tree and writes JSON. It does not open a socket. That is deliberate. Any network step comes later, and it stays optional.

#!/usr/bin/env python3
from __future__ import annotations

import argparse
import hashlib
import json
import sys
from fnmatch import fnmatch
from pathlib import Path


def iter_files(root: Path):
    for path in sorted(root.rglob("*")):
        if path.is_file() and ".git" not in path.parts:
            yield path


def digest(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def stamp(root: Path) -> dict:
    files = {p.relative_to(root).as_posix(): digest(p) for p in iter_files(root)}
    return {"root": root.as_posix(), "files": files}


def load_json(path: Path) -> dict:
    try:
        return json.loads(path.read_text())
    except json.JSONDecodeError as exc:
        raise SystemExit(f"bad_json:{path}:{exc}") from exc


def is_denied(rel: str, fence: dict) -> bool:
    allow = fence.get("allow", [])
    if rel in allow:
        return False
    if any(fnmatch(rel, pattern) for pattern in fence.get("deny_globs", [])):
        return True
    return rel not in allow


def check_plan(fence: dict, plan: dict) -> list[str]:
    writes = plan.get("writes")
    if not isinstance(writes, list) or not writes:
        return ["plan_has_no_writes"]
    errors = []
    for item in writes:
        rel = item.get("path") if isinstance(item, dict) else None
        if not rel or rel.startswith("/") or ".." in Path(rel).parts:
            errors.append(f"unsafe_path:{rel}")
            continue
        if is_denied(rel, fence):
            errors.append(f"outside_fence:{rel}")
    return errors


def verify(root: Path, fence: dict, before: dict) -> list[str]:
    disk = stamp(root)["files"]
    errors = []
    keys = set(before.get("files", {})) | set(disk)
    for rel in sorted(keys):
        prior = before.get("files", {}).get(rel)
        if prior != disk.get(rel) and is_denied(rel, fence):
            errors.append(f"denied_path_moved:{rel}")
    return errors


def main() -> int:
    parser = argparse.ArgumentParser()
    sub = parser.add_subparsers(dest="cmd", required=True)

    stamp_cmd = sub.add_parser("stamp")
    stamp_cmd.add_argument("root")
    stamp_cmd.add_argument("out")

    plan_cmd = sub.add_parser("check-plan")
    plan_cmd.add_argument("fence")
    plan_cmd.add_argument("plan")

    verify_cmd = sub.add_parser("verify")
    verify_cmd.add_argument("root")
    verify_cmd.add_argument("fence")
    verify_cmd.add_argument("before")

    args = parser.parse_args()
    if args.cmd == "stamp":
        payload = stamp(Path(args.root))
        Path(args.out).write_text(json.dumps(payload, indent=2) + "\n")
        print(f"stamp_ok files={len(payload['files'])}")
        return 0
    if args.cmd == "check-plan":
        errors = check_plan(load_json(Path(args.fence)), load_json(Path(args.plan)))
        if errors:
            print("plan_rejected")
            print("\n".join(errors))
            return 2
        print("plan_accepted")
        return 0
    errors = verify(Path(args.root), load_json(Path(args.fence)), load_json(Path(args.before)))
    if errors:
        print("verify_failed")
        print("\n".join(errors))
        return 3
    print("verify_ok")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Compile it before you trust it. A broken gate is worse than no gate, because you will lean on it.

python3 -m py_compile write_fence.py && echo compile_ok

Verification: compile_ok. If this fails, fix the script. Do not invent a plan on top of a checker that will not even import.

Step 3. Stamp the pre-image

A fence without a before-image is a wish. I hash every file under scratch/, including the one the fence denies. Otherwise I cannot prove it stayed still.

python3 write_fence.py stamp scratch pre-stamp.json
python3 -c 'import json; d=json.load(open("pre-stamp.json")); assert ".env" in d["files"]; print("pre_stamp_ok", len(d["files"]))'

Verification: the tool prints stamp_ok files=3, then the one-liner prints pre_stamp_ok 3. Why assert that .env is in the stamp? Because the file I most want to protect needs a before-hash. A fence that never recorded it cannot tell you it changed.

Short pause. If this stamp disappeared, would you still send the task? I would not.

Step 4. Build a task packet that omits denied bodies

The model, if you call one at all, gets a packet. It does not get the tree. The packet names allowed paths, names forbidden paths, and copies hashes only for the allowlist. No file bodies from denied paths.

python3 - << 'PY'
import json
pre = json.load(open("pre-stamp.json"))
fence = json.load(open("fence.json"))
allow = fence["allow"]
packet = {
    "task": "Add a one-line docstring to add(). Return JSON with a writes list. Do not apply it.",
    "may_write": allow,
    "must_not_change": [path for path in pre["files"] if path not in allow],
    "pre_hashes": {path: pre["files"][path] for path in allow},
}
json.dump(packet, open("task.json", "w"), indent=2)
print("task_packet_ok")
PY
grep -q TOKEN task.json && echo leaked || echo task_redacted_ok

Verification: task_packet_ok, then task_redacted_ok. If you see leaked, the packet is unsafe. Stop and rebuild it. I do not want a draft call to be the moment a local token leaves the disk.

Step 5. Reject a plan that walks outside the fence

A plan is not an apply. It is a list of intended writes. Here is a bad one. It looks helpful. It is not.

cat > plan-bad.json << 'EOF'
{"writes":[{"path":"src/calc.py","note":"add a docstring"},{"path":".env","note":"switch token"}]}
EOF
python3 write_fence.py check-plan fence.json plan-bad.json; echo "exit:$?"

Verification: output contains plan_rejected and outside_fence:.env, and the exit code is 2. If this plan is accepted, stop and fix is_denied. The rest of the tutorial is meaningless if the bad plan passes.

Now the narrow plan. Same task. No extra path.

cat > plan-ok.json << 'EOF'
{"writes":[{"path":"src/calc.py","note":"document the return"}]}
EOF
python3 write_fence.py check-plan fence.json plan-ok.json; echo "exit:$?"

Verification: plan_accepted and exit 0. Only then may a model draft replace plan-ok.json. Not before. Absolute paths and .. segments must fail too. Try "path": "../scratch/.env" if you want a second negative test. I keep that fixture next to the happy path so a future edit cannot quietly widen the gate.

Step 6. Optional draft, then an optional disposable apply

This is the first point where a hosted assistant belongs in the workflow. I sometimes ask for a JSON plan, and only a JSON plan, after pre_stamp_ok and a rejected bad fixture both exist. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator-supplied claim I will repeat is narrow: MonkeyCode has free model access and a free server option. I am not stating a token allotment, a model id, a machine size, or an expiry. Those details move. A tutorial should not freeze a number I did not re-check against a primary page today.

How would I use that, if the current docs still match? I would send task.json, not the repo, and I would demand a writes list back. I would not ask the model to apply the edit itself. If check-plan exits 2, I still learned something. The draft failed closed. I would not boot a server to see what a rejected plan does when it is allowed to touch disk.

A free server, when you actually want one, is a place to apply an already accepted plan in a workspace you can delete. It is not a reason to skip the stamp. Apply src/calc.py only. Then verify against the pre-image. The checker reads the live tree, so the post-stamp is whatever is on disk now.

printf 'def add(a, b):\n    """Return the sum."""\n    return a + b\n' > scratch/src/calc.py
python3 write_fence.py verify scratch fence.json pre-stamp.json; echo "exit:$?"

Verification: verify_ok and exit 0. The allowed file changed. That is fine. The denied file did not. If you did the apply on a disposable server workspace instead of this laptop, run the same verify there, or copy the tree back and verify locally. Same rule either way.

Step 7. Prove the checker catches a sneaky write

Do not trust a happy path you have not broken on purpose. What exactly did you allow? One function file. So a one-byte edit to .env must fail the job, even if the function looks perfect.

printf 'TOKEN=changed\n' > scratch/.env
python3 write_fence.py verify scratch fence.json pre-stamp.json; echo "exit:$?"

Verification: verify_failed, a line denied_path_moved:.env, and exit 3. Restore the file or delete the scratch tree. Do not keep a failed workspace because the docstring looks nice.

If a free server produced that tree, I would delete the workspace and return to the pre-image. I would not hand-edit the forbidden file back and call the apply clean. If the free server is disposable, why keep a result that already broke the fence?

The decision table I keep beside the script

Check	Pass line	If it fails
Fence parses and allowlist is non-empty	`fence_ok 2`	Stop. Do not draft a prompt.
Script compiles	`compile_ok`	Stop. A broken gate is not a warning.
Pre-image contains denied files	`pre_stamp_ok 3`	Stop. You cannot prove a later move.
Task packet has no denied body	`task_redacted_ok`	Stop. Rebuild the packet.
Bad plan fixture	exit `2` and `outside_fence:.env`	Stop. The checker is lying.
Accepted plan	`plan_accepted`	Do not apply. Ask for a narrower plan.
Live tree after apply	`verify_ok`	Discard the workspace. Do not merge.

That table is the artifact. The script is how I execute it. Neither one scores a model. They score whether I stayed inside a list I wrote before the call.

Limits, and who should walk away

Inside an allowed path, this proves nothing about behavior. verify_ok can still ship a wrong add. You want a unit test for that. This tutorial does not replace one.

fnmatch is not gitignore. A sloppy glob is a hole. Symlinks, hard links, and a second process writing the same tree are out of scope. I would not point this at a live checkout that a watcher is also mutating.

I did not run this against a live hosted endpoint while writing. If you try free model access or a free server, confirm the current terms on the project page first. Do not treat a blog post as a pricing sheet. Availability can change, and I would rather you see a 404 than trust a frozen claim.

Who should not use the approach? Anyone whose allowlist is the whole repo. Anyone who needs new top-level files in the same breath — extend allow, re-stamp, then ask. Anyone looking for a secret scanner. Denying .env is a policy choice, not detection. And anyone who will override exit code 3 because a deadline moved. The fence only works if a failed verify is allowed to waste the afternoon.

If a fenced scratch repo is the experiment you wanted anyway, read the current MonkeyCode free-model and free-server notes yourself, then run Steps 1 through 7 before you send anything you would hate to see rewritten.

Pin the Failure Class. Refuse the Free Retry When the Label Flips.

Dakota Liu — Sun, 11 Oct 2026 21:42:52 +0000

A remote retry is a decision, not a reflex. If two local runs do not share one failure class, I do not boot a box, and I do not send the fixture anywhere.

Did the error change because the code changed, or because the check is unstable? Answer that on disk first. This tutorial builds a small gate you can run from an empty directory. The script is a proposal. I have not timed it against a live service for this draft, and I will not pretend otherwise.

What "working" means here

You finish with a ledger file, not a vibe. The ledger says fix_locally, hold, or eligible_for_isolated_server.

Each step below has a command and a check. If the check fails, you stop. You do not skip ahead to a model just because the call is free.

Free is not the same as harmless. A free call can still leak a path, a token, or a half-written secret into someone else's log. A free server can still see whatever you mount. So the first artifact stays local.

Step 1. Create the scratch tree and prove the interpreter

Start from nothing. A dirty checkout will mix old failures into the new label, and then you will classify a ghost.

mkdir -p "$HOME/failclass/fixtures" "$HOME/failclass/ledger"
cd "$HOME/failclass"
python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -c "import sys; print(sys.prefix)"

Verification: the printed prefix must contain failclass/.venv. If it points at /usr or a conda root, deactivate and start over.

Why so fussy? Because a later non-zero exit from the wrong interpreter is not the failure you think you captured.

Write one boring fixture. A missing key is enough. You already know the cause, and that is the point.

# fixtures/case_missing_key.py
def load_budget(doc):
    return int(doc["remaining"])

if __name__ == "__main__":
    load_budget({})

Run it.

python fixtures/case_missing_key.py; echo "exit=$?"

You want a KeyError and a non-zero exit. If the exit is 0, this is not a failure fixture. Replace it. Do not classify a success.

Step 2. Freeze a four-label vocabulary

I keep the vocabulary rude and short. A long taxonomy is how a retry sneaks through while you are still naming things.

Class	What you saw locally	Send to a model?	Boot a server?
`local_typo`	SyntaxError, NameError, missing import	No	No
`contract`	KeyError, TypeError, assertion on your schema	No	No
`needs_runtime`	Fails only when a port, service, or path layout is required	Only after the class is stable	Yes, one fixture, nothing else
`unknown`	The two runs disagree, or you cannot say the cause in one sentence	No	No

Need a fifth label? Add it in the file, commit the change, and re-capture both runs. Do not invent the label in a comment you will forget.

The classifier runs the script twice, maps the exception, and writes ledger/latest.json. This is unexecuted proposal code. Read it before you trust it.

# classify_failure.py
import hashlib
import json
import re
import subprocess
import sys
from pathlib import Path

CLASSES = {
    "SyntaxError": "local_typo",
    "NameError": "local_typo",
    "ModuleNotFoundError": "local_typo",
    "KeyError": "contract",
    "TypeError": "contract",
    "AssertionError": "contract",
    "ConnectionRefusedError": "needs_runtime",
    "FileNotFoundError": "needs_runtime",
}

def run_once(script: str) -> dict:
    proc = subprocess.run(
        [sys.executable, script],
        capture_output=True,
        text=True,
        timeout=15,
    )
    err = proc.stderr or ""
    match = re.search(r"^(\w+Error):", err, re.M)
    exc = match.group(1) if match else "NoException"
    digest = hashlib.sha256(err.encode()).hexdigest()[:16]
    return {
        "script": script,
        "exit": proc.returncode,
        "exc": exc,
        "class": CLASSES.get(exc, "unknown"),
        "stderr_sha256_16": digest,
    }

def decide(first: dict, second: dict) -> str:
    if first["class"] != second["class"] or first["class"] == "unknown":
        return "hold"
    if first["class"] == "needs_runtime":
        return "eligible_for_isolated_server"
    if first["class"] in {"local_typo", "contract"}:
        return "fix_locally"
    return "hold"

def main() -> int:
    script = sys.argv[1]
    first = run_once(script)
    second = run_once(script)
    action = decide(first, second)
    record = {
        "first": first,
        "second": second,
        "stable": action != "hold",
        "action": action,
    }
    Path("ledger/latest.json").write_text(json.dumps(record, indent=2) + "\n")
    print(json.dumps({"class": first["class"], "action": action}))
    return 0 if action != "hold" else 2

if __name__ == "__main__":
    raise SystemExit(main())

Notice the hash. Same class with a different stderr digest is a warning, not a trophy. I still allow fix_locally in that case, but I want the drift visible.

Would you rather discover that drift after a remote rewrite? I would not.

Step 3. Verify the stable path, then force a flip

Run the boring fixture through the gate.

python classify_failure.py fixtures/case_missing_key.py
python -m json.tool ledger/latest.json

Check three fields. action is fix_locally. Both class values are contract. The process exit code is 0.

If class is unknown, the regex missed your interpreter's traceback format. Fix the regex. Do not "fix" it by hard-coding a pass.

Now a fixture that changes its mind. It uses a flag file under /tmp so the two runs disagree. Delete that flag when you are finished.

# fixtures/case_flip.py
from pathlib import Path

flag = Path("/tmp/failclass-flip")
if flag.exists():
    flag.unlink()
    raise ConnectionRefusedError("simulated runtime miss")
flag.write_text("1", encoding="utf-8")
raise KeyError("remaining")

rm -f /tmp/failclass-flip
python classify_failure.py fixtures/case_flip.py; echo "exit=$?"
rm -f /tmp/failclass-flip

You want exit code 2 and "action": "hold". One run saw contract. The other saw needs_runtime. That disagreement is the result that matters.

If this comes back stable, you did not get two different runs, or the flag path is not writable. Stop and fix the fixture.

Would you paste that flip into a chat and say "make the tests pass"? The model will pick one story. Your ledger says you do not have one story yet.

Step 4. Read the ledger before any network call

python - <<'PY'
import json
from pathlib import Path
rec = json.loads(Path("ledger/latest.json").read_text())
allowed = {"hold", "fix_locally", "eligible_for_isolated_server"}
if rec["action"] not in allowed:
    raise SystemExit("bad action")
print(rec["action"])
PY

The order is the policy.

hold stops the workflow. Capture a smaller fixture, or run a third time and keep the disagreement visible. No server process. No prompt file leaves the laptop.
fix_locally means the patch is yours. Open the fixture. Add the missing key or the missing import. Re-run until the process exits 0 because the bug is gone, not because you relabeled it.
eligible_for_isolated_server is the only branch where a disposable machine is even on the table. Copy that one script. Do not copy your home directory, your .env, or your SSH agent socket.

A model can still be wrong on branch 3. The gate does not make it right. It only stops you from asking a remote box to repair a typo.

Why two runs, not a vibe check

People retry because the button is there. I get it. The red text looks like a task, and a coding model looks like a faster colleague.

But a colleague who has not seen a stable failure will invent a cause. Have you watched a patch "fix" a KeyError by catching Exception? That is what an unstable label invites.

Two runs are cheap. They take seconds. They also create a file you can diff tomorrow. A chat scroll does not.

If the second run is annoying to set up, that annoyance is the signal. Your fixture is still coupled to a clock, a network, or a file you forgot to pin.

I am not telling you to ban models. I am telling you to ban the unordered retry. Fix it locally when the class is contract or local_typo. Use an isolated runtime only when the class stays needs_runtime. Hold when the story changes underneath you.

Step 5. Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator supplied two availability claims for this draft: MonkeyCode has free model access, and it has a free server option. I am not adding a token count, a model id, a region, a hardware shape, or an expiry. Those details go stale, and this page will not invent them.

Read the current product docs before you depend on either claim. If the docs conflict with this paragraph, the docs win.

Use the claims only inside the gate.

Free model access is for a stable needs_runtime note, or for a review of a patch you already applied locally. Send the ledger JSON and the single fixture. Do not send the repository.
The free server option is the isolated runtime for that same fixture, and only after action is eligible_for_isolated_server. If the image cannot run from one copied file and no secrets, it is the wrong server for this workflow.

Keep this prompt as a local file until you choose to send it.

action: eligible_for_isolated_server
constraint: name the smallest runtime assumption. do not edit unrelated files.
fixture:
<one script>
ledger:
<latest.json>

If the client hides the outbound body, do not send. You cannot classify a request you cannot see.

That is the only pitch. If you already have those free options, run this gate once before the next call and count how often the action is hold. I would rather hear that count than a slogan.

Limitations

Two runs can both be lucky. A wrapped exception can hide ConnectionRefusedError inside TypeError, and the table will call it contract. The 15-second timeout is a starting guess, not a measured limit. If the fixture hangs, TimeoutExpired crashes this proposal on purpose so you notice. stderr formats also differ across Python versions.

I have not run this on Windows, so the /tmp flag and the venv activate line are Unix assumptions. Treat that as a gap, not as a portability claim.

Who should not use this approach:

Anyone in an incident that can destroy data or expose credentials. Follow the incident process, not a blog gate.
Anyone who needs a security review. Class names are not a threat model.
Anyone who does not have the right to copy the code onto a third-party server.
Anyone hunting a published quota. This draft refuses to print one.

Skip it too if the failure is non-deterministic by design, such as a load test. Stability across two samples is the wrong oracle there. You need a budget and a metric, not a traceback label.

Done looks like this

Re-run the missing-key fixture and read fix_locally. Re-run the flip fixture and read exit 2. Remove /tmp/failclass-flip if it is still there.

If you cannot explain the action field without opening a chat window, you are not done. Name the break twice. If the label flips, the free retry can wait.

Probe Loopback First. Leave the Ticket Unwritten If /health Lies.

Dakota Liu — Sat, 10 Oct 2026 18:16:23 +0000

I will not publish a generated service until a local contract accepts the health body I actually received. A tidy README is not a running process. If /health lies, the ticket stays unwritten, and the bundle stays on disk.

Generated apps are cheap to start now. They are still easy to ship half-awake. Would you upload a tree because a draft looked finished, or because a loopback probe matched a file you froze first?

This is a from-zero tutorial for that second habit. Every stage has a command and a check you can rerun. Treat the samples as a local recipe, not as output copied from a remote host, and not as a benchmark.

What done means here

Done is four files and one decision record. The files are a contract, a stdlib server, a sealer, and a verifier. The record is publish_ticket.json, and it appears only after the probe matches.

Free model access can help you draft the handler. A free server option can later hold a bundle that already passed. I was given those two availability claims for this draft. They are not measurements, and they are not a promise about next month.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not naming models, quotas, regions, or hardware. I am not saying how long a free slot lasts. If you care about those facts, read the current product docs and ignore any memory of an older limit. When this paragraph and the docs disagree, the docs win.

I am also not trying to prove a remote disk is empty. I am trying to prove that the process I might publish can answer a frozen contract on loopback. Those are different jobs. Mixing them is how a local pass gets narrated as a production fact.

Step 1: Freeze the contract before the handler

Create an empty directory and confirm the interpreter. I do not want a missing python3 to become an excuse to try the same check on a host I do not control.

mkdir -p healthgate && cd healthgate
python3 --version

Verification: the shell prints a Python 3 version line. If it does not, stop and fix the local toolchain. Do not skip ahead to a publish step.

Now write health_contract.json. I keep the expected body tiny on purpose. A health route that needs a paragraph is already doing too much, and a status code alone will not catch a lying JSON object.

{
  "path": "/health",
  "bind_host": "127.0.0.1",
  "port": 8765,
  "timeout_seconds": 2,
  "max_body_bytes": 256,
  "required": {
    "status": "ok",
    "component": "healthgate"
  },
  "forbidden_keys": ["token", "password", "secret", "email"]
}

Why ban those keys? A health document gets copied into logs, screenshots, and status pages. I do not want a generated handler to echo a credential because a prompt said to include debug context. Would you want that string in a screenshot later?

Check the file before you trust it:

python3 -c "import json; c=json.load(open('health_contract.json')); assert c['bind_host']=='127.0.0.1'; assert c['max_body_bytes']<=256; print('contract_ok', c['path'])"

Verification: the command prints contract_ok /health. An assertion error means the contract drifted. Do not start the server yet. A running process against a bad contract is just a faster way to bless the wrong shape.

Step 2: Bind loopback and return only the required keys

The server below is a teaching stub. It is not a framework, and it is not a deploy script. It listens on 127.0.0.1, serves one path, and refuses to log request lines. Logging every hit is how a debug token leaks into a scrollback you later paste somewhere.

# serve_health.py
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json

BODY = {"status": "ok", "component": "healthgate"}

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/health":
            self.send_response(404)
            self.end_headers()
            return
        raw = json.dumps(BODY).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def log_message(self, fmt, *args):
        return

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

Start it in one terminal with python3 serve_health.py. In a second terminal, read the body yourself:

python3 -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8765/health', timeout=2).read().decode())"

Verification: you should see a JSON object with status and component only. Connection refused means the process is down. Extra keys mean the stub drifted from the contract. Fix either problem before a ticket exists.

A model draft can type this file faster than I can. I would still run the probe myself. A suggested handler is text. A probe is a socket. Which one are you actually publishing?

Step 3: Seal an allowlist, not the working tree

I copy named files into bundle/. I do not archive the directory I am editing. Scratch notes, .env files, and editor backups do not get a free ride because they sat beside the server. Have you ever zipped a repo just to try a host, then remembered a scratch file was in there?

# seal_bundle.py
import shutil
from pathlib import Path

ALLOW = ["health_contract.json", "serve_health.py", "verify_health.py"]
root = Path("bundle")
if root.exists():
    shutil.rmtree(root)
root.mkdir()
for name in ALLOW:
    src = Path(name)
    if not src.is_file():
        raise SystemExit(f"missing:{name}")
    shutil.copy2(src, root / name)
print("sealed", len(ALLOW))

Run this before verify_health.py exists. It should fail. That failure is the check. A partial bundle is not a bundle, and a sealer that skips missing files will quietly publish an older tree.

python3 seal_bundle.py

Verification now: the exit code is non-zero and the message starts with missing:. After step 4, rerun the sealer and list the tree.

python3 seal_bundle.py
find bundle -type f | sort

Verification later: exactly three files, no dotfiles, and no ticket inside bundle/. The ticket is a local decision. Shipping it as application content would confuse a host into thinking a pass is a route. I want the host, if I use one at all, to receive the program, not my grade sheet.

Step 4: Parse the body and fail closed

The verifier does not start your server. You already did that. It reads the contract, refuses any bind host other than loopback before it opens a socket, then GETs the path. Forbidden keys fail first. Value mismatches fail next. Extra keys fail last. A huge body cannot sneak through as a pass, because the read is capped and an oversize buffer is a failure.

# verify_health.py
import hashlib, json, sys, urllib.request
from pathlib import Path

c = json.loads(Path("health_contract.json").read_text())
if c["bind_host"] != "127.0.0.1":
    sys.exit("refusing_non_loopback")
url = f"http://{c['bind_host']}:{c['port']}{c['path']}"
try:
    with urllib.request.urlopen(url, timeout=c["timeout_seconds"]) as resp:
        raw = resp.read(c["max_body_bytes"] + 1)
        code = resp.status
except Exception as exc:
    sys.exit(f"probe_failed:{type(exc).__name__}")
if code != 200 or len(raw) > c["max_body_bytes"]:
    sys.exit("bad_status_or_size")
try:
    body = json.loads(raw.decode("utf-8"))
except json.JSONDecodeError:
    sys.exit("not_json")
if not isinstance(body, dict):
    sys.exit("not_object")
for key in c["forbidden_keys"]:
    if key in body:
        sys.exit(f"forbidden:{key}")
for key, value in c["required"].items():
    if body.get(key) != value:
        sys.exit(f"mismatch:{key}")
if set(body) != set(c["required"]):
    sys.exit("unexpected_keys")
ticket = {
    "decision": "local_pass",
    "url": url,
    "sha256": hashlib.sha256(raw).hexdigest(),
    "body_bytes": len(raw),
}
Path("publish_ticket.json").write_text(json.dumps(ticket, indent=2) + "\n")
print("ticket_written", ticket["sha256"][:12])

urlopen raises on many HTTP errors, so a wrong path often shows up as probe_failed rather than a polite 404. That is fine for this gate. I would rather see a hard miss than a parser that tries to interpret an error page as health.

With the server still running, write the ticket and read it back:

python3 verify_health.py
python3 -c "import json; t=json.load(open('publish_ticket.json')); assert t['decision']=='local_pass'; print('ticket_ok', t['body_bytes'])"

Verification: one line starting with ticket_written, then ticket_ok and a byte count no higher than 256. The hash is an integrity note for the bytes you just probed. It is not a signature, and it is not authentication. Do not describe it as either.

Stop the server and probe again. I expect probe_failed and a non-zero exit. Delete a stale ticket when the probe fails. Otherwise yesterday's pass pretends to bless today's crash.

python3 verify_health.py || rm -f publish_ticket.json
test ! -f publish_ticket.json && echo "ticket_absent_after_failure"

Verification: ticket_absent_after_failure prints, and publish_ticket.json is gone. If the file remains, you did not fail closed. Do not talk yourself into uploading anyway.

Step 5: Only then ask whether a host is relevant

I use a short table so the publish decision stays boring. Boring is what I want. Excitement belongs in the product, not in the gate.

Check	Pass	Fail
Ticket	`decision` is `local_pass`	Do not upload
Bundle list	allowlist only	Reseal and reprobe
Body hash	matches the ticket	Delete the ticket
Bind host	still `127.0.0.1` in the contract	Refuse the probe

A drafting model fits earlier, while serve_health.py is still text on disk. It is a weak fit for the question "should I publish?" once you have skipped the probe. Why spend a free call to narrate a check your terminal can already fail?

A free server option is a possible later place for bundle/, and only if today's terms still cover this kind of toy process. I have not measured that option. I am not claiming it is empty, fast, permanent, or the right host for anything with real users. If the terms changed, the local ticket is still useful and the upload is not.

If the ticket is green and you want a next read, open the current MonkeyCode notes on free model access and the free server option, then decide whether this sealed stub belongs there at all. Skip the product if the docs do not match the job. The gate does not need a logo to fail a bad body.

Limits, and who should walk away

Loopback success does not prove TLS, DNS, idle cutoff, or disk survival on any host. The forbidden-key list is not a secret scanner. It will miss a key named api_key unless you add it. Add it if your drafts keep inventing that name. Do not put customer data in /health and then congratulate the contract.

Skip this approach when the process cannot bind 127.0.0.1 for a check, or when health output must vary per request. Skip it for multi-service setups, meshes, and anything that needs a formal change approval. A three-file Python gate will feel strict and still be the wrong control. Use the pipeline you already trust.

I am also not giving you a latency number. A number from a laptop would be folklore by the time you read it. The artifact is the exit code, not a chart. If someone asks how fast the free host was, the honest answer from this page is: I did not measure it.

Rehearse the lie

Change BODY so it includes "token": "demo". Restart the server. Run the verifier. You want forbidden:token, a non-zero exit, and no fresh ticket. Put the body back afterward. Then add "build": "local" and confirm that unexpected keys fail too.

python3 verify_health.py; echo "exit:$?"

If that exit were zero while a token sat in the JSON, would you still publish? I would not. The host is optional. The match is the point. Rehearse the failure once before you trust the success path, because a gate you have never seen fail is just a script you hope works.

Close the loop

Freeze the contract. Bind loopback. Seal named files. Write the ticket only when the body matches, and leave it unwritten when the probe fails. A draft from anywhere is still text until the probe runs. A host, free or paid, does not get to overrule a health body that does not parse.

Freeze the Intent File. Fail the Plan If It Grows.

Dakota Liu — Fri, 09 Oct 2026 14:32:44 +0000

A free model will happily plan a bigger change than the one you asked for. I do not spend that call until a local file says what "enough" means.

You know the feeling? You typed "rename the helper," and the plan came back with a new HTTP client, a lockfile edit, and a path you never named. Who approved that growth? Not me. Not the ticket. The model did, because nothing on disk was allowed to say no.

This is a from-zero tutorial for that no. Intent file, plan fixture, checker, then a verification command after every stage. The samples are fixtures you can read in a terminal. I am not describing a production outage, a customer, or a measured quota.

What I am actually refusing

I want one rule you can reuse tomorrow. The human writes the bound. The model may propose work inside the bound. If the plan adds a path, a host, a dependency, or a command, the job fails closed before any free server sees it.

Is that a sandbox? No. Is it code review? Also no. It is a small refusal you can run before the interesting part starts.

A DEV thread this week poked at a sharper fear: a model that notices a task is aimed at someone else's system and stays quiet. I am not re-running that experiment, and I am not treating a title as a measurement. I am asking a narrower question. What does your runner do when the plan grows past the sentence you wrote?

Step 1. Write the intent before you open a chat

Start empty. I do not want a chat transcript to be the spec.

mkdir -p intent-gate/fixtures
cd intent-gate

Put this in intent.json.

{
  "intent_id": "rename-helper-014",
  "summary": "Rename format_name to format_label in pkg/names.py only.",
  "allowed_paths": ["pkg/names.py"],
  "allowed_hosts": [],
  "allowed_deps": [],
  "allowed_commands": ["python -m pytest tests/test_names.py"],
  "forbidden": ["network", "lockfile", "shell_interpolation"]
}

Verify the file is actually JSON.

python -c "import json; json.load(open('intent.json')); print('intent_ok')"

You want intent_ok. Broken JSON? Stop. Why would I let a model parse the bound I failed to parse myself?

Step 2. Pin the checker before you pin a model

Save widen_check.py. It loads both files. It exits 0 only when every path, host, dependency, and command in the plan is already listed, and the intent id matches.

#!/usr/bin/env python3
"""Local plan gate. Unexecuted against a live server in this draft."""
import json
import sys

def load(path):
    with open(path, encoding="utf-8") as handle:
        return json.load(handle)

def main():
    intent = load(sys.argv[1])
    plan = load(sys.argv[2])
    reasons = []
    pairs = (
        ("paths", "allowed_paths"),
        ("hosts", "allowed_hosts"),
        ("deps", "allowed_deps"),
        ("commands", "allowed_commands"),
    )
    for key, field in pairs:
        allowed = set(intent.get(field, []))
        for item in plan.get(key, []):
            if item not in allowed:
                reasons.append(f"widen:{key}:{item}")
    if plan.get("intent_id") != intent.get("intent_id"):
        reasons.append("intent_id_mismatch")
    if reasons:
        print("reject")
        for reason in reasons:
            print(reason)
        return 2
    print("accept")
    print(intent["intent_id"])
    return 0

if __name__ == "__main__":
    sys.exit(main())

Compile it. Still no network.

python -m py_compile widen_check.py && echo compile_ok

compile_ok means the gate parses. It does not mean the logic is right. That is the next two steps. Skip them and you only proved Python can read a file.

Step 3. Make the wide plan fail in public

Write fixtures/plan_wide.json.

{
  "intent_id": "rename-helper-014",
  "paths": ["pkg/names.py", "pkg/net.py"],
  "hosts": ["example.invalid"],
  "deps": ["requests"],
  "commands": [
    "python -m pytest tests/test_names.py",
    "curl https://example.invalid"
  ]
}

Run it.

python widen_check.py intent.json fixtures/plan_wide.json; echo exit:$?

I expect reject, then widen:paths:pkg/net.py, widen:hosts:example.invalid, widen:deps:requests, and a widened command line, then exit:2.

Did one of those lines go missing? Then the checker is softer than the fixture, and the fixture is the spec. Fix the script. Do not "try it on the model and see." Seeing is how extra hosts survive.

Step 4. Make the narrow plan pass, and keep it narrow

Write fixtures/plan_fit.json.

{
  "intent_id": "rename-helper-014",
  "paths": ["pkg/names.py"],
  "hosts": [],
  "deps": [],
  "commands": ["python -m pytest tests/test_names.py"]
}

python widen_check.py intent.json fixtures/plan_fit.json; echo exit:$?

Expected text is accept, then rename-helper-014, then exit:0.

Tempted to delete the wide fixture once this looks green? Don't. A gate that only meets friendly plans is a diary entry. I keep both files, and I rerun both commands whenever the checker changes.

Step 5. Only now do a free model or a free server enter

Disclosure: This article was prepared as part of MonkeyCode's product outreach. This is the first point where that product sits in the method. The claim I will repeat is small: free model access, and a free server option. I am not naming a model. I am not stating a token count, a machine size, or how long either offer lasts. Those numbers go stale. Read the project docs on the day you run the gate, and trust the docs over any post, including this one.

What do I send? The intent id, the summary, and the allowlists. Not the tree. Not environment files. Not a shell line built by gluing tool arguments together.

The reply has to become fixtures/plan_from_model.json before it can matter. Prose is not a plan. If the model chats, I fail closed. I do not scrape a command out of a paragraph.

python widen_check.py intent.json fixtures/plan_from_model.json; echo exit:$?

Exit 2 means I do not "repair it on the server." I either edit intent.json because I truly want the wider job, or I drop the plan. Exit 0 means the free server may run the one command already listed in allowed_commands.

Would I let that server invent a second command? No. The command was frozen in step 1. A free minute is not a new spec.

Read the result like a table, not a vibe

Plan signal	Local result	What I do next
Extra path, host, dependency, or command	exit 2	No server minute
Intent id does not match	exit 2	Discard; do not retry the same file
Exact allowlist match	exit 0	Optional run of the listed command
Prose instead of JSON	no score	Fail closed; do not parse English for a shell line

Where this breaks, and who should walk away

String equality is blunt. pkg/names.py and ./pkg/names.py are different strings here, and a relative path can still sneak past a tired reviewer. Normalize before you share this gate. I left normalization out so the fixture stays readable. That is a limitation, not a feature I want you to miss.

Inside an allowed file, the checker is blind. A wrong rename still looks like accept. An already-allowed dependency can still be compromised. Nothing here proves a free server is empty, isolated, or logged. If the job touches production secrets, payments, or another person's systems, do not use a tutorial script as the control. You want a real sandbox, a reviewer, and an owner who can refuse.

Skip it if you wanted an unbounded generation pass. The method starts with a file you were willing to type by hand. No file, no free call. Why spend a hosted minute to discover you never decided the bound?

After the exit codes match

I would commit the checker and both fixtures, and I would make the wide plan keep failing in CI. The fit plan must keep passing. That pair is the artifact. The model is a guest.

If you want to try the same order, start from the reject fixture, not from a signup page. When both exit codes match this tutorial, then check MonkeyCode's current free model access and free server option and decide whether they fit the job you already wrote down.

Plant the Canary Before You Spend a Free Model Call

Dakota Liu — Thu, 08 Oct 2026 10:19:48 +0000

If the prompt still holds a credential, do not call the model. Scan the fixture on your laptop, prove the scanner can fail, and only then spend a free model call or a free server run.

I keep seeing the same shortcut in agent writeups. Someone wants a quick demo, so they paste a working tree. The tree still has a .env.

Sometimes it has a cloud key. Sometimes it has a customer id that never belonged in a prompt. Why hand that to a model you are only trying?

People are arguing, loudly, about what a model does when the task looks sensitive. I am not recycling a score from a thread I cannot re-run. I want a gate I can fail on my own laptop before any free call.

This is a from-zero tutorial for that preflight. It is not a benchmark. I am not claiming a measured latency, a saved dollar figure, or a quota. The scripts are unexecuted examples. Run them yourself.

What I will and will not claim

Two availability claims drive the later steps: free model access, and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That sentence is the relationship. I am not auditing the service in this draft. I will not name a model, invent a token allotment, describe hardware, or promise how long a free server stays up.

Those facts go stale. On the day you run this, read the page you can reload. If it disagrees with an old post, believe the page.

Pull the product out and the gates still stand. They are local on purpose.

What you need before step 1

You need Python 3, a shell, and a directory that is not your production checkout. You also need a rule you can say out loud. The model may propose. You apply.

Can you keep those roles apart when you are in a hurry? If the answer is no, skip any remote run. A toy fixture under your home directory is enough. Your real repo is not the demo.

Step 1 — Create a fixture the model is allowed to see

Start empty. Add one function and one test. Add nothing else.

mkdir -p $HOME/prompt-gate-demo/src
cd $HOME/prompt-gate-demo
printf 'def add(a, b):\n    return a + b\n' > src/add.py
printf 'from src.add import add\n\ndef test_add():\n    assert add(2, 3) == 5\n' > test_add.py
python3 -m py_compile src/add.py test_add.py && echo STEP1_PASS
find . -name .env -o -name '*.pem' -o -name '*.key'

Verification is two checks, not one. STEP1_PASS must print. The find must print nothing. If either check fails, delete the directory and start again. Cleaning it up later is how a key survives into the prompt.

Step 2 — Plant a canary and prove the scanner fails closed

A scanner that has never seen a secret is theater. Why trust a green result you have not tried to break?

Save the following as scan_prompt.py next to the demo, not inside it. It is a teaching scanner. It is not a replacement for a dedicated secret-scanning tool your team already trusts.

#!/usr/bin/env python3
# Unexecuted example. Fail closed if a denylisted pattern appears.
import pathlib
import re
import sys

PATTERNS = [
    ('aws_access_key', re.compile(r'AKIA[0-9A-Z]{16}')),
    ('private_key', re.compile(r'-----BEGIN (?:RSA |OPENSSH |EC )?PRIVATE KEY-----')),
    ('assignment', re.compile(r'(?i)\b(api_key|secret|token|password)\b\s*[:=]\s*\S+')),
    ('canary', re.compile(r'CANARY_SECRET_DO_NOT_SEND')),
]
SKIP = {'.git', '.venv', 'node_modules', '__pycache__'}
ALLOW = {'.py', '.md', '.txt', '.toml', '.json'}

def files(root: pathlib.Path):
    for path in root.rglob('*'):
        if not path.is_file():
            continue
        if any(part in SKIP for part in path.parts):
            continue
        if path.suffix.lower() not in ALLOW:
            continue
        yield path

def main() -> int:
    root = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else '.')
    hits = 0
    for path in files(root):
        text = path.read_text(encoding='utf-8', errors='replace')
        for name, cre in PATTERNS:
            if cre.search(text):
                hits += 1
                print(f'HIT {name} {path}')
    print('PASS' if hits == 0 else 'FAIL')
    return 0 if hits == 0 else 2

if __name__ == '__main__':
    raise SystemExit(main())

Run the boring side first, then the mean side.

python3 scan_prompt.py $HOME/prompt-gate-demo
# expect: PASS and exit 0

printf '\nAPI_KEY=CANARY_SECRET_DO_NOT_SEND\n' >> $HOME/prompt-gate-demo/src/add.py
python3 scan_prompt.py $HOME/prompt-gate-demo
echo exit=$?
# expect: HIT canary, then FAIL and exit=2

If the canary run exits 0, stop. The rest of this tutorial is invalid until the scanner fails closed. Delete the canary line, scan again, and demand PASS before you continue. I would rather lose ten minutes here than explain a leaked key later.

Step 3 — Copy an allowlisted bundle

A passing scan is not permission to zip the tree. Hidden files stay behind. So does anything you did not name.

rm -rf $HOME/prompt-gate-bundle
mkdir -p $HOME/prompt-gate-bundle
cp $HOME/prompt-gate-demo/src/add.py $HOME/prompt-gate-bundle/add.py
cp $HOME/prompt-gate-demo/test_add.py $HOME/prompt-gate-bundle/test_add.py
find $HOME/prompt-gate-bundle -type f -print
python3 scan_prompt.py $HOME/prompt-gate-bundle

Verification: the find prints exactly two paths, then the scanner prints PASS. Three files means you copied a surprise. Stop and delete the bundle. Do not debug the surprise by uploading it.

Step 4 — Spend the free model call on that bundle only

This is the first moment free model access belongs in the workflow. I paste the two files and one ask: make add reject non-integers, and keep the existing test green. I do not paste the parent directory. I do not paste shell history. I do not paste a dashboard screenshot for context.

What do I check in the reply before I touch a server?

It names files I already have.
It does not ask me to export a secret or read a credential file.
It does not add a network call the test never required.
I can save the text without executing it.

I write that reply to proposal.md outside the bundle. The proposal is untrusted input. It is not a shell script, and it is not a merge.

If the surface in front of you no longer offers a free call, stop. Do not upload the real repository somewhere else just to finish the demo.

Step 5 — Apply only after a test you wrote

A free server is optional, and it comes last. It runs a command you already typed, against a directory you already scanned. It does not get to invent the command.

cp -R $HOME/prompt-gate-bundle $HOME/prompt-gate-apply
# Apply the proposal by editing add.py yourself.
python3 -m pytest $HOME/prompt-gate-apply/test_add.py
echo pytest_exit=$?
python3 scan_prompt.py $HOME/prompt-gate-apply

Verification: pytest_exit=0, and the second scan prints PASS. If pytest is missing, install it in a local virtualenv or rewrite the check as a plain assert runner. A missing tool is not a reason to skip the check.

Only after that local pass do I consider a free server run. Upload the apply directory, not your home directory. Run the same test command because you typed it. If the current free-server page lists extra limits, those limits win over this article.

The stop/go table

I keep this next to the terminal. A demo that is almost done does not get a special row.

Scan	Canary proven this session	Bundle	Model call	Server run
FAIL	yes or no	any	no	no
PASS	no	any	no	no
PASS	yes	extra files	no	no
PASS	yes	exact allowlist	yes, bundle only	only after local tests pass

Pick the row that matches the terminal, not the row you wish you were on.

What this does not do

Regexes miss secrets in images, archives, and encoded blobs. A canary in a .py file does not prove you would catch a key inside notebook output. Encoding a secret so the pattern misses is not a fix. It is how people fool their own gate.

Do not use this approach as a workplace data-loss program. If your team already mandates a scanner, use that one. Do not use it when the task truly needs a live environment file.

Redesign the task until a fake value is enough. Do not treat a free server as a long-lived host, a customer-data store, or a way around terms you have not read.

I also will not pretend a free option is permanent. Availability you saw today can disappear. That is why the gates live on your machine, in files you can re-run without an account.

A five-minute re-run the next morning

Tomorrow the demo will feel familiar, and that is when people skip the canary. I do not.

I delete prompt-gate-apply, plant the canary again, and require exit=2 before I remove it. Then I rebuild the bundle from the two cp lines, not from memory. Then I ask the model the same question and diff the new proposal against proposal.md.

Did the reply grow a request for environment variables? Throw it away. Did the file list change? Throw it away. Did your local pytest fail? The server stays off.

That re-run is the whole method. The product is only the last hop, and only when the page in front of you still offers the free path.

Where I would spend the trial

If the fixture scans clean and the test already passes locally, the remaining choice is where to spend the trial. I would open the current MonkeyCode docs, confirm that free model access and the free server option are still listed, and only then send the two-file bundle. If the page does not list them, I keep the local gates and I skip the upload.