DEV Community: Sam Rivera

Build a Silent-Default Kill List for Agent Diffs

Sam Rivera — Sun, 06 Sep 2026 20:30:29 +0000

The agent finished a patch while I made coffee.
Green tests sat next to a config I never requested.
What did it assume while I stepped away?

My goal tonight is merging nothing with silent defaults.
The budget is ninety minutes, then I abandon the branch.
I wanted a packet I could copy, not a lecture.

Cheap diffs still cost review hours

Agents write fast, but review hours do not scale.
That gap is exactly where silent defaults live.
A missing flag becomes a production surprise later.

Have you seen a helpful force flag you never typed?
Green tests still pass when that flag appears.
Green tests do not prove the agent asked permission.

This piece is not another architecture survival guide.
I do not need twenty agentic buzzwords tonight.
I need a fail-closed list and one command.

What the packet contains

Three files make the whole copyable artifact tonight.
You keep an allowlist, a scanner, and evidence.
If extras appear, then the merge dies immediately.

I already shipped a broader readiness checklist once.
Tonight the scope stays intentionally narrower than that.
We only hunt values that nobody actually declared.

Time box this to ninety minutes from clone to decision.
Keep paid APIs at zero if a free path exists.
Any undeclared default kills the branch without debate.

Gate 1 — freeze the question first

Write the task in one sentence before anyone runs.
If you cannot, you are simply not ready.
The agent will fill that silence for you.

# task.txt
Add a --timeout-s flag. Default 15. Nothing else.

Ask yourself a harder question about required absences.
What must stay missing from this patch entirely?
Write those absences down as real requirements.

Absence is a requirement here, not a vague vibe.

# forbidden.txt
force
retry
insecure
auto_approve

Gate 2 — declare allowed defaults

Put every legal default in one boring JSON file.
Skip comments and skip hidden environment fallbacks tonight.
If a value is not here, it is illegal.

{
  "timeout_s": 15,
  "output": "json",
  "max_tokens": 512
}

Why JSON instead of a markdown checklist alone?
I can diff it and I can grep it.
I cannot grep a vibe during a tired review.

Gate 3 — pin tools and paths

Agents assume extra tools when the prompt stays soft.
List the allowed tools and list the allowed files.
Everything else counts as a failed gate immediately.

# allowed-tools.txt
read_file
replace_in_file
run_tests

# allowed-paths.txt
src/cli.ts
src/config.ts
tests/cli.test.ts

Did the agent touch a deploy file you never named?
That is a silent default of scope, not help.
Kill the branch and do not negotiate with the diff.

Gate 4 — run the scanner

Here is the copyable utility I keep in tools.
It is intentionally boring Python, and that is the point.
Boring is a feature when agents start to improvise.

This scanner is a template.
Run the fixture before you trust it.
Save the file as kill_defaults.py in the repo.

#!/usr/bin/env python3
"""Fail closed when an agent diff introduces silent defaults."""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path


def load_json(path: Path) -> dict:
    return json.loads(path.read_text(encoding="utf-8"))


def load_lines(path: Path) -> list[str]:
    text = path.read_text(encoding="utf-8")
    return [ln.strip() for ln in text.splitlines() if ln.strip()]


def added_lines(diff_text: str) -> list[str]:
    out: list[str] = []
    for line in diff_text.splitlines():
        if line.startswith("+++") or line.startswith("---"):
            continue
        if line.startswith("+"):
            out.append(line[1:])
    return out


def added_paths(diff_text: str) -> list[str]:
    paths: list[str] = []
    for line in diff_text.splitlines():
        if line.startswith("+++ b/"):
            paths.append(line[6:])
    return paths


def observed_from_lines(lines: list[str]) -> dict:
    observed: dict[str, str] = {}
    skip = {"const", "let", "var", "return", "export"}
    for line in lines:
        if ":" not in line:
            continue
        left, right = line.split(":", 1)
        key = left.strip().strip("\"'{} ")
        if not key or " " in key or key in skip:
            continue
        observed[key] = right.strip().strip(",'\"")
    return observed


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--allowlist", required=True)
    parser.add_argument("--forbidden", required=True)
    parser.add_argument("--allowed-paths", required=True)
    parser.add_argument("--diff", required=True)
    parser.add_argument("--out", required=True)
    args = parser.parse_args()

    out = Path(args.out)
    out.mkdir(parents=True, exist_ok=True)

    allow = load_json(Path(args.allowlist))
    forbidden = load_lines(Path(args.forbidden))
    allowed_paths = load_lines(Path(args.allowed_paths))
    diff_text = Path(args.diff).read_text(encoding="utf-8")

    added = added_lines(diff_text)
    paths = added_paths(diff_text)
    observed = observed_from_lines(added)

    extras = {k: v for k, v in observed.items() if k not in allow}
    forbidden_hits = sorted(
        {
            *[k for k in observed if k in forbidden],
            *[w for w in forbidden if any(w in line for line in added)],
        }
    )
    path_hits = [p for p in paths if p not in allowed_paths]

    (out / "declared.json").write_text(json.dumps(allow, indent=2) + "\n")
    (out / "observed.json").write_text(json.dumps(observed, indent=2) + "\n")
    (out / "extras.json").write_text(json.dumps(extras, indent=2) + "\n")
    (out / "forbidden_hits.txt").write_text(
        "\n".join(forbidden_hits) + ("\n" if forbidden_hits else "")
    )
    (out / "path_hits.txt").write_text(
        "\n".join(path_hits) + ("\n" if path_hits else "")
    )

    failed = bool(extras or forbidden_hits or path_hits)
    verdict = "FAIL_CLOSED" if failed else "PASS"
    (out / "verdict.txt").write_text(verdict + "\n")
    print(verdict)
    if extras:
        print("extras:", json.dumps(extras))
    if forbidden_hits:
        print("forbidden:", ", ".join(forbidden_hits))
    if path_hits:
        print("paths:", ", ".join(path_hits))
    return 2 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

Run it against the agent patch like this:

git diff main > /tmp/agent.patch
python3 kill_defaults.py \
  --allowlist allowed-defaults.json \
  --forbidden forbidden.txt \
  --allowed-paths allowed-paths.txt \
  --diff /tmp/agent.patch \
  --out review-packet
echo $?

Exit code zero means the packet stayed clean.
Any other code means I stop the merge.
I do not allow a just-this-once exception here.

Gate 5 — demand evidence files

The scanner must write a folder, not a feeling.

review-packet/
  declared.json
  observed.json
  extras.json
  forbidden_hits.txt
  path_hits.txt
  verdict.txt

If extras.json is not an empty object, fail closed.
If verdict.txt is missing, treat that as failure.
Missing evidence is a failure, never a skip.

Copyable decision table

Gate	Evidence file	Fail closed when
Freeze the task	`task.txt`	The sentence is missing or fuzzy
Allowlist	`allowed-defaults.json`	Observed keys are not declared
Forbidden names	`forbidden_hits.txt`	Any hit exists
Path pin	`path_hits.txt`	Any path sits outside the list
Verdict	`verdict.txt`	File missing or not `PASS`

A failure fixture you can replay

I keep a dirty patch just for the scanner.
Do not trust a gate you have never failed.
This fixture must break the command on purpose.

fixtures/silent-retry.patch looks like this:

--- a/src/config.ts
+++ b/src/config.ts
@@
-export const defaults = { timeout_s: 15 };
+export const defaults = { timeout_s: 15, retry: 99 };

Who asked for retry set to ninety-nine?
Not me, and not the allowlist file either.

python3 kill_defaults.py \
  --allowlist allowed-defaults.json \
  --forbidden forbidden.txt \
  --allowed-paths allowed-paths.txt \
  --diff fixtures/silent-retry.patch \
  --out /tmp/packet-fail
echo $?   # expect 2
cat /tmp/packet-fail/extras.json

You should see retry land in extras.json.
If the scanner stays quiet, throw the scanner away.
A mute gate is only another silent default.

Gate 6 — time box and rollback

Ninety minutes, then I pick exactly one exit.

Merge only when extras.json stays an empty object.
Strip the extras and then rerun the scanner.
Abandon the branch and keep the packet.

Rollback is not a speech, it is a command.

git restore .
git switch -c abandon/silent-defaults-$(date +%Y%m%d)
mkdir -p abandoned
cp -R review-packet "abandoned/$(date +%Y%m%dT%H%M)"

Why keep the packet after you abandon the branch?
So next weekend I do not repeat the miss.
The extras file is the actual written lesson.

Where a free model path fits

I needed a throwaway box for the agent run.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode is an open-source project with free model access.
It also publishes a free server option for experiments.
Treat that box as disposable compute, never as production.

The packet does not depend on that vendor at all.
Swap the model tomorrow and keep the same gates.
If extras.json still fills, you still stop cold.

Need a free-model path for a weekend packet?
Their public project is one place to try.

Who should not use this

Do not use this packet as a security audit.
It will not catch prompt injection in tool output.
It will not catch leaked secrets in extra files.

Do not use this on huge generated refactors either.
A three-thousand-line agent PR defeats the time box.
Split the task or reject the run outright.

Skip this if you already have formal change control.
Skip this if nobody on the team reads diffs.
A packet cannot replace a careful human eyeball.

Limitations I will not hide

The scanner is a blunt regex over a patch.
It misses computed defaults hiding inside helper functions.
It also misses values loaded from the environment.

Semantic assumptions still slip through this cheap net.
A declared timeout can still be the wrong number.
The gate only proves the value was declared first.

I still read the diff after the scanner passes.
The packet only makes silence expensive to ship.
That cost is the entire point of the ritual.

What I copy tomorrow

Copy the three files into the repo tonight.
Commit them before the next agent session starts.
Then run the failure fixture exactly one time.

If the fixture does not fail, you stop.
Fix the scanner and do not start the agent.
A green fixture is a lie you should not keep.

Which extra field does your agent keep injecting lately?
Should that field be allowlisted, or killed on sight?

Pin CLI JSON Keys Before an Agent Renames Them

Sam Rivera — Sat, 05 Sep 2026 19:09:23 +0000

Tuesday's digest folder filled with empty files. The summarizer CLI had exited 0 every night. Cron still ran jq -r .excerpt. The field no longer existed.

A Sunday agent pass had renamed excerpt to blurb for readability. Unit tests still passed. They checked that stdout parsed as a dict and that the process code was zero. Nobody had pinned the keys a wrapper already consumed.

This write-up is a lab fixture, not a production postmortem. The clocks below are time boxes, not fleet metrics. The work is one frozen JSON contract for a tiny --format json CLI, plus a local checker that exits 1 when an agent shuffles keys.

Cron remembers the old name

Wrappers are stubborn in a way dashboards are not. A crontab line written six months ago still asks for .excerpt. A GitHub Action still pipes the same jq filter. The agent never sees those callers. It sees a Python file and a README that can be rewritten in one diff.

Cheap generation makes that rewrite feel free. The debt shows up later, in a job that writes zero-byte artifacts and still reports success. Exit code 0 is not a document of shape.

Think of a labeled kitchen drawer. The CLI used to keep the scissors under excerpt. Someone moved the scissors and wrote blurb on the front. The night job still opens excerpt, then files the emptiness as a completed task.

Keys are the public surface

For a small JSON CLI, the public surface is the key set, the types, and the extra-key policy. Help text is commentary. Docstrings are commentary. A test that only calls json.loads is commentary with a green checkmark.

The contract here is a JSON file checked into the repo. It lists required keys, their types, and whether unknown keys are allowed. The checker loads one real CLI payload and compares. It does not call a network model. It does not crawl GitHub Actions. It only answers whether tonight's object still matches the keys a wrapper already parses.

Forty-five minutes and zero dollars is enough for this cut. If the script grows past one file plus one schema, stop. Rollback is deleting schemas/cli_json_v1.json and scripts/check_json_contract.py.

A CLI that can drift on purpose

Save this as app.py. The default path emits excerpt. A second branch, flipped by an environment variable, emits blurb so the break can be reproduced without waiting for a real agent patch.

#!/usr/bin/env python3
"""Minimal summarizer CLI with a JSON mode that can drift."""

from __future__ import annotations

import json
import os
import sys
from pathlib import Path


def summarize(source: Path) -> dict:
    text = source.read_text(encoding="utf-8")
    payload = {
        "source": str(source),
        "bytes": source.stat().st_size,
        "excerpt": text.strip()[:120],
    }
    if os.environ.get("CLI_JSON_DRIFT") == "1":
        payload["blurb"] = payload.pop("excerpt")
    return payload


def main() -> int:
    if "--format" not in sys.argv or "json" not in sys.argv:
        print("usage: app.py --format json --source <file>", file=sys.stderr)
        return 2
    try:
        src_flag = sys.argv.index("--source")
        source = Path(sys.argv[src_flag + 1])
    except (ValueError, IndexError):
        print("missing --source", file=sys.stderr)
        return 2
    if not source.is_file():
        print("source not found", file=sys.stderr)
        return 2
    json.dump(summarize(source), sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    sys.exit(main())

A fixture file keeps the demo honest. Save fixtures/app.log with two short log lines. The exact wording does not matter. The keys do.

One versioned contract file

Save this as schemas/cli_json_v1.json. Version the filename. Do not overwrite v1 in place when a key is retired. That is how wrappers get a grace period instead of a surprise.

{
  "name": "summarizer-json-v1",
  "additionalProperties": false,
  "required": ["source", "bytes", "excerpt"],
  "properties": {
    "source": {"type": "string", "minLength": 1},
    "bytes": {"type": "integer", "minimum": 0},
    "excerpt": {"type": "string"}
  }
}

additionalProperties set to false is the sharp edge. Agents love adding confidence, model, or notes. Those extra fields look harmless until another tool serializes the whole object into a store that rejects unknown columns. Surprise keys count as a break.

A checker with no extra packages

Save this as scripts/check_json_contract.py. Standard library only. No JSON Schema package, no network, no git inspection.

#!/usr/bin/env python3
"""Exit 1 when CLI JSON drifts from a frozen key contract."""

from __future__ import annotations

import json
import sys
from pathlib import Path
from typing import Any


def load_json(path: Path) -> Any:
    return json.loads(path.read_text(encoding="utf-8"))


def type_ok(value: Any, expected: str) -> bool:
    mapping = {
        "string": str,
        "integer": int,
        "number": (int, float),
        "boolean": bool,
        "object": dict,
        "array": list,
    }
    if expected == "integer" and isinstance(value, bool):
        return False
    wanted = mapping[expected]
    return isinstance(value, wanted)


def check(schema: dict, payload: Any) -> list[str]:
    errors: list[str] = []
    if not isinstance(payload, dict):
        return ["payload is not an object"]
    required = schema.get("required", [])
    properties = schema.get("properties", {})
    extras_ok = schema.get("additionalProperties", True)
    for key in required:
        if key not in payload:
            errors.append(f"missing required key: {key}")
    for key, value in payload.items():
        spec = properties.get(key)
        if spec is None:
            if not extras_ok:
                errors.append(f"undeclared key: {key}")
            continue
        expected = spec.get("type")
        if expected and not type_ok(value, expected):
            errors.append(
                f"{key}: expected {expected}, got {type(value).__name__}"
            )
        if expected == "string" and spec.get("minLength") and len(value) < spec["minLength"]:
            errors.append(f"{key}: empty string")
        if expected == "integer" and "minimum" in spec and value < spec["minimum"]:
            errors.append(f"{key}: {value} below minimum {spec['minimum']}")
    return errors


def main() -> int:
    if len(sys.argv) != 3:
        print(
            "usage: check_json_contract.py <schema.json> <payload.json>",
            file=sys.stderr,
        )
        return 2
    schema = load_json(Path(sys.argv[1]))
    payload = load_json(Path(sys.argv[2]))
    errors = check(schema, payload)
    if errors:
        print("JSON contract failed:")
        for item in errors:
            print(f"  - {item}")
        return 1
    print(f"JSON contract ok: {schema.get('name', 'unnamed')}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The checker never executes the agent. It never grades the quality of the summary. It only compares shapes. Shape is what cron actually consumes.

Reproduce the rename

Create a short log, emit JSON, and validate the happy object.

mkdir -p fixtures schemas scripts
printf '%s\n' 'ERROR disk 91%' 'WARN retry 3' > fixtures/app.log
python3 app.py --format json --source fixtures/app.log > /tmp/cli.json
python3 scripts/check_json_contract.py schemas/cli_json_v1.json /tmp/cli.json
echo $?

A healthy run prints JSON contract ok: summarizer-json-v1 and exits 0. Then flip the drift switch the way an agent rename would.

CLI_JSON_DRIFT=1 python3 app.py --format json --source fixtures/app.log > /tmp/cli-drift.json
python3 scripts/check_json_contract.py schemas/cli_json_v1.json /tmp/cli-drift.json
echo $?
cat /tmp/cli-drift.json

Expected lines from that second run:

JSON contract failed:
  - missing required key: excerpt
  - undeclared key: blurb

Exit code 1. That is the whole alarm. The CLI still exited 0. The contract did not.

A type break is worth a third shot. Hand-edit /tmp/cli.json so bytes becomes "412" with quotes, then run the checker again. The report should mention bytes: expected integer, got str. Agents stringify fields for safety. Wrappers that later add the field then concatenate instead of counting.

Draft the contract off the laptop

A sample payload is enough to start the file. Some solo builders paste that payload into a remote draft box so the laptop never holds a model runtime.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project with free model access and a free server option. Operator notes for this account also include a 10-million-token free allowance on that drafting path. Those are availability notes, not hardware specs, not latency numbers, not a promise that any given draft will be correct. The contract still has to land in git and run locally against a payload the model never saw.

The safe loop is narrow. Capture one real --format json object on the machine that already runs the CLI. Sanitize paths and log text. Draft cli_json_v1.json on the free server from that object. Copy the file back. Run check_json_contract.py on a second payload. If the draft invents optional keys the CLI does not emit, delete those keys before committing. Remote drafts are generous with extras. Wrappers are not.

Keep private logs off the remote side. Fake the source string. Fake the excerpt. Key names are the only thing the contract needs.

Pass and fail at a glance

Payload vs contract	Exit	Why it matters
Exact required keys, right types	0	Wrapper still parses
Required key renamed	1	Cron still queries the old name
Extra key, additionalProperties false	1	Surprise field leaks downstream
Extra key, additionalProperties true	0	Only during a staged alias window
`bytes` sent as a string	1	jq math and Python addition both lie
Empty JSON array	1	The contract is one object
CLI process 0, contract 1	fail the change	Process success is not shape success

Paste that table into the pull request when the CLI is being improved. Reviewers can argue about aliases. They should not argue about whether .excerpt still exists.

Lies the checker will not catch

The script does not read English. An excerpt that is always the string TODO will pass. A source path that points at a file the wrapper should never see will pass if the type is string. Semantic lies need a different fixture.

It does not handle NDJSON streams, pretty-print drift inside string values, or locale-formatted numbers. It does not version HTTP APIs. It does not replace consumer-driven contracts for a fleet of services.

Unknown-key policy can be gamed. Setting additionalProperties to true for now is how the freeze dies. If a new key is intentional, cut cli_json_v2.json and move wrappers on a schedule.

Boolean-as-integer is handled. Union types are not. If a field is null on empty logs, say so in a later file. Do not hide nulls by deleting the key. Missing keys and nulls are different to jq.

Skip this freeze

Leave the files unwritten when the CLI has no JSON mode. Leave them unwritten when a team already publishes OpenAPI or JSON Schema from the same code that emits the object. Leave them unwritten for pretty TTY output meant for humans. A contract will only punish help text.

Also skip it for one-off notebooks and for binaries whose output is a file on disk, not stdout. This ritual assumes a wrapper that reads stdout and stays quiet when a key vanishes.

One-release aliases, then cut v2

If a wrapper still keys off excerpt while the CLI prefers blurb, emit both for one tagged release. Put both names in v1 with additionalProperties still false. Then ship v2 that drops excerpt, and move the crontab in the same change. Two releases. Not a forever alias bag.

Drop the ritual if schema edits take longer than the CLI change, if someone starts generating the contract from the same agent that edits app.py, or if the checker is taught to warn instead of exit 1. A warning on JSON drift is how empty digest folders return.

The schema file is the afternoon experiment. Point it at one --format json command. Keep the wrapper's key list even if the Python script gets deleted. The keys were always the product.

If a wrapper still reads a retired field, write down the old name, the new name, and the release that will drop the alias. That triple is enough to plan the crontab change. Stop there.

Build a Production-Readiness Checklist for a Free-Model CLI

Sam Rivera — Fri, 04 Sep 2026 16:59:23 +0000

Friday hit 4:41 p.m. on my laptop clock.
My tiny CLI finally returned a clean summary.

A coworker wanted it on Monday morning.
He would paste real tickets with real names.
He would paste messy Slack threads without thinking.

I froze with a green test suite in front of me.
The merge checks were green too, somehow.
So what did production-ready even mean tonight?

Green tests still do not mean production-ready, right?
They mean the happy path survived one laptop.

I needed a checklist that could fail closed.
I wanted a file a teammate could copy.

This post is that file.
Copy it, break it, or trash it when a gate stays red.

The constraint I actually had

I ship small AI CLIs as a solo builder.
There is no platform team behind me.

I sometimes point them at free model access.
I sometimes park runners on a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode currently offers both of those pieces.
I treat them as a budget, not a promise.

I will not name models here.
I will not quote quotas that can move overnight.

The checklist does not care who hosts the weights.
It cares what you can prove before a real user.

Time box: ninety minutes on a weekday evening.
Extra spend: zero dollars beyond my existing machine.

If a gate needs a paid SLA, I stop.
Abandonment is cheaper than a silent wrong summary.

What production means for this CLI

This is not five nines of uptime.
This is not a public status page either.

It means one person can run the same flags twice.
The shape stays stable. The exit codes stay boring.

A bad model answer must not leak a secret.
It must not hang the shell until lunch.

It must not print success around empty JSON.
Have you seen a CLI succeed on empty JSON?

The artifact: evidence JSON plus a checker

I keep evidence in one JSON file.
The checker is boring Python on purpose.

If a required field is missing, we do not ship.
Fail closed. Skip the maybe-Monday comments.

{
  "cli_name": "ticket-summarize",
  "contract_version": "2026-09-05.1",
  "timeout_ms": 8000,
  "max_input_chars": 4000,
  "redaction_patterns": ["sk-", "Bearer ", "-----BEGIN"],
  "empty_output_policy": "fail",
  "rollback_cmd": "git revert --no-edit HEAD && npm unlink ticket-summarize",
  "abandon_if": [
    "timeout_ms > 15000",
    "cannot pin contract_version",
    "no local fixture for empty output"
  ]
}

Save it as ship/readiness.json beside the CLI.
Wikis rot. A file next to the binary does not.

Gate 1: Freeze the contract before the model

What does success look like in one object?
I only allow title and next_action fields.

I write the contract first on purpose.
Then I let the model fill those two fields.

# contract.py
from pydantic import BaseModel, Field

class Summary(BaseModel):
    title: "str = Field(min_length=8, max_length=80)"
    next_action: str = Field(min_length=8, max_length=160)

If the model adds helpful extra keys, I drop them.
Why keep surprises in a ticket summary?

Fail closed: invalid schema exits with code 2.
Exit 0 is reserved for a valid Summary object.

Gate 2: Pin a timeout you can actually feel

Eight seconds is my default ceiling.
Users wait a bit, then they hit Ctrl-C.

A free server can stall without warning.
Your shell should not stall with it.

import urllib.request
import sys

req = urllib.request.Request(url, data=payload, method="POST")
try:
    with urllib.request.urlopen(req, timeout=8) as resp:
        body = resp.read(64_000)
except TimeoutError:
    sys.exit("timeout: abandon this run")

Need more than fifteen seconds for one summary?
I abandon the production idea the same night.

Why ship a hang dressed up as intelligence?

Gate 3: Empty output is a checked failure fixture

I keep fixtures/empty.json in the repo.
The fixture file contents are literally empty braces.

Every build runs the parser against that fixture.
Does it exit 2? If not, the gate is red.

python parse_summary.py fixtures/empty.json
echo $?   # must print 2

I use the same exit code for truncated JSON.
I refuse any best-effort sentence on failure.

Here is the parser I actually keep.

# parse_summary.py
import json, sys
from contract import Summary

raw = open(sys.argv[1]).read().strip()
if not raw:
    sys.exit(2)
try:
    data = json.loads(raw)
    Summary.model_validate(data)
except Exception:
    sys.exit(2)
print("ok")

If this gate is red, the CLI stays on my laptop.
No coworker. No Monday. No fake confidence.

Gate 4: Pin the prompt, not a vibe

I version the prompt like application code.
That means a file plus a hash.

sha256sum prompts/summarize.v1.txt > ship/prompt.sha256

Store that hash inside readiness.json too.
Prompt drift is a ship blocker, not a footnote.

Do I know which model slot answered me?
I log the slot id the runtime gave me.

I do not hard-code vendor names I cannot verify.
The log line is enough evidence for a solo shop.

import hashlib, os, sys

def hash_file(path):
    data = open(path, "rb").read()
    return hashlib.sha256(data).hexdigest()

evidence["prompt_sha256"] = hash_file("prompts/summarize.v1.txt")
evidence["slot_id"] = os.environ.get("MODEL_SLOT", "unset")
if evidence["slot_id"] == "unset":
    sys.exit("fail-closed: no slot id")

No slot id means no production run.
Guessing is not a release strategy, is it?

Gate 5: Redact before the packet leaves the laptop

Free servers are other people's computers.
Treat every paste as hostile input.

I strip tokens, bearer headers, and PEM banners.
Then I send the leftover text, capped at 4000 chars.

import re

SECRETS = [
    r"sk-[A-Za-z0-9]+",
    r"Bearer\s+\S+",
    r"-----BEGIN [A-Z ]+-----",
]

def redact(text: str) -> str:
    out = text
    for pat in SECRETS:
        out = re.sub(pat, "[REDACTED]", out)
    return out[:4000]

If redaction wrecks the ticket meaning, I abort.
I do not send a hollow string and call it done.

Would you paste a customer key into a free model?
I would not. The gate exists so I cannot forget.

Gate 6: Record wall clock and size, then cap both

I do not invent token prices in this checklist.
I record wall clock and character counts only.

evidence["wall_ms"] = elapsed
evidence["chars_in"] = len(payload)
evidence["chars_out"] = len(body)
if evidence["wall_ms"] > 8000:
    sys.exit("fail-closed: over time budget")

Ninety minutes to wire the checklist itself.
Zero extra dollars on top of the free path.

If the free server disappears tomorrow, I still have fixtures.
I demo from fixtures, or I stop talking about Monday.

Abandonment is a feature here.
It is not a personality flaw.

Gate 7: One rollback command you already typed

A checklist without rollback is a mood board.
I refuse to ship on vibes.

I keep a command I have run on a throwaway clone.
The dry-run log lives in ship/.

git revert --no-edit HEAD
npm unlink ticket-summarize
hash -r

If unlink makes you nervous, skip global links.
Use a directory prefix instead.

alias ticket-summarize="$PWD/bin/ticket-summarize"
# rollback:
unalias ticket-summarize

Can you undo this in two minutes flat?
If not, it is not production-ready yet.

Gate 8: An evidence pack a teammate can re-run

I dump a folder, not a screenshot.
I do not accept a Slack screenshot as evidence.
I do not accept my own memory as evidence.

ship/
  readiness.json
  prompt.sha256
  fixture.empty.exit.log
  timeout.log
  redact.sample.txt
  rollback.dry-run.log

The checker only reads this folder.
It never trusts my recollection of last night.

# prod_ready.py
import json, pathlib, sys

root = pathlib.Path("ship")
need = [
    "readiness.json",
    "prompt.sha256",
    "fixture.empty.exit.log",
    "timeout.log",
    "redact.sample.txt",
    "rollback.dry-run.log",
]
missing = [n for n in need if not (root / n).exists()]
if missing:
    print("missing:", ", ".join(missing))
    sys.exit(2)

data = json.loads((root / "readiness.json").read_text())
if data.get("empty_output_policy") != "fail":
    sys.exit("fail-closed: empty output must fail")
if int(data.get("timeout_ms", 99999)) > 15000:
    sys.exit("fail-closed: timeout too loose")
print("production checklist: green")

Run it from the repo root.

python prod_ready.py

Red output means I message the coworker immediately.
The message is short: not this week.

The ninety-minute sequence

Write contract.py and the empty fixture.
Add parse_summary.py and confirm exit 2.
Wrap the HTTP call in an eight-second timeout.
Hash the prompt and store it under ship/.
Run redaction on a sample ticket with a fake key.
Type the rollback command once and save the log.
Run prod_ready.py and stop on any missing file.

That list is the whole evening ritual.
There is no architecture review on this path.
There is no extra committee on the calendar.

Where a free model and a free server fit

I use them on the canary path only.
I keep customer secrets off that path.
I keep payroll notes off that path too.

MonkeyCode's free model access and free server option can exercise gates 2 through 6.
Those options are not an uptime contract.

They are a networked sandbox I can fail against.
If a gate needs guaranteed uptime, I wait.

I ship a fixture-only demo instead.
Want to point the same checker at a free slot?

Export MODEL_SLOT, drop files in ship/, and run prod_ready.py.
That is the only invite in this post.

Who should not use this

Do not use this checklist for medical text.
Do not use it for legal advice or payroll files.

Do not use it if you cannot redact pastes.
Do not use it if you need a streaming UI.

Do not use it to hide a missing contract.
The contract is the product, not the model.

Teams with a real SRE rotation have better gates.
They can steal the empty fixture idea.
They can ignore the rest of this post.

Limitations I will not hide

The checker cannot see silent prompt drift on a host.
It only sees the hash you stored locally.

A free server can change answers between runs.
The timeout gate catches hangs, not clever nonsense.

Schema checks catch shape, not a wrong next action.
A valid object can still be unhelpful.

This is production-readiness for a solo CLI.
This checklist is not a compliance program.
This checklist is not a vendor audit either.

Fail-closed criteria, one copyable list

Missing ship/ file: exit 2.
Empty or truncated model output: exit 2.
Run over 8s, or config over 15s: abandon.
Unset slot id: abandon.
Redaction would destroy the ticket: abort the send.
Rollback never dry-run: do not link the binary.

If any bullet turns into an argument, the answer is no.
I used to argue with myself at 4:41 p.m.

The file argues faster than I do.

What field is missing from your readiness.json?
Tell me the constraint. I will add the gate before the next ship.

A Schema-Driven Test Data Factory on Free Infrastructure

Sam Rivera — Fri, 04 Sep 2026 13:18:49 +0000

Monday morning. My local database has 14 tables and zero rows. The feature I'm building needs realistic test data. Hand-writing fixtures would take an hour. Copy-pasting the same three names feels wrong.

So I built a small factory. It takes a schema, asks a free model to fill it, and returns JSON. The whole thing runs on MonkeyCode's free server option. No paid tier. No local GPU.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why hand-written fixtures lie

Hand-written fixtures are too clean. They use the same three names. The same two dates. The same one product. Your code passes tests because the data is too predictable.

Generated data is messier. It exposes edge cases. It makes your pagination actually paginate. It finds the bug where your code assumes every email is unique.

The architecture

Three pieces. A schema definition. A model call. A validation pass.

The schema is a list of tables with fields and types. The model reads it and generates rows. The validator checks the output against the schema. Anything invalid gets regenerated.

Step 1: Define the schema

{
  "users": {
    "count": 25,
    "fields": {
      "id": "integer auto_increment",
      "name": "string full_name",
      "email": "string email",
      "created_at": "datetime within_last_year"
    }
  },
  "orders": {
    "count": 80,
    "fields": {
      "id": "integer auto_increment",
      "user_id": "integer reference users.id",
      "amount": "float between 5 and 500",
      "status": "enum pending,paid,refunded"
    }
  }
}

The field descriptions are hints, not strict types. The model decides how to fill them.

Step 2: The server

# data_factory.py — runs on the free server
from fastapi import FastAPI, Request
from openai import OpenAI
import os, json

app = FastAPI()
client = OpenAI(
    base_url=os.environ["MONKEYCODE_BASE_URL"],
    api_key=os.environ["MONKEYCODE_API_KEY"],
)

@app.post("/generate")
async def generate(req: Request):
    schema = await req.json()
    prompt = (
        "Generate realistic test data for this database schema. "
        "Return JSON where each key is a table name and each value is a list of row objects. "
        "Respect field types. Use varied realistic values. "
        "Do not repeat the same name, email, or amount more than twice.\n\n"
        + json.dumps(schema, indent=2)
    )
    r = client.chat.completions.create(
        model=os.environ.get("MONKEYCODE_MODEL", "default"),
        messages=[{"role": "user", "content": prompt}],
        temperature=0.8,
        max_tokens=2000,
    )
    return json.loads(r.choices[0].message.content)

About 25 lines. Temperature 0.8 because we want variety, not determinism.

Step 3: Validate the output

The model sometimes returns malformed JSON or wrong types. The validator catches that.

def validate(data, schema):
    errors = []
    for table, spec in schema.items():
        if table not in data:
            errors.append(f"missing table: {table}")
            continue
        rows = data[table]
        if len(rows) != spec["count"]:
            errors.append(f"{table}: expected {spec['count']} rows, got {len(rows)}")
        for row in rows:
            for field, ftype in spec["fields"].items():
                if field not in row:
                    errors.append(f"{table}.{field}: missing")
                elif ftype.startswith("integer") and not isinstance(row[field], int):
                    errors.append(f"{table}.{field}: expected int, got {type(row[field]).__name__}")
                elif ftype.startswith("float") and not isinstance(row[field], (int, float)):
                    errors.append(f"{table}.{field}: expected float, got {type(row[field]).__name__}")
    return errors

The validator doesn't fix mistakes. It tells you what to regenerate.

What I measured

I ran the factory against three schemas. A two-table e-commerce schema. A five-table blog schema. A ten-table SaaS schema. Each schema ran three times.

Schema	Tables	Rows	Valid JSON	Schema-compliant
E-commerce	2	105	3/3	3/3
Blog	5	210	3/3	2/3
SaaS	10	450	3/3	1/3

The two-table schema worked every time. The ten-table schema failed twice. The model forgot fields. It mixed up foreign keys. It generated duplicate emails.

The failure point is context length. A ten-table schema with field descriptions is a long prompt. The model starts strong and degrades.

Where it broke

Three failure patterns repeated.

Field amnesia. The model generated 8 of 12 fields for a table. The missing fields were always at the end of the schema definition.

Foreign key drift. The model generated user_id values that didn't exist in the users table. It didn't correlate across tables.

Duplicate values. With 80 orders, the model reused the same amounts. The "do not repeat" instruction works for small counts and fails for large ones.

The fix that worked

I split generation into per-table calls. One table at a time. The model sees a smaller prompt and produces better output.

for table, spec in schema.items():
    prompt = f"Generate {spec['count']} rows for this table: {json.dumps({table: spec})}"
    # call model, validate, append

This raised the ten-table schema success rate from 1/3 to 3/3. The cost is more requests. The quality is worth it.

Who should not use this

Skip this if you need:

Exact foreign key integrity. The model won't maintain referential integrity across tables.
Deterministic output. Temperature 0.8 means every run is different.
Large datasets. 10,000 rows will take too long and hit rate limits.

This is for development and testing. Not for staging with production-scale data.

Limitations

My numbers are one run on one day. Free quotas change. The model name is configurable because it will change.

The validator checks types and presence. It doesn't check semantics. A "name" field could contain "John" or "J" or "x". The model decides.

Try it

Copy the server. Copy the validator. Define your schema. Run it.

MonkeyCode's free tier includes 10M tokens and a free server option. My three-schema test used about 18,000 tokens. That's 0.18% of the allowance.

The factory is now part of my local dev setup. I run it once, get 450 rows of test data, and delete the database when I'm done.

One question I keep coming back to. How do you validate semantic correctness? My validator checks types, not meaning. A model could generate "price: -5" and my validator would pass it. What's your approach?

Token Math: What 10 Million Free Tokens Actually Buy a Solo Developer

Sam Rivera — Fri, 04 Sep 2026 11:00:47 +0000

The number that stopped me

I did the math last Tuesday. It stopped me cold.

Ten million tokens sounds like a fortune. Until you divide it by what one call actually eats.

I was building a small agent. Each run pulled 30 issues, summarized each one, and logged the result. Nothing fancy.

The math: 30 issues × 800 input tokens each = 24,000 tokens per run. Plus 150 output tokens per summary = 4,500. Total: 28,500 tokens per run.

At that rate, 10 million tokens lasts 350 runs. If the job runs hourly, that's 14 days. Two weeks. Then the quota is gone.

Free tiers are not infinite. They are budgets with a number on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free tier numbers I reference are MonkeyCode's current offering: 10 million tokens and a free server. Quotas change. Verify before you build.

Why "calls" is the wrong unit

Most people think in calls, not tokens. That's the mistake.

A call has three parts:

The system prompt. Often 200–500 tokens, paid every time.
The input. Your data, your context, your diff.
The output. The model's answer.

Here's a table I use now:

Task	Input tokens	Output tokens	Total per call
Summarize a GitHub issue	800	150	950
Review a small PR diff	4,000	600	4,600
Fix broken JSON	1,500	300	1,800
Classify a support ticket	500	50	550
Extract fields from a PDF page	3,000	200	3,200

Now divide 10 million by each row. You get the real number: how many tasks you can actually run.

Build a 40-line budget calculator

I wrote a small CLI to stop guessing. It takes your prompt size, your data size, and your expected output. It tells you how many runs your quota survives.

#!/usr/bin/env python3
"""token_budget.py — how many runs does your free quota survive?"""
import sys

QUOTA = 10_000_000  # tokens per month, replace with your actual quota

def estimate_tokens(system_prompt: str, data: str, max_output: int) -> int:
    # Rough heuristic: 4 chars ≈ 1 token for English text.
    return len(system_prompt) // 4 + len(data) // 4 + max_output

def main() -> None:
    if len(sys.argv) < 4:
        print("Usage: token_budget.py 'system prompt' 'data file' max_output")
        sys.exit(1)

    system_prompt = sys.argv[1]
    data = open(sys.argv[2]).read()
    max_output = int(sys.argv[3])

    per_call = estimate_tokens(system_prompt, data, max_output)
    runs = QUOTA // per_call

    print(f"Estimated tokens per call: {per_call:,}")
    print(f"Runs your quota survives: {runs:,}")
    print()
    print("At these frequencies, the quota lasts:")
    for label, runs_per_day in [("every minute", 1440), ("every 10 min", 144), ("hourly", 24)]:
        days = runs / runs_per_day
        print(f"  {label}: {days:.1f} days")

if __name__ == "__main__":
    main()

Run it:

python3 token_budget.py "Summarize this issue in 80 words." issue.txt 150

Output:

Estimated tokens per call: 950
Runs your quota survives: 10,526
At these frequencies, the quota lasts:
  every minute: 7.3 days
  every 10 min: 73.1 days
  hourly: 438.6 days

Wait. 10,526 runs hourly is 438 days. But my earlier example said 350 runs. What changed?

The system prompt. My first estimate used a long instruction block with categories and examples. The calculator uses a short one. That's the whole point.

The three levers that stretch your quota

You control three numbers. Change them and the budget moves.

Lever 1: Shrink the system prompt.
Every token in your instructions is paid on every single call. Cut 200 tokens from a system prompt and you save 200 × runs. On 10,000 runs, that's 2 million tokens.

Lever 2: Trim the input.
You don't need the whole issue body. You need the first 500 characters and the title. Truncate before you send, not after.

Lever 3: Cap the output.
Set max_tokens to the smallest number that still gives a usable answer. A summary doesn't need 500 tokens. It needs 150.

Here's the same task with all three levers applied:

	Before	After
System prompt	320 tokens	80 tokens
Input	2,400 tokens	900 tokens
Max output	500 tokens	150 tokens
Total per call	3,220	1,130
Runs per 10M quota	3,105	8,849

Same task. 2.8× more runs. Zero quality loss that I could measure.

What the numbers mean for your build

Now you can make decisions before you write code.

Hourly polling agent? At 1,130 tokens per run, you get 8,849 runs. That's 368 days. The quota is not your bottleneck.
Batch job processing 10,000 records? At 3,200 tokens per record, you get 3,125 records. The quota is your wall. You need batching or a paid tier.
Interactive chat? 50 turns per user per day at 2,000 tokens per turn is 100,000 tokens per user per day. Ten users. Ten days. Done.

The last one is the trap. Interactive apps burn quotas 100× faster than batch jobs.

The failure mode nobody mentions

Quota exhaustion is silent.

Your calls start failing. Or worse, they start returning empty responses. Your logs fill with errors. Your users see broken features. Nobody sends you an email saying "your free tier ran out."

You need a counter. Track cumulative tokens per day. Alert yourself at 80% of quota. That's the difference between a planned migration and an emergency.

Who should not build on a free quota

Free tokens are for experiments, prototypes, and low-volume tools.

Skip them if:

You have paying users. Your uptime is their problem too.
You process sensitive data. Free servers are not a data isolation promise.
Your task is batch-heavy. 100,000 records will eat 10 million tokens in a weekend.

The part nobody sells you

Ten million tokens is a real budget. It's enough for a lot of solo builds. It's not enough for everything.

The difference is math. And math is the one thing you can control.

Calculate first. Build second. Watch the counter always.

What's the first task you'd put through this calculator? I'd start with the one you're already doing manually.

MonkeyCode provides free models that can run this workflow.

Build a Fail-Closed Ship Folder for a Small AI CLI

Sam Rivera — Thu, 03 Sep 2026 14:26:49 +0000

Last Tuesday I had a zip file ready.
The CLI called a free AI server.
I was one send away from a teammate.

Then I froze.
What would I keep if it failed at 2am?
A README is not evidence.

That is the whole problem.
Free models help a solo side project.
They are not a production contract.

I wanted a folder I could copy.
Eight files. One checker. Zero vibes.
If a file is missing, we do not ship.

The goal and the hard stop

Goal: hand a small AI CLI to one teammate.
Budget: forty-five minutes and zero dollars.
Backend: a free model on a free server.

Missing input: proof it should leave my laptop.
Not a blog post. Not a demo GIF.
A folder a reviewer can grep.

If the checker exits nonzero, I stop.
No "just this once" merge.
That is the fail-closed rule.

Why a folder, not another runtime gate

Runtime gates belong in CI later.
This is earlier and meaner.
This is evidence you keep next to the binary.

A gate is a check that runs once.
A ship folder is a check you reread tomorrow.
Can you explain the failure without Slack archaeology?

I usually cannot.
So we write the archaeology first.
Then the teammate can reject the zip with receipts.

What goes in `ship/`

Create this next to your CLI.
Do it before you polish flags.

mkdir -p ship out

cat > ship/GOAL.md << 'EOF'
# GOAL
User: one teammate on macOS or Linux.
Task: turn a stack trace into a 12-line summary.
Never: write files outside ./out.
EOF

cat > ship/BOUNDARY.md << 'EOF'
# BOUNDARY
Time box: 45 minutes to fill this folder.
Money: $0. Free model. Free server.
Blast radius: local ./out only.
Network: one HTTPS endpoint.
EOF

Each file has one job.
If you cannot fill it, you are not ready.
Would you sign that sentence in git?

1. GOAL.md — one task, one user

Three lines max. Then stop writing.
Who runs it. What it returns. What it must never do.

If the goal needs a second user, stop.
This checklist is for one-task tools.
Second users hide second failure modes.

2. BOUNDARY.md — time, cost, blast radius

Declare the walls before the demo.
Did you promise overnight cron?
Then this folder is the wrong artifact.

Cron needs a different runbook.
I am not writing that runbook today.
Forty-five minutes is the wall.

3. PROMPT_FIXTURE.json — one frozen input

Do not "try a few prompts" in chat.
Freeze one fixture you can replay.
Production is not a playground.

cat > ship/PROMPT_FIXTURE.json << 'EOF'
{
  "id": "trace-001",
  "input": "TypeError: 'NoneType' object is not subscriptable\n  File app.py, line 41, in load",
  "expect_contains": ["NoneType", "line 41"],
  "expect_forbidden": ["rm -rf", "curl | sh", "API_KEY"]
}
EOF

If the model cannot hit this fixture, we do not ship.
No extra retries. No prompt shopping after handoff.
Would you debug a moving target on someone else's laptop?

4. FAIL_CLOSED.md — what silence means

Free servers fail in boring ways.
Timeouts. Empty JSON. Partial sentences.
Write the rule in English first.

cat > ship/FAIL_CLOSED.md << 'EOF'
# FAIL_CLOSED
Empty body → exit 2.
Missing JSON key `summary` → exit 2.
Fixture miss → exit 2.
Unknown HTTP status → exit 2.
Never retry more than once.
Never print a guessed summary.
EOF

Would you rather look careful or look broken?
Broken. Always broken.
A guessed summary is a lie with extra steps.

5. ROLLBACK.md — the exit you can run tired

If I cannot roll back in one minute, I do not ship.
Tired-me is the real operator.
Clever-me does not get a vote at 2am.

cat > ship/ROLLBACK.md << 'EOF'
# ROLLBACK
1. Ctrl-C the CLI.
2. rm -rf ./out
3. git switch main
4. Tell the teammate: revert to v0, no AI path.
EOF

No database. No shared cache.
If you need those, you left the small-tool lane.
Stay in the lane.

6. COST.md — even free has a budget

Free is not infinite.
I still write a kill switch I will honor.
I do not invent a vendor quota I cannot see.

cat > ship/COST.md << 'EOF'
# COST
Backend: free model access plus a free server option.
Local cap: stop after 20 fixture runs today.
If the server asks for a card, abandon.
If two calls exceed 30s, abandon.
EOF

Write only the cap you will enforce yourself.
Anything else is marketing in a runbook.
I refuse that mix.

7. HEALTH.json — a tiny canary shape

Keep a schema, not a dashboard.
Your CLI must print this on --health.
If ok is false, exit 2.

cat > ship/HEALTH.schema.json << 'EOF'
{
  "ok": true,
  "http_status": 200,
  "bytes": 1,
  "has_summary": true,
  "fixture_id": "trace-001",
  "elapsed_ms": 1
}
EOF

No pretty table. JSON only.
Humans lie. Schemas argue back.
Which one do you want at handoff?

8. ABANDON.md — when I walk away

Shipping includes quitting.
Write the walk-away rules before the first success.
Success makes people greedy.

cat > ship/ABANDON.md << 'EOF'
# ABANDON
Walk away if:
- fixture fails twice in a row
- health JSON cannot be parsed
- rollback takes more than one minute
- I start adding a second task
EOF

Second task is how side projects die.
One task. Then stop.
Can you say no to a "tiny extra flag"?

The checker: refuse to print READY

This script is the whole product.
It does not call the model.
It only certifies the folder.

#!/usr/bin/env python3
"""Fail closed unless ./ship has required evidence."""
from __future__ import annotations

import json
import sys
from pathlib import Path

REQUIRED = [
    "GOAL.md",
    "BOUNDARY.md",
    "PROMPT_FIXTURE.json",
    "FAIL_CLOSED.md",
    "ROLLBACK.md",
    "COST.md",
    "ABANDON.md",
]


def die(msg: str, code: int = 2) -> None:
    print(f"NOT READY: {msg}", file=sys.stderr)
    raise SystemExit(code)


def main(root: Path) -> None:
    if not root.is_dir():
        die(f"missing directory {root}")

    for name in REQUIRED:
        path = root / name
        if not path.is_file():
            die(f"missing {name}")
        if path.stat().st_size < 24:
            die(f"{name} is too small to be evidence")

    goal = (root / "GOAL.md").read_text(encoding="utf-8")
    if "Never:" not in goal:
        die("GOAL.md must declare a Never: line")

    fail_closed = (root / "FAIL_CLOSED.md").read_text(encoding="utf-8")
    if "exit 2" not in fail_closed:
        die("FAIL_CLOSED.md must mention exit 2")

    rollback = (root / "ROLLBACK.md").read_text(encoding="utf-8")
    if "rm -rf ./out" not in rollback:
        die("ROLLBACK.md must include rm -rf ./out")

    abandon = (root / "ABANDON.md").read_text(encoding="utf-8")
    if "Walk away" not in abandon:
        die("ABANDON.md must say Walk away")

    fixture = json.loads((root / "PROMPT_FIXTURE.json").read_text(encoding="utf-8"))
    for key in ("id", "input", "expect_contains", "expect_forbidden"):
        if key not in fixture:
            die(f"PROMPT_FIXTURE.json missing {key}")
    if not fixture["input"].strip():
        die("PROMPT_FIXTURE.json input is empty")
    if "API_KEY" not in fixture["expect_forbidden"]:
        die("fixture must forbid leaking API_KEY")

    print("READY")
    raise SystemExit(0)


if __name__ == "__main__":
    target = Path(sys.argv[1] if len(sys.argv) > 1 else "./ship")
    main(target)

Save it as check_ship.py.
Then run the happy path.

python3 check_ship.py ./ship
# expected stdout: READY

Now the failure fixture I keep in git.
Delete rollback. Watch the checker refuse.

mv ship/ROLLBACK.md /tmp/ROLLBACK.md
python3 check_ship.py ./ship
echo "exit=$?"
# expected stderr: NOT READY: missing ROLLBACK.md
# expected exit=2
mv /tmp/ROLLBACK.md ship/ROLLBACK.md

If that does not fail, the checker is theater.
Theater is how I almost mailed the zip.
Do you still trust a green demo after this?

A tiny CLI stub that honors the folder

Label this as a template, not a benchmark.
I am not publishing latency numbers I did not record today.
The stub shows fail-closed shape only.

#!/usr/bin/env python3
"""Template CLI. Unset env vars must fail before any HTTP call."""
from __future__ import annotations

import json
import os
import sys
import time
from pathlib import Path
from urllib.error import URLError
from urllib.request import Request, urlopen


def fail(msg: str) -> None:
    print(json.dumps({"ok": False, "error": msg}))
    raise SystemExit(2)


def main() -> None:
    base = os.environ.get("AI_BASE_URL")
    key = os.environ.get("AI_API_KEY")
    if not base or not key:
        fail("missing AI_BASE_URL or AI_API_KEY")

    fixture = json.loads(Path("ship/PROMPT_FIXTURE.json").read_text())
    payload = json.dumps({"input": fixture["input"]}).encode("utf-8")
    req = Request(
        base.rstrip("/") + "/summarize",
        data=payload,
        headers={
            "content-type": "application/json",
            "authorization": f"Bearer {key}",
        },
        method="POST",
    )

    started = time.monotonic()
    try:
        with urlopen(req, timeout=30) as resp:
            raw = resp.read()
            status = resp.status
    except URLError as exc:
        fail(f"network: {exc}")

    elapsed_ms = int((time.monotonic() - started) * 1000)
    if status != 200 or not raw:
        fail(f"bad response status={status} bytes={len(raw)}")

    try:
        body = json.loads(raw.decode("utf-8"))
    except json.JSONDecodeError:
        fail("body is not JSON")

    summary = body.get("summary")
    if not isinstance(summary, str) or not summary.strip():
        fail("missing summary")

    low = summary.lower()
    for needle in fixture["expect_contains"]:
        if needle.lower() not in low:
            fail(f"fixture miss: {needle}")
    for banned in fixture["expect_forbidden"]:
        if banned.lower() in low:
            fail(f"forbidden content: {banned}")

    Path("out").mkdir(exist_ok=True)
    Path("out/summary.txt").write_text(summary + "\n", encoding="utf-8")
    print(json.dumps({
        "ok": True,
        "http_status": status,
        "bytes": len(raw),
        "has_summary": True,
        "fixture_id": fixture["id"],
        "elapsed_ms": elapsed_ms,
    }))


if __name__ == "__main__":
    main()

The first fail-closed gate is the shell, not Python.
Unset means we do not start.

export AI_BASE_URL="${AI_BASE_URL:?missing}"
export AI_API_KEY="${AI_API_KEY:?missing}"
python3 check_ship.py ./ship
python3 summarize.py

Where does a free backend enter this story?
I point AI_BASE_URL at a server that offers free model access.
I keep the key out of git.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For this template I aim those env vars at MonkeyCode's free model access and free server option.
The folder does not depend on that name.
Remove the name and the checker still works.

If you need a concrete place to point the env vars, that is the only product note.
Try the folder first. Then pick a backend.

Numbered ship procedure

Fill all eight files in ship/.
Run python3 check_ship.py ./ship.
Delete ROLLBACK.md once, confirm exit 2, restore it.
Run the CLI against the frozen fixture.
Confirm health JSON on stdout with ok: true.
Run the rollback commands once, for real.
Only then zip the CLI for a teammate.

Skip any step and you are demoing, not shipping.
Which step would you skip under time pressure?
That is the step that needs the checker.

Who should not use this

Do not use this for multi-tenant products.
Do not use this for medical, legal, or payroll text.
Do not use this if you need an SLA.

This is a solo-builder folder.
It protects a teammate from a half-built CLI.
It does not replace monitoring.

Also skip it if you cannot name one fixture.
If every prompt is "whatever the user pastes," stop.
You do not have a tool yet.

Limitations I will not hide

The checker does not prove the model is good.
It proves you wrote evidence.
Those are different claims.

Free servers change behavior without a changelog.
A passing fixture today can fail Thursday.
Re-run the checker after every prompt edit.

I did not publish latency numbers here.
I do not have a stable public benchmark for this backend.
If a number is not in your own health JSON, it is rumor.

The 45-minute box is a discipline tool.
It is not a performance claim.
If you cannot fill ABANDON.md in that box, the tool is too big.

What I do after READY

I send the zip plus the ship/ folder.
The teammate can reject it with evidence.
That is the point.

No dashboards. No weekly review ritual.
One folder. One checker. One rollback.

If the first run after handoff fails, we abandon.
We do not tune the prompt on their machine.
Tuning happens on mine, then I re-check.

Which evidence file did you fail to fill first?
GOAL, COST, or ABANDON?
That answer tells me the next template to write.

The Free-Model Canary: 60 Lines Before I Trust a Free AI Server

Sam Rivera — Wed, 02 Sep 2026 13:33:19 +0000

Last month I watched a teammate point our internal CLI at a free model endpoint. The env var looked harmless: LLM_BASE_URL. The first request returned a clean answer, so the build moved on.

Two days later the CLI started returning empty strings in the afternoon. No error, no timeout, just blank responses. The endpoint hadn't changed. Our assumptions about it had.

That's the real problem with free model servers: they come with a contract you haven't read. The contract lives in behavior, not in the README. When AI makes code cheap, the dependency surface gets wider, and every new endpoint becomes a piece of technical debt.

One project I've been checking with that mindset is MonkeyCode. It's an open-source tool that gives solo builders free models and a free server, which is a useful combination for a side project that shouldn't cost money to run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm not going to repeat a token allowance or uptime figure from its README, because I haven't benchmarked it yet. That's what the canary below is for.

Why a canary, not a benchmark

A benchmark tells you how fast a server is. A canary tells you whether you can depend on it at all. It makes six small assertions about the way an OpenAI-compatible endpoint should behave, and it fails loudly when a free server breaks the rules.

I run something like this against every new endpoint before I wire it into a project. The script is deliberately small because I want it used. A 200-line load test gets skipped; a 60-line smoke test gets committed.

The checks are boring on purpose. Bad auth should return 401. Empty messages should return 4xx. A valid prompt should return content. Quota headers should be visible. A slow worst-case call should stay under a threshold you choose. These are not performance goals; they are dependency boundaries.

The 60-line canary

Save this as free_model_canary.py:

#!/usr/bin/env python3
'''free_model_canary.py: smoke-test an OpenAI-compatible API before you build on it.'''

import sys
import time

import httpx

ENDPOINT = sys.argv[1]
API_KEY = sys.argv[2]
MODEL = sys.argv[3]
BASE = ENDPOINT.rsplit('/chat/completions', 1)[0]


def post(client, payload, headers):
    return client.post(ENDPOINT, json=payload, headers=headers, timeout=15)


def main():
    failures = []

    with httpx.Client() as client:
        # 1. A bad key must fail with 401, not hang or silently succeed.
        bad_key_headers = {'Authorization': 'Bearer definitely-wrong'}
        bad = post(
            client,
            {'model': MODEL, 'messages': [{'role': 'user', 'content': 'hi'}]},
            bad_key_headers,
        )
        if bad.status_code != 401:
            failures.append(f'auth: expected 401, got {bad.status_code}')

        # 2. An empty message list must be rejected.
        headers = {'Authorization': f'Bearer {API_KEY}'}
        empty = post(client, {'model': MODEL, 'messages': []}, headers)
        if empty.status_code < 400:
            failures.append(f'empty payload: expected 4xx, got {empty.status_code}')

        # 3. A minimal valid request must return content.
        ok = post(
            client,
            {
                'model': MODEL,
                'messages': [{'role': 'user', 'content': 'Reply with exactly BLOOP.'}],
                'max_tokens': 10,
            },
            headers,
        )
        if ok.status_code != 200:
            failures.append(f'valid request: expected 200, got {ok.status_code}')
        else:
            text = ok.json()['choices'][0]['message']['content'].strip()
            if 'BLOOP' not in text:
                failures.append(f'content: expected BLOOP, got {text!r}')

        # 4. Quota headers should be visible so you can watch your budget.
        if not any('ratelimit' in key.lower() for key in ok.headers):
            failures.append('quota headers: no ratelimit header found')

        # 5. A quick latency sample: the slowest of 5 tiny calls stays under 3s.
        samples = []
        for _ in range(5):
            start = time.perf_counter()
            post(
                client,
                {'model': MODEL, 'messages': [{'role': 'user', 'content': 'hi'}], 'max_tokens': 1},
                headers,
            )
            samples.append((time.perf_counter() - start) * 1000)
        worst_ms = max(samples)
        print(f'latency sample (ms): {[round(x) for x in samples]}, worst={round(worst_ms)}')
        if worst_ms > 3000:
            failures.append(f'latency: worst {round(worst_ms)}ms exceeds 3000ms')

        # 6. The model list endpoint should be reachable.
        try:
            models = client.get(f'{BASE}/models', headers=headers, timeout=10)
            if models.status_code != 200:
                failures.append(f'model list: expected 200, got {models.status_code}')
        except httpx.HTTPError as exc:
            failures.append(f'model list: {exc}')

    if failures:
        print('CANARY FAIL')
        for failure in failures:
            print(' -', failure)
        sys.exit(1)

    print('CANARY PASS')


if __name__ == '__main__':
    main()

What the checks catch

The auth check catches the most embarrassing failure mode: a server that accepts any key and logs nothing. If your bad-key request returns 200, your logs will be full of phantom users.

The empty-payload check validates the server's basic contract. An OpenAI-compatible endpoint should reject a message list with nothing in it. Some free servers skip this validation and store the payload, which only delays the problem.

The content check looks for a marker token. Asking the model to reply with exactly BLOOP gives you a fixed string to compare. This catches servers that return an empty choices array, or that stream content when you asked for a single response.

The quota-header check is about observability. Even a rough x-ratelimit-remaining value is enough to schedule your background jobs around the wall. No header means the wall is invisible.

The latency sample is a sanity floor, not a benchmark. Five tiny requests are enough to catch a server that is swapping models or cold-starting containers. It can't tell you about tail latency at scale.

The model-list check confirms the server knows what it is serving. A broken /models endpoint means the API surface is unstable, and your code will chase that instability later.

These six checks take about thirty seconds. For that time you get a written record of what the free model server actually promised you, instead of what its landing page promised.

How to run it

Create a clean environment and run it with three arguments:

python -m venv .venv
source .venv/bin/activate
pip install httpx
python free_model_canary.py https://your-server/v1/chat/completions your-key your-model

A passing run looks like this:

latency sample (ms): [320, 410, 288, 522, 1890], worst=1890
CANARY PASS

If the script prints CANARY FAIL, the server is telling you how it will break your build. A common failure shape is an endpoint that passes connection and JSON checks but hides quota headers. That is the quietest way a free tier fails: no warning, no budget counter, just a 429 at the worst moment.

A decision table for free model servers

A pass is a green light, not a blank one. Here's the table I use:

Where the code runs	Trust a free server after canary?	Extra guardrails
Local experiment or prototype	Yes	Keep a disposable API key
Scheduled job that can retry	Yes	Add a dead-letter queue and alarm on empty responses
Production API with external users	No	Pay for an SLA or keep a self-hosted fallback
PII, health, or financial data	Never	No free tier, no exception

The canary checks the server, not your data. If your workflow touches regulated data, this script is not your approval process.

Where the canary lies

The script is a snapshot. It does not prove uptime over a month, fairness during a flash crowd, or the durability of your prompts. It also cannot tell you whether a free server will change its quota policy next week. Free model tiers can be generous at launch and become unusable before a hackathon ends.

That's why the quota-header check matters. A server that exposes x-ratelimit-remaining is giving you a fighting chance to detect a drain before it becomes a failure. A server that hides the limit is a background surprise.

Who should not use free model servers

If you are building something that people depend on while they sleep, a free server is a dependency without a contract. Run the canary, learn what the endpoint can do, then buy the thing you intend to depend on.

Free model access is a tool for discovery, not a substrate for a product. MonkeyCode's free models and free server make sense for prototypes, internal CLIs, and weekend experiments. My rule is simple: free is fine until the user becomes someone other than me.

If you run this canary against a free model endpoint and it fails on the quota-header check, tell me which headers the server returned. That missing field is the next thing I want to encode in the script.

Free AI Servers Drift. Here's a 6-Gate Fail-Closed Filter Before Merge

Sam Rivera — Tue, 01 Sep 2026 12:40:05 +0000

Last Tuesday, my free endpoint returned a valid JSON contract. The next call returned a summary. Same prompt. Same model label. No version bump. I almost merged code that expected a schema and instead got a paragraph.

Free tiers are not the enemy. Silent drift is.

When you wire a free AI server into your PR pipeline, you accept three facts: shared compute, changing model configs, and zero guarantee. So you need gates that fail closed. This is the checklist I now run before any AI-generated suggestion touches a merge branch.

I built these gates against an open-source gateway called MonkeyCode. Why? It gives solo devs free model access and a free server for trial workloads. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Convenient, yes. Safe by default? No. So I test every claim.

Gate 1: Pin the response contract

Your prompt must define an exact shape. For a bug triage task, I require a JSON object with severity, summary, and file fields. If the response is not parseable JSON, the gate fails immediately.

{
  "severity": "high",
  "summary": "Null pointer on empty input",
  "file": "src/parse.ts"
}

No fallback. No partial acceptance.

Gate 2: Snapshot a baseline

Run the same prompt ten times. Record output length, hashes, and tokens per call. Store those as baseline.json. Later, compare every new response against that range.

for i in $(seq 1 10); do
  curl -s your-monkeycode-endpoint -d '{"prompt":"triage this bug"}' \
    | jq -r '.output' | sha256sum
done

If the hash variance crosses an evidence threshold, the gate flags it.

Gate 3: Time-box and cost-cap

Free servers queue. You need a timeout and a token budget. I use 8 seconds and a hard cap of 600 tokens. The gate reads usage metadata from the response and rejects when either limit is hit.

if response.elapsed > 8 or response.usage.total_tokens > 600:
    reject("over budget")

Track this weekly. Drift often starts as a slow climb.

Gate 4: Apply semantic checks

Gates are not just about format. I block words like "maybe" and "I think" in a severity field. I also require that any recommended patch line appears in the actual diff. If not, fail.

forbidden = ["maybe", "I think", "perhaps"]
if any(w in text for w in forbidden):
    reject("uncertain language")

This catches models that pattern-match without reading context.

Gate 5: Run as a separate CI job

Never put this gate inside the main build. It must be an independent job that can fail loudly. Pull request merge action waits for it. If the job fails, the merge button stays red.

job:
  run: python gate.py --diff event.diff
  on_failure: block-merge

One job. One exit code. No exceptions.

This gate made the free server usable. It also made my PRs slower by four seconds. That is fine. A red build costs less than a broken release.

Gate 6: Have a human rollback condition

If the gate fails three consecutive runs, do not auto-retry. Pause the workflow and open an issue with the logs. Your robot assistant should stop and ask, not hammer the server again.

if failure_count >= 3: abort_workflow("manual review required")

Silent retries hide the problem. Make the problem visible.

Decision table for your team

Gate	Fail condition	Action
Contract	invalid JSON or missing field	block
Baseline	hash variance > 5%	warn
Budget	timeout / token cap	block
Semantic	forbidden phrase	block
Evidence	patch line absent	block
Retry	3 consecutive failures	pause

Copy this table. Adapt it to your task. Run the checklist once a week. Drift is not a model personality trait. It is an infrastructure property. Treat it that way.

Limitations

This gate does not catch subtle logic mistakes. It catches drift, not correctness. If your loop has an off-by-one error, the schema is still valid. Use unit tests for that.

Who should not use this? Teams without a follow-up human review. If you automate merges from an LLM without inspection, you are shipping a bet, not software. Also skip it if you are prototyping locally and checking everything by hand. A sandbox experiment does not need a CI ceremony.

Free servers are excellent for experiments. Production needs a fail-closed admission system. My next build is a centralized gate service so I can share evidence across repos. That only works if you tell me what your drift signal was.

What is the one response field your free endpoint silently dropped? Leave it below and I'll build the missing gate.

Free AI Coding Tools: 7 Gates Before You Wire Them Into CI

Sam Rivera — Mon, 31 Aug 2026 11:42:37 +0000

Two weeks ago, I started building a small AI tool that auto-labels PRs. The goal is simple: no more manual triage. The budget is zero. So I went hunting for a free model endpoint with a free server.

I found MonkeyCode, an open-source project that offers 10 million free tokens and a free server tier. That's enough to build something real. But free infrastructure fails differently. Rate limits, cold starts, silent timeouts. Your CI doesn't care about goodwill. It cares about exit codes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I've been burned by free endpoints before. A 429 at midnight. A cold start that ate the whole pipeline. A JSON response that was 200 OK with an empty choices array. So this time I built a gate checklist before trusting the free tier.

The checklist below applies to any free API. Run it before you wire the tool into CI. It takes 20 minutes and it can save you a 3 a.m. debugging session.

Gate 1: Authentication failure must be visible

The fastest way to know if an API is production-ready: break the key.

curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer wrong-key" \
  https://monkeycode.example.com/v1/chat

Expect a 401 and a JSON body that says invalid_token. If you get a 200 or a blank 500, that endpoint will hide outages later. You'll debug a mysterious blank response when you could have gotten a clear signal.

Gate 2: Rate limit headers must exist

Free servers throttle. Don't guess when you'll be throttled. Make the server tell you.

for i in $(seq 1 50); do
  curl -s -D - -o /dev/null https://monkeycode.example.com/v1/chat \
    -H "Authorization: Bearer $TOKEN" \
    -d '{"prompt":"ping"}' | grep -i "ratelimit"
done

Look for x-ratelimit-remaining and retry-after. If neither appears, treat the endpoint as flaky-by-design. You'll get a 429 with zero context, and your retry logic will be a shot in the dark.

Gate 3: Cold start latency is a feature you didn't ask for

Free servers sleep. The first request after idle may take 5 seconds. Your CI timeout is usually 30 seconds. That's fine. But a 5-second cold start every 5 minutes will make your pipeline crawl.

Measure the gap:

for i in $(seq 1 5); do
  curl -s -o /dev/null -w "request $i: %{time_total}s\n" \
    -H "Authorization: Bearer $TOKEN" \
    -d '{"prompt":"ping"}' https://monkeycode.example.com/v1/chat
  sleep 10
done

If the first request is an outlier, plan a warm-up job or adjust your timeout. Otherwise, your first CI job of the day will always be the slow one.

Gate 4: Oversized input must not kill the process

Models have token limits. Free proxies sometimes don't. Send a 10k-word file and watch:

head -c 20000 /dev/urandom | base64 > big.txt
curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer $TOKEN" \
  --data-binary @big.txt https://monkeycode.example.com/v1/chat

A good endpoint returns 400 with max_tokens_exceeded. A fragile one hangs or drops the connection. If it hangs, your CI job times out and you have no idea why.

Gate 5: Output quality passes a fixed benchmark

Don't trust vibes. Keep a file of 10 prompts with known-good answers. Run them before every release.

python3 verify_responses.py prompts.json

If two or more responses regress, block the merge. That's your fail-closed rule. The model will change. The benchmark will catch it.

Gate 6: Errors propagate to the caller

Wire the AI call into a test route. Trigger a timeout. Does your app return 502 or 200 with an empty string? If it returns 200, your users will see silent failures. Fix that before shipping. A gate script can only catch so much. This one requires a human to read the response.

Gate 7: Rollback is one flag away

The checklist ends where operations begin. Can you disable the AI feature in a single environment variable? If not, build that switch now. When the free tier breaks at 2 a.m., you want a one-line fix, not a deployment.

The 20-minute gate script

Here's a compact script that runs Gates 1, 3, and 4 and fails closed:

#!/usr/bin/env bash
set -euo pipefail
ENDPOINT="${1:?usage: gate.sh <endpoint>}"
TOKEN="${2:?usage: gate.sh <endpoint> <token>}"

echo "Gate 1: auth failure"
CODE=$(curl -s -o /tmp/gate1.json -w "%{http_code}" -H "Authorization: Bearer wrong" "$ENDPOINT")
[[ "$CODE" == "401" ]] || { echo "FAIL"; exit 1; }

echo "Gate 3: cold start"
FIRST=$(curl -s -o /dev/null -w "%{time_total}" -H "Authorization: Bearer $TOKEN" -d '{"prompt":"hi"}' "$ENDPOINT")
echo "first request: ${FIRST}s"
[[ $(echo "$FIRST > 10" | bc) -eq 0 ]] || { echo "FAIL"; exit 1; }

echo "Gate 4: oversized input"
CODE=$(curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $TOKEN" -d "@big.txt" "$ENDPOINT")
[[ "$CODE" == "400" || "$CODE" == "413" ]] || { echo "FAIL"; exit 1; }

echo "All gates passed"

Save it, run it monthly. Free tiers change without notice.

Who should not use this checklist

This is for solo developers and small teams. If you run regulated workloads, need 99.9% uptime, or process any personal data, don't use a free endpoint. The checklist reduces risk; it doesn't eliminate it.

The hard truth

Free tokens are a trial, not a contract. The offer that looked generous in August might be gone in September. I already plan to re-run these gates every month and keep a rollback flag ready.

What's the most unexpected way a free AI endpoint has failed in your own pipeline? I'm collecting failure logs for the next checklist round.

Your Free AI Server Will 429 at 2 AM: A Budget-Watch Checklist That Fails Loud

Sam Rivera — Sun, 30 Aug 2026 10:49:41 +0000

The first time my CLI hit a free AI endpoint, the 429 arrived at 2:47 AM. No alert. No log. Just a script that silently returned garbage to a paying user.

I'd deployed a small summarizer to a free server because the math looked great: 10,000,000 tokens, zero server cost, ship it. The math forgot one thing — budgets need guards, not hope.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that offers generous free tier access — 10 million tokens and a free server option. That's genuinely useful for solo builders. But "free" doesn't mean "unlimited," and it definitely doesn't mean "self-monitoring."

So I wrote a small checklist. Each gate has a command to verify it and a fail-closed rule. If a gate trips, the feature disables itself — no silent degradation.

The 8-Gate Fail-Closed Checklist

1. Token Budget Gate

Know your practical ceiling before you start. The 10M number is marketing; your real budget is max_tokens_per_request * requests_per_hour * hours.

# Check remaining quota if your provider exposes it
curl -s https://api.monkeycode.ai/v1/usage | jq .remaining_tokens

Fail closed: if remaining tokens drop below 20% of your daily estimate, stop new requests.

2. Latency Gate

Free servers often share CPU with noisy neighbors. One slow response stalls your entire pipeline.

Measure p95 latency over 100 requests. Keep the threshold in a config file.

./bench.sh --url $ENDPOINT --requests 100 | jq .p95_ms

Fail closed: p95 > 5s → switch to a cached stub response.

3. Response Shape Gate

LLMs return JSON that sometimes isn't JSON. Validate the schema before touching downstream code.

import jsonschema
jsonschema.validate(data, schema)  # raise on any mismatch

Fail closed: schema invalid → return a fixed error object, not the raw model output.

4. Content Safety Gate

Free models can drift under adversarial prompts. Run a tiny blocklist check on the output.

echo "$RESPONSE" | grep -iE 'drop table|rm -rf' && exit 1

Fail closed: blocklist hit → log the incident and return "unsafe output."

5. Cost Spike Gate

A bug in your retry loop can burn 10M tokens in an hour. Add a rate limiter that counts tokens per minute.

# in-memory token counter
if tokens_this_minute > 50_000:
    raise RuntimeError("budget spike")

Fail closed: spike detected → stop the worker and alert.

6. Free Server Uptime Gate

Free servers restart, move, or throttle without notice. Your code should treat the endpoint as ephemeral.

Run a health check every 60 seconds:

while true; do
  curl -sf $ENDPOINT/health || systemctl stop my-ai-service
  sleep 60
done

Fail closed: health check fails twice → shut down gracefully.

7. Data Retention Gate

Don't send sensitive data to a free endpoint you don't control. Strip PII before the request.

clean = redact_pii(text)
response = call_model(clean)

Fail closed: if PII detector can't run, refuse the request.

8. Rollback Gate

Your AI feature is a canary, not a core dependency. Keep the previous non-AI path one env var away.

if [[ "$USE_AI" == "true" ]]; then
  python ai_summarize.py
else
  python simple_truncate.py
fi

Fail closed: any gate above fails → set USE_AI=false automatically.

The 40-Line Watchdog That Ties It Together

Here's a minimal bash watchdog I run on the free server. It checks latency every 30 seconds and flips a flag when the gate breaks.

#!/usr/bin/env bash
# free-tier-watch.sh
ENDPOINT="$1"
FLAG=/tmp/ai_healthy

check() {
  local code=$(curl -o /dev/null -s -w "%{http_code}" --max-time 3 "$ENDPOINT/health")
  local ms=$(curl -o /dev/null -s -w "%{time_total}" --max-time 3 "$ENDPOINT/health")
  if [[ "$code" == "200" && "$ms" < "5.0" ]]; then
    touch "$FLAG"
  else
    rm -f "$FLAG"
    echo "$(date) FAILED code=$code ms=$ms" >> /tmp/ai_failures.log
  fi
}

while true; do
  check
  sleep 30
done

Your app reads /tmp/ai_healthy. If the file is missing, it falls back to the stub path. The failure is loud, visible, and reversible.

What This Checklist Does Not Do

This does not make a free server production-grade for high concurrency. It doesn't handle multi-region failover, auth, or compliance.

Use it for internal tools, personal assistants, and low-stakes automations. If a failed response could hurt a human or cost you money, pay for a contract.

I still hit 429s. But now they wake me up with a log line instead of a silent KeyError.

What's your missing gate? Mine was token spikes — I'd love to hear the one that bit you.

Your AI Code Reviewer Is Untested: 45 Lines to Score It Before Merge Rights

Sam Rivera — Sat, 29 Aug 2026 09:38:07 +0000

Every developer is a reviewer now. The reviewer itself? No test suite.

Picture the scene. A PR lands. The AI reviewer comments "Looks good, the null check is handled." The null check was planted in that diff to be caught. The team merges.

I built a 45-line harness that stops that scene. It scores any OpenAI-compatible endpoint against seeded bugs. If the reviewer misses too many, it exits 1. Fail-closed. No merge rights without evidence.

The harness is endpoint-agnostic. I pointed it at MonkeyCode's open-source project, which offers free model access and a free server option — no credit card needed to run the experiment. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why test a reviewer like code

A reviewer is a tool. Tools get tests. Your linter has fixtures. Your formatter has snapshots. Your AI reviewer gets merge buttons.

The failure is quiet. The review is confident, formatted, and wrong. Nobody checks whether the reviewer can see the bugs that matter to your team.

I wanted three numbers before trusting any review: precision, recall, determinism. This harness gives me the first two. A saved transcript gives me the third.

The fixture: one diff, one planted bug

A fixture is a folder with two files:

fixtures/
  01-null-check/
    diff.patch
    bugs.json
  02-missing-await/
    diff.patch
    bugs.json

diff.patch is a real diff from your repo with one bug seeded in. bugs.json is the ground truth:

[{"file": "app.py", "line": 12, "issue": "calls .strip() on a value that can be None"}]

Five fixtures minimum. Ten is better. Use your own diffs, not tutorial examples. Your team's bug patterns are the ones that matter.

The harness

Python, one dependency (httpx), 45 lines:

#!/usr/bin/env python3
"""Score an AI code reviewer on seeded bugs. Exit 1 = fail closed."""
import json, os, sys
from pathlib import Path

import httpx

ENDPOINT = os.getenv("REVIEW_ENDPOINT")
API_KEY = os.getenv("REVIEW_API_KEY")
FIXTURES = Path(os.getenv("FIXTURES_DIR", "./fixtures"))
RECALL_FLOOR = float(os.getenv("RECALL_FLOOR", "0.8"))
PRECISION_FLOOR = float(os.getenv("PRECISION_FLOOR", "0.5"))

PROMPT = """Review this diff. Reply with JSON only:
[{"file": "...", "line": 0, "severity": "error", "issue": "..."}]
Report real bugs only. No style nits. Empty array if clean.

DIFF:
{diff}"""

def ask(diff: str) -> list:
    r = httpx.post(
        ENDPOINT,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"messages": [{"role": "user", "content": PROMPT.format(diff=diff)}]},
        timeout=120,
    )
    r.raise_for_status()
    return json.loads(r.json()["choices"][0]["message"]["content"])

def main() -> int:
    fixtures = sorted(FIXTURES.iterdir())
    if not fixtures:
        print("FAIL gate 1: no fixtures")
        return 1
    tp = fp = fn = 0
    for fx in fixtures:
        diff = (fx / "diff.patch").read_text()
        truth = {(b["file"], b["line"]) for b in json.loads((fx / "bugs.json").read_text())}
        found = {(f["file"], f["line"]) for f in ask(diff)}
        tp += len(truth & found)
        fp += len(found - truth)
        fn += len(truth - found)
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    print(f"precision={precision:.2f} recall={recall:.2f}")
    if recall < RECALL_FLOOR or precision < PRECISION_FLOOR:
        print("FAIL: below floor. Keep review human-only.")
        return 1
    print("PASS: reviewer may comment, never approve.")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Run it:

export REVIEW_ENDPOINT="<your-compatible-endpoint>"
export REVIEW_API_KEY="$KEY"
python test_reviewer.py   # 0 = pass, 1 = fail closed

Why 45 lines? Because a reviewer test that needs a framework never gets run. This one runs anywhere Python exists.

Six gates before merge rights

Copy these into your repo. Each gate produces one artifact.

Fixture coverage. At least five diffs with planted bugs. Artifact: the fixtures/ folder, committed.
Recall floor. Catch at least 80% of planted bugs. A reviewer that misses a planted null-check is doing vibes, not review.
Precision budget. At least 50% of findings must be real. Below that, the team learns to ignore the bot. Ignored bots are worse than no bot.
Determinism sample. Run one fixture three times. Findings should be mostly stable. Save the transcripts.
Fail-closed wiring. Timeout, bad JSON, or HTTP 500 means exit 1. The bot comments "unreviewed," not a green check.
Override log. Record every human "dismiss." If two weeks of overrides show the bot is noise, turn it off. That's the exit criterion.

Where MonkeyCode fits

The harness does not care which endpoint it points at. That is the point.

I used MonkeyCode's free model access and free server option because both fit a solo-builder budget. The free tier currently includes a 10M token allowance, enough for several hundred fixture runs. No version numbers, no benchmark claims — the harness measures what your diffs actually trigger.

Free endpoints are shared. They can be slow. The 120-second timeout handles that: when the endpoint dies, the harness fails closed. That is a feature, not a workaround.

Who should skip this

Skip this checklist if any of these describe you:

No CI at all. Run the harness locally, but don't wire it to merge.
You want AI to replace human review. This only tests whether it can comment.
Your team dismisses every bot finding. Fix the culture first. A faster bot won't help.

Limitations

Line-based matching is crude. The reviewer says "line 12 is null-unsafe." You match on file plus line. Real reviewers describe bugs in prose. Fine. This fixture is a canary, not a full benchmark.

Free tiers change. Quotas move. The harness re-measures every run. That's why it exists.

Run it before you trust the next review. Expect the first run to hurt. That's the harness earning its keep.

What bug class does your team miss most in review? I'm building fixture number six around that one.

Your Free AI Server Will Fail Quietly. Five Gates to Make It Loud.

Sam Rivera — Fri, 28 Aug 2026 03:13:50 +0000

Your Free AI Server Will Fail Quietly. Five Gates to Make It Loud.

The model can be innocent. The server cannot.

Earlier this week I wrote a fail-closed checklist for AI-generated code. That list guards against the model writing something dangerous. This list guards against something duller: the server around it dying at 2 a.m. while the model stays online the whole time.

Nobody sees that failure until a user does.

The setup

I am testing MonkeyCode for a small side build: a log-summarizing API. The project gives you free model access and a free server option, which is exactly the toy setup I like. Ten lines of app logic. Zero dollars. One honest problem: free infrastructure is someone else's best effort.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Before you judge, my plan was simple. I deliberately killed my own server to see where the stack would fail. Then I wrote gates that make each failure loud.

The kill test

Here is the failure sequence, reproduced on purpose.

The server process died. No restart policy.
Connections hit a dead socket. Nothing answered.
The client had no timeout and waited forever.
No health probe. No alert. No log line.
Four hours later, the model was still happy. The server was still dead. The tool was still broken.

The model was innocent the whole time. The harness was the guilty one. The problem was never intelligence. It was silence.

So here are five gates, ordered from cheapest to most annoying.

Gate 1: A kill switch that outlives the process

A crash bug can take down your app. It can also take down your ability to disable the app. So the switch lives outside the app.

KILL_FILE = "/tmp/disable-monkeycode"

@app.post("/summarize")
def summarize(logs: str):
    if os.path.exists(KILL_FILE):
        raise HTTPException(503, "disabled by operator")
    ...

Why a file and not a database row? Because the DB may be down when you need the switch most. A file survives restarts. You can touch it from cron. You can remove it by hand.

touch /tmp/disable-monkeycode   # fail closed
rm /tmp/disable-monkeycode      # reopen

Gate 2: A budget counter you control

Free endpoints rarely warn before they stop you. Sometimes they just return errors. So count every request yourself and stop early.

def spend(estimated_tokens: int):
    state = load_state()
    if state["tokens"] + estimated_tokens > MAX_TOKENS_PER_DAY:
        raise GateError("budget exhausted")
    state["tokens"] += estimated_tokens
    save_state(state)

The counter resets daily, lives in a JSON file, and blocks before the provider does. When in doubt, fail on the conservative side.

Gate 3: A timeout shorter than your patience

A hung call is worse than a failed call.

response = requests.post(MODEL_URL, json={"text": logs}, timeout=8)

Eight seconds. Then a clean 504. To be fair, this gate would not have saved the dead-socket outage above. It saves the other outage, the one where the endpoint hangs instead of dying. Free endpoints hang. It is practically their hobby.

Gate 4: An output canary that checks the reviewer

The model is your reviewer: it reads logs and writes a summary. Somebody has to review the reviewer.

data = response.json()
if not isinstance(data.get("summary"), str):
    return {"summary": "degraded: bad shape", "ok": False}

The exact shape check will vary. The principle will not: garbage must fail loudly, not flow downstream.

Gate 5: A health probe with a witness

A server that runs is not alive. A server that answers /healthz is.

@app.get("/healthz")
def healthz():
    return {"ok": True}

Now point a free uptime checker at it. Every minute, it calls this path. If the box dies, you get an email. This is Gate 5 because the first four already cost you one outage.

Decision table

Gate	Dev	Canary	Prod
Kill switch	ON	ON	ON
Budget counter	OFF	WARN	BLOCK
Timeout	15s	8s	8s
Output canary	WARN	BLOCK	BLOCK
Health probe	manual	30s	10s

Copy the table. Adjust the numbers to your risk appetite.

The five-minute deploy

On a free server, the whole ceremony looks like this.

git clone <your-repo> && cd app
pip install fastapi uvicorn requests
touch /tmp/disable-monkeycode
uvicorn app:app --host 0.0.0.0 --port 8080 &
curl -i localhost:8080/summarize   # expect 503
rm /tmp/disable-monkeycode
curl -s localhost:8080/healthz     # expect {"ok":true}

Test the fail-closed path before you test the happy path. The happy path is only happy by accident.

Limitations

The JSON budget file is not atomic. Two simultaneous requests can race and overspend. For a solo tool, that is fine. For anything bigger, move the counter to SQLite.

The example API has no auth. That means anyone can spend your free tokens. Ten minutes of work: require a header token.

And none of these gates create a SLA. Free infrastructure is best-effort by definition. The goal is loud failure, not false confidence.

Who should skip this

Teams with compliance requirements. Products with paying customers. Anyone whose uptime appears in a contract. For you, this checklist is the minimum, not the gold standard.

Try it

If you want to break these gates on purpose, MonkeyCode's free model access plus the free server option make a cheap lab. I broke mine within an hour. That hour taught me more than a week of reading docs.

Which gate did your last free-tier outage skip first? That is the failure field I am missing for the next version of this checklist.

DEV Community: Sam Rivera

Build a Silent-Default Kill List for Agent Diffs

Cheap diffs still cost review hours

What the packet contains

Gate 1 — freeze the question first

Gate 2 — declare allowed defaults

Gate 3 — pin tools and paths

Gate 4 — run the scanner

Gate 5 — demand evidence files

Copyable decision table

A failure fixture you can replay

Gate 6 — time box and rollback

Where a free model path fits

Who should not use this

Limitations I will not hide

What I copy tomorrow

Pin CLI JSON Keys Before an Agent Renames Them

Cron remembers the old name

Keys are the public surface

A CLI that can drift on purpose

One versioned contract file

A checker with no extra packages

Reproduce the rename

Draft the contract off the laptop

Pass and fail at a glance

Lies the checker will not catch

Skip this freeze

One-release aliases, then cut v2

Build a Production-Readiness Checklist for a Free-Model CLI

The constraint I actually had

What production means for this CLI

The artifact: evidence JSON plus a checker

Gate 1: Freeze the contract before the model

Gate 2: Pin a timeout you can actually feel

Gate 3: Empty output is a checked failure fixture

Gate 4: Pin the prompt, not a vibe

Gate 5: Redact before the packet leaves the laptop

Gate 6: Record wall clock and size, then cap both

Gate 7: One rollback command you already typed

Gate 8: An evidence pack a teammate can re-run

The ninety-minute sequence

Where a free model and a free server fit

Who should not use this

Limitations I will not hide

Fail-closed criteria, one copyable list

A Schema-Driven Test Data Factory on Free Infrastructure

Why hand-written fixtures lie

The architecture

Step 1: Define the schema

Step 2: The server

Step 3: Validate the output

What I measured

Where it broke

The fix that worked

Who should not use this

Limitations

Try it

Token Math: What 10 Million Free Tokens Actually Buy a Solo Developer

The number that stopped me

Why "calls" is the wrong unit

Build a 40-line budget calculator

The three levers that stretch your quota

What the numbers mean for your build

The failure mode nobody mentions

Who should not build on a free quota

The part nobody sells you

Build a Fail-Closed Ship Folder for a Small AI CLI

The goal and the hard stop

Why a folder, not another runtime gate

What goes in ship/

1. GOAL.md — one task, one user

2. BOUNDARY.md — time, cost, blast radius

3. PROMPT_FIXTURE.json — one frozen input

4. FAIL_CLOSED.md — what silence means

5. ROLLBACK.md — the exit you can run tired

6. COST.md — even free has a budget

7. HEALTH.json — a tiny canary shape

8. ABANDON.md — when I walk away

The checker: refuse to print READY

A tiny CLI stub that honors the folder

What goes in `ship/`