DEV Community: Casey Li

Token Buckets Do Not Belong on Free Inference

Casey Li — Sun, 20 Sep 2026 11:51:35 +0000

Admission control is a fence, not a conversation. A free language-model call is a variable-latency guess with an untrusted prompt surface, and it does not belong on the path that decides whether a request may enter a production service.

The temptation is obvious in an industry that now asks models to classify almost everything. Abuse often looks like language. Support tickets look like language. Even a User-Agent string can be dressed up as a story about intent. Teams then sketch a “smart shedder”: send the request metadata to a model, ask whether the caller looks abusive, and only then hit the real API. That design treats a control-plane verdict as if it were a paragraph.

It is not. A token bucket, a GCRA check, or an Envoy local rate-limit decision has to be cheap, deterministic, fail-closed, and cheaper than the work it protects. Inference is none of those things. Under a flood, the defender who asks a model for every admit/deny also pays for every attack packet in latency and quota. The free tier does not cancel that inversion. It makes the inversion quieter, because the bill arrives as a cold start, a queue, or a sudden 429 from the model host rather than as an invoice line.

Category error, not a missing prompt

Load shedding is physics. Tokens refill against a clock. Burst is a number. The protected resource has a measured concurrency limit. None of that becomes more true because a completion is fluent. A model can narrate why a 429 happened. It cannot be the 429.

The failure mode is not merely “the model was wrong once.” The failure mode is non-replayable policy. Two replicas given the same headers can emit different verdicts. A retry can flip from deny to admit because the sampler moved. An attacker who owns the body text owns a channel into the policy, which is the opposite of a fence. Highway toll booths do not interview drivers in natural language while cars are still rolling. They weigh, they count, they drop a barrier.

A useful test is brutally simple. If the admit path is slower than the p99 of the service behind it, the guard is theatre. If the admit path fails open when the model is cold, the guard is an invitation. If the admit path parses untrusted prose as policy, the guard is a prompt-injection gadget bolted onto the front door.

A local bucket that stays local

The following limiter is deliberately boring. It is an in-process token bucket with a monotonic clock. It is not a global cluster limiter. It exists to keep the verdict on this side of the network, in code a reviewer can step through without a trace from a vendor.

# token_bucket.py
from __future__ import annotations

import threading
import time
from dataclasses import dataclass, field


@dataclass
class TokenBucket:
    rate_per_sec: float
    burst: float
    tokens: float = field(init=False)
    updated: float = field(init=False)
    _lock: threading.Lock = field(default_factory=threading.Lock, repr=False)

    def __post_init__(self) -> None:
        if self.rate_per_sec <= 0 or self.burst <= 0:
            raise ValueError("rate and burst must be positive")
        self.tokens = float(self.burst)
        self.updated = time.monotonic()

    def allow(self, n: float = 1.0) -> bool:
        if n <= 0:
            raise ValueError("cost must be positive")
        with self._lock:
            now = time.monotonic()
            elapsed = now - self.updated
            self.updated = now
            self.tokens = min(self.burst, self.tokens + elapsed * self.rate_per_sec)
            if self.tokens >= n:
                self.tokens -= n
                return True
            return False

Wrap it so that “ask a model” cannot sneak back in through a helper named smart_allow. The point of the wrapper is not style. It is a hard refuse for any intent that is actually an admission verdict.

# admission.py
from __future__ import annotations

from token_bucket import TokenBucket

VERDICT_INTENTS = frozenset({"admit", "deny", "shed", "throttle", "ban"})


class InferenceForbidden(RuntimeError):
    pass


class AdmissionGate:
    def __init__(self, bucket: TokenBucket) -> None:
        self.bucket = bucket

    def decide(self, intent: str, cost: float = 1.0) -> str:
        if intent in VERDICT_INTENTS:
            return "admit" if self.bucket.allow(cost) else "deny"
        raise InferenceForbidden(
            f"{intent!r} is not an admission verdict; do not call this gate"
        )

    def draft_via_model(self, intent: str, llm_complete, prompt: str) -> str:
        if intent in VERDICT_INTENTS:
            raise InferenceForbidden(
                "admission verdicts must be local; refusing inference route"
            )
        return llm_complete(prompt)

A tiny test file pins the contract. The first case proves the bucket still sheds. The second case proves a forged “please classify this User-Agent” path cannot become the verdict.

# test_admission.py
import time
from admission import AdmissionGate, InferenceForbidden
from token_bucket import TokenBucket


def test_bucket_denies_when_empty():
    gate = AdmissionGate(TokenBucket(rate_per_sec=1.0, burst=1.0))
    assert gate.decide("admit") == "admit"
    assert gate.decide("admit") == "deny"


def test_refill_after_wait():
    gate = AdmissionGate(TokenBucket(rate_per_sec=10.0, burst=1.0))
    assert gate.decide("admit") == "admit"
    time.sleep(0.15)
    assert gate.decide("admit") == "admit"


def test_model_cannot_own_verdict():
    gate = AdmissionGate(TokenBucket(rate_per_sec=1.0, burst=1.0))
    def fake_llm(_prompt: str) -> str:
        return "admit"  # attacker-shaped completion
    try:
        gate.draft_via_model("admit", fake_llm, "User-Agent: friendly crawler")
        raise AssertionError("inference route should have been refused")
    except InferenceForbidden:
        pass

Run it without dressing the command up as an agent loop.

python -m pytest test_admission.py -q

That is the whole production-shaped core: a clock, a float, a lock, and a refuse. Anything that needs a paragraph happens after the status code exists.

Red flags that mean the model is already on the fence

Watch the request path, not the slide deck. A handler that concatenates method, path, headers, and a slice of the body into a prompt before it checks a counter has already moved policy into the untrusted input. A timeout around that call that defaults to admit is fail-open dressed as resilience. A cache keyed on the prompt text is still a model verdict, only delayed and still injectable.

Watch the economics next. If attack traffic increases model calls one-for-one, the shedder is an amplifier. Free quota does not change the shape. It only hides the amplifier until the quota is gone, at which point the gate either blocks legitimate traffic or swings open. Both outcomes are worse than a dumb 429 from a local bucket that never left the process.

Watch observability last. A good deny can be rebuilt from client_id, cost, tokens_left, and a monotonic timestamp. A model deny cannot. Post-incident review then becomes literary criticism of a completion that no longer exists. That is not an error budget. That is folklore.

Better alternatives, in the order they should be tried

Keep the first limiter as close to the socket as the stack allows. limit_req in nginx, Envoy’s local rate limit filter, or an API-gateway token bucket all sit in front of application code and do not parse English. If the service is already in-process, GCRA or the bucket above is enough for a single instance. When many instances must share a budget, move the counter to Redis or a dedicated limiter service with a short TTL and a fail-closed posture, not to a chat endpoint.

Identity belongs in the same local world. API keys, mTLS, and signed service tokens decide who. The bucket decides how many. Mixing those questions into one completion produces a policy no one can rotate. Rotate keys in the identity layer. Change rates in config. Leave prose out of both.

There is still a lawful place for a model, and it is downstream. After the gate has emitted deny, structured counters can be turned into an incident note, a customer-facing explanation, or a weekly cluster of rejected shapes. That work is offline or near-offline. It can retry. It can be wrong without opening the front door.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access is a fit for that downstream draft: take a JSON line of denied_total, top_paths, and retry_after_ms, and ask for a paragraph a human will edit. The free server option is a fit for running the pytest file and a throwaway process that hammers AdmissionGate without touching production sockets. Neither option is a fit for decide().

A scratch driver that stays honest looks like this. It never prints a model’s opinion about admit/deny. It only prints local verdicts, then optionally ships a summary string somewhere else.

# hammer.py
import json
from collections import Counter
from admission import AdmissionGate
from token_bucket import TokenBucket

gate = AdmissionGate(TokenBucket(rate_per_sec=20.0, burst=5.0))
counts = Counter()
for i in range(50):
    counts[gate.decide("admit")] += 1
summary = {
    "admitted": counts["admit"],
    "denied": counts["deny"],
    "n": 50,
}
print(json.dumps(summary))
# llm_complete is allowed only on summary, never on a live request.

Exit criteria for pulling inference off the fence

Leave the model on the admit path the moment any of these become true. The verdict is binary. The verdict must be replayable from logs. The verdict must fail closed. The verdict must cost less than the protected handler. The input to the verdict includes untrusted natural language. Attack volume can scale the guard’s own dependency. Two replicas must not disagree for the same key in the same window.

If three of those fire at once, the extraction is not a refactor. It is an incident. Put the bucket back in process, or in the proxy, and keep the model on the ticket that explains what already happened.

Who should not use this sketch

This article’s bucket is a single-process fence. Multi-region products that need one shared budget should not copy it into twelve stateless replicas and hope. Services that already sit behind a mature gateway should not add a second limiter in application code “for intelligence.” Teams whose compliance story requires a global, audited policy engine should use that engine, not a pytest file from a blog.

The downstream drafting step has limits of its own. A free model can invent a cause of denial that never appeared in the counters. Treat its paragraph as a draft. Keep the JSON as the record. Anyone who needs the explanation to be evidence should render the explanation from templates bound to those counters, not from a completion.

Admission control ages badly when it becomes a personality. The durable version is a small number, a clock, and a refuse. Let language models write the footnote after the barrier has already dropped. Readers who want a disposable box for the tests above can point them at MonkeyCode’s free server and keep the live fence where it already belongs: in process, on the proxy, and off the prompt.

The Lease Loop Is Not a Chat Completion

Casey Li — Sat, 19 Sep 2026 09:56:33 +0000

A fencing token is not a paragraph. It is a counter that must rise on a clock, die on a deadline, and never ask a model whether the holder still deserves the lock. Free inference has no place in that loop.

Teams keep stuffing chat completions into leader election anyway. A node goes quiet. Logs look messy. Someone wires a cheap model to read the noise and vote on who should keep the lease. The analogy is a lighthouse that phones a critic before each flash. The critic is witty. The reef does not care.

Latency is the first crack. Lease renewal is a budget measured in hundreds of milliseconds, not in token streams. A free path can stall, queue, or return a truncated object when the provider is busy. The lock expires while the prose is still arriving. Split brain is not a literary device. It is two writers sharing one offset.

Non-determinism is the second crack. The same health snapshot can yield keep, drop, or a sentence that fails JSON. Control planes need the same inputs to produce the same fence. A model that hedges is not a quorum. It is a coin with extra faces.

The third crack is authority. A completion can be filtered, rate limited, or rewritten by a safety layer the caller does not own. Leader election that depends on a vendor's refusal class has already given that vendor a veto over the write path. The veto stays invisible in dashboards that only chart HTTP 200.

What the loop actually is

A lease is a timed exclusive. The holder proves liveness by raising an epoch before expiry. Downstream storage must reject any write whose epoch is stale. The model of the world is small: holder id, epoch, expiry. Nothing in that tuple is improved by a temperature setting.

PostgreSQL can carry a fencing token without ceremony. The schema and renewer below are a worked example, not a multi-region consensus protocol. Treat them as labeled sample code.

CREATE TABLE worker_leases (
  name        text PRIMARY KEY,
  holder      text NOT NULL,
  epoch       bigint NOT NULL,
  expires_at  timestamptz NOT NULL
);

CREATE INDEX ON worker_leases (expires_at);

# lease_loop.py — keep this module free of HTTP clients and model SDKs
import os
import socket
import time
import psycopg

LEASE_NAME = "order-writer"
TTL_SECONDS = 15
RENEW_EVERY = 5
HOLDER = f"{socket.gethostname()}:{os.getpid()}"
DSN = os.environ["LEASE_DATABASE_URL"]

ACQUIRE = """
INSERT INTO worker_leases (name, holder, epoch, expires_at)
VALUES (%s, %s, 1, now() + make_interval(secs => %s))
ON CONFLICT (name) DO UPDATE
SET holder = EXCLUDED.holder,
    epoch = worker_leases.epoch + 1,
    expires_at = EXCLUDED.expires_at
WHERE worker_leases.expires_at < now()
RETURNING epoch;
"""

RENEW = """
UPDATE worker_leases
SET epoch = epoch + 1,
    expires_at = now() + make_interval(secs => %s)
WHERE name = %s
  AND holder = %s
  AND expires_at > now()
RETURNING epoch;
"""

def acquire(conn):
    with conn.cursor() as cur:
        cur.execute(ACQUIRE, (LEASE_NAME, HOLDER, TTL_SECONDS))
        row = cur.fetchone()
        conn.commit()
        return None if row is None else int(row[0])

def renew(conn):
    with conn.cursor() as cur:
        cur.execute(RENEW, (TTL_SECONDS, LEASE_NAME, HOLDER))
        row = cur.fetchone()
        conn.commit()
        return None if row is None else int(row[0])

def hold():
    with psycopg.connect(DSN) as conn:
        epoch = acquire(conn)
        if epoch is None:
            raise SystemExit("lease held elsewhere")
        while True:
            time.sleep(RENEW_EVERY)
            epoch = renew(conn)
            if epoch is None:
                raise SystemExit("lost lease; refuse all writes")
            print(f"fence={epoch}", flush=True)

if __name__ == "__main__":
    hold()

The control flow is the design. Acquire or renew returns an integer fence, or it returns nothing. There is no branch that asks a model to interpret the row. Writers that use the lease must send that epoch with every mutating call. A storage layer that cannot see the fence is not protected by the loop.

# fenced_write.py
def append_order(conn, epoch, payload):
    with conn.cursor() as cur:
        cur.execute(
            """
            INSERT INTO orders (payload, fence)
            SELECT %s, %s
            WHERE EXISTS (
              SELECT 1 FROM worker_leases
              WHERE name = %s
                AND holder = %s
                AND epoch = %s
                AND expires_at > now()
            )
            """,
            (payload, epoch, LEASE_NAME, HOLDER, epoch),
        )
        if cur.rowcount != 1:
            conn.rollback()
            raise RuntimeError("stale fence; drop the write")
        conn.commit()

A free completion that explains why the insert failed is optional theatre after the fact. It must not retry the insert, bump the epoch, or pick a new holder. Commentary can wait. Membership cannot.

A reproducible red-flag check

Opinion is cheap. Import graphs are not. The next artifact fails CI when the lease package grows an inference client. It is a proposed gate, not a claim that a production fleet already runs it.

# test_lease_has_no_inference.py
from pathlib import Path
import ast

LEASE_DIR = Path("lease")
FORBIDDEN_IMPORTS = {
    "openai",
    "anthropic",
    "httpx",
    "aiohttp",
    "requests",
    "google",
}
FORBIDDEN_NAME_FRAGMENTS = (
    "chat.completions",
    "messages.create",
    "generate_content",
)

def _imported_roots(tree: ast.AST) -> set[str]:
    roots = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for alias in node.names:
                roots.add(alias.name.split(".")[0])
        elif isinstance(node, ast.ImportFrom) and node.module:
            roots.add(node.module.split(".")[0])
    return roots

def test_lease_modules_stay_offline():
    files = list(LEASE_DIR.glob("*.py"))
    assert files, "lease package missing"
    offenders = []
    for path in files:
        text = path.read_text(encoding="utf-8")
        tree = ast.parse(text)
        bad = _imported_roots(tree) & FORBIDDEN_IMPORTS
        hits = [frag for frag in FORBIDDEN_NAME_FRAGMENTS if frag in text]
        if bad or hits:
            offenders.append(f"{path}: imports={sorted(bad)} calls={hits}")
    assert not offenders, "inference reached the lease path:\n" + "\n".join(offenders)

Run it the boring way.

mkdir -p lease
cp lease_loop.py fenced_write.py lease/
python -m pytest test_lease_has_no_inference.py -q

The test is heuristic. A determined engineer can hide a completion behind a local wrapper named health.py. That is why the gate is an exit criterion, not a personality test. If the wrapper appears, the lease package is already on the wrong side of the line. Metrics exporters belong in another process, so the import ban does not become an excuse to smuggle a model through a telemetry helper.

Field guide: when not to put a model on the fence

A free model is a useful intern for reading a postmortem. It is a reckless intern for holding the flashlight. The red flag is not "an LLM exists in the company." The red flag is a prompt that can change membership. If the text of a completion can extend a lease, drop a peer, or choose which replica drains a queue, the design has confused explanation with authority.

Timeout arithmetic is the next flag. If the model's p95 exceeds half the TTL, the loop is already insolvent. Padding the TTL to wait for tokens just lengthens the split-brain window. Schema drift in the reply is a third. Control messages that need a JSON repair step do not belong next to a fencing token. A fourth flag is softer and easier to miss: a dashboard that treats "the model said the node looks sick" as equivalent to a missed heartbeat.

Better alternatives are dull on purpose. A leases table plus make_interval covers a single primary. PostgreSQL advisory locks cover even smaller footprints. etcd elections and Consul sessions cover a cluster that already speaks consensus. ZooKeeper remains a valid, if heavy, fencing service. None of these tools ask for a system prompt, and none of them improve when the invoice for tokens is zero.

Use the table as a stop/go card during review, not as a personality quiz.

Observation	Stay on the local lease	Exit the design
Renewer import graph includes an LLM or generic HTTP client	Keep shipping the SQL loop	Fail closed; strip the client
p95 of any completion exceeds half the TTL	Do not add a model to buy time	Remove the model; shorten the path
Failure drill succeeds with inference unreachable	Model is commentary only	Keep commentary off the renewer
Failure drill fails without inference	—	The model was holding the fence
A human cannot state the fence rule in one sentence without mentioning a model	Rewrite the sentence until the model disappears	Stop the rollout

Exit when the renewer imports a network client that is not the database. Exit when a human cannot state the fence rule without mentioning a model. Exit when a failure drill cannot be run with the inference provider unreachable. If the drill cannot pass in that condition, inference was never optional.

Scratch generation is a different job. A reviewer may want extra negative fixtures: a truncated JSON body, a 429, a refusal, a slow stream. Those fixtures can be drafted on a disposable box.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are a reasonable place to generate those fixtures and to run the import checker against a throwaway clone. They are not a reasonable place to host lease_loop.py. The lease either talks to Postgres, or it does not talk.

Limitations

The example ignores clock jump, long garbage-collection pauses, and network partitions that leave the SQL session half-open. now() inside PostgreSQL is better than the client's wall clock, and it is still not Raft. Teams with a multi-region write path need a real consensus group, not a Python while True.

The CI checker does not prove liveness. It only forbids a class of imports. It will miss a sidecar that renews the lease over a Unix socket to an agent. Treat that sidecar as part of the lease package and apply the same rule. The checker will also nag a legitimate requests call if someone parks it in lease/; move the call, do not weaken the gate.

Free inference is the wrong owner of adjacent decisions that look like commentary and act like membership. Draining a Kafka partition, choosing which replica is mostly caught up, or rewriting a readiness probe because logs seem unhealthy are fencing problems in costume. The same stop/go card applies.

Who should ignore this field guide

Operators who already run etcd or ZooKeeper and never call a model from the election path can skim the checker and move on. Researchers studying lock contention on synthetic traces may use a model offline, against recorded data, with no ability to write epoch. That study is not a control loop.

The approach is a poor fit for product demos that exist to show an agent running the cluster. A demo can narrate a lease. It should not hold one. It is also a poor fit for anyone hoping a free completion will replace a quorum because the invoice is zero. The invoice is not the failure domain. Dual writers are.

A lease loop that stays boring will look under-engineered in a screenshot. That is the point. The lighthouse does not workshop the next flash. It flashes, or it goes dark, and the chart records the gap.

Privilege Grants Do Not Belong on Free Inference

Casey Li — Fri, 18 Sep 2026 08:19:12 +0000

A privilege grant is a production write. Free inference is a best-effort drafting surface. Piping a model completion into IAM, Cedar, Rego, or Kubernetes RBAC apply turns a prompt artifact into lasting access, with no named owner and no freeze hash an auditor can cite.

That path is not a merge check in different clothes. A bad CI gate fails closed, and a revert restores the tree. A noisy canary can shift traffic back. A generated PutRolePolicy or ClusterRoleBinding stays attached after the chat ends. Tokens evaporate. Grants do not.

Vibe-coded YAML has made the swap easy to miss. A model will emit a document that unblocks a deploy and reads like engineering. The file is still a completion. Naming it policy.json does not create a control.

The leak is usually mundane. A developer pastes an access-denied error into a free endpoint, asks for a fix, and receives a role policy that adds s3:* on *. The next line in the same shell is aws iam put-role-policy. Nothing in that sequence recorded a reviewer, a content hash, or an expiry. The model was treated as a notary. It is not one.

A useful analogy is a night clerk cutting master keys from a plausible memo. The memo can be well written. The metal still opens the building. Authorization documents deserve the same suspicion as key cutting. The rest of this article treats generated privilege documents as untrusted input, keeps free-model drafting in a sandbox, and requires a human-signed freeze file before any apply CLI is allowed to run. The gate is deliberately boring. Boring is the point.

What the apply path must prove

An apply path for identity policy has to answer three facts in code, not in chat: who reviewed this exact byte string, when that review expires, and which actions or resources the document is still forbidden to contain even after review. Those checks belong next to the policy file, in version control, runnable on a laptop. They do not belong inside the model.

A completion can narrate least privilege while emitting a wildcard. Syntax and narration are different jobs. The artifact below is a small Python gate plus tests. It is a proposal, not a measured production deployment. Teams should replace the reviewer identity and the forbidden-action set with their own, and should not treat the timestamps in the samples as a vendor SLA.

A freeze file the apply wrapper can trust

The freeze file is the only document the wrapper believes. It binds a reviewer identity to the SHA-256 of the policy bytes and to a short expiry. If the policy is edited by one character, the hash dies and apply dies with it.

{
  "policy_sha256": "replace-with-real-hash",
  "reviewer": "security-oncall@example.com",
  "reviewed_at": "2026-09-18T12:00:00Z",
  "expires_at": "2026-09-19T12:00:00Z",
  "ticket": "SEC-1842",
  "source": "human-authored"
}

The source field is not decoration. Values other than human-authored fail closed. Drafts produced on a free inference endpoint can sit beside the policy as draft.md. They must not be the source of the grant. A model name in reviewer fails closed for the same reason: a reviewer has to survive a staffing change.

# policy_apply_gate.py
# Proposal: refuse IAM-shaped JSON unless a human freeze file matches.
from __future__ import annotations

import datetime as dt
import hashlib
import json
import sys
from pathlib import Path
from typing import Any

FORBIDDEN_ACTIONS = {
    "*",
    "iam:*",
    "iam:PassRole",
    "iam:CreateUser",
    "iam:AttachUserPolicy",
    "iam:PutUserPolicy",
    "sts:AssumeRole",
    "s3:*",
    "kms:*",
    "ec2:*",
}


class PolicyRefused(Exception):
    pass


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def iter_statements(policy: dict[str, Any]) -> list[dict[str, Any]]:
    document = policy.get("PolicyDocument", policy)
    statements = document.get("Statement", [])
    if isinstance(statements, dict):
        return [statements]
    if not isinstance(statements, list):
        raise PolicyRefused("Statement must be an object or a list")
    return statements


def refuse_wild_grants(policy: dict[str, Any]) -> None:
    for stmt in iter_statements(policy):
        effect = stmt.get("Effect", "Allow")
        actions = stmt.get("Action", [])
        resources = stmt.get("Resource", [])
        if isinstance(actions, str):
            actions = [actions]
        if isinstance(resources, str):
            resources = [resources]
        if effect == "Allow" and not stmt.get("Sid"):
            raise PolicyRefused("Allow statements need a stable Sid")
        if "NotAction" in stmt or "NotResource" in stmt:
            raise PolicyRefused("NotAction/NotResource is not allowed on this path")
        if effect != "Allow":
            continue
        for action in actions:
            if action in FORBIDDEN_ACTIONS or str(action).endswith(":*"):
                raise PolicyRefused(f"forbidden action {action}")
        for resource in resources:
            if resource == "*":
                raise PolicyRefused("wildcard resource")
            if str(resource).startswith("arn:aws:iam::") and str(resource).endswith("*"):
                raise PolicyRefused(f"forbidden identity resource {resource}")


def refuse_stale_or_foreign_freeze(policy_bytes: bytes, freeze: dict[str, Any]) -> None:
    expected = freeze.get("policy_sha256")
    actual = sha256_bytes(policy_bytes)
    if expected != actual:
        raise PolicyRefused("freeze hash does not match policy bytes")
    if freeze.get("source") != "human-authored":
        raise PolicyRefused("freeze source must be human-authored")
    reviewer = freeze.get("reviewer") or ""
    if "@" not in reviewer:
        raise PolicyRefused("reviewer must be an identity, not a model name")
    expires = dt.datetime.fromisoformat(str(freeze["expires_at"]).replace("Z", "+00:00"))
    now = dt.datetime.now(dt.timezone.utc)
    if expires <= now:
        raise PolicyRefused("freeze expired")
    if expires - now > dt.timedelta(hours=36):
        raise PolicyRefused("freeze window is too wide for a privilege grant")


def evaluate(policy_path: Path, freeze_path: Path) -> None:
    policy_bytes = policy_path.read_bytes()
    policy = json.loads(policy_bytes.decode("utf-8"))
    freeze = json.loads(freeze_path.read_text(encoding="utf-8"))
    refuse_wild_grants(policy)
    refuse_stale_or_foreign_freeze(policy_bytes, freeze)


def main(argv: list[str]) -> int:
    if len(argv) != 3:
        print(
            "usage: policy_apply_gate.py <policy.json> <POLICY_REVIEW.json>",
            file=sys.stderr,
        )
        return 2
    try:
        evaluate(Path(argv[1]), Path(argv[2]))
    except (PolicyRefused, OSError, json.JSONDecodeError, KeyError, ValueError) as exc:
        print(f"REFUSE APPLY: {exc}", file=sys.stderr)
        return 1
    print("freeze ok; still requires a human to run the cloud CLI")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

The last print line is load-bearing. A green gate is not apply. It is permission for a human to type the cloud command. Wrapping aws iam put-role-policy inside the same script would recreate the original bug with extra ceremony. The wrapper stops at the door. A person still has to turn the key.

Tests that fail closed

Chat transcripts hide wildcards behind fluent explanations. Tests do not. The cases below are meant to run on a laptop with no cloud credentials, which is the only environment this proposal trusts for drafting.

# test_policy_apply_gate.py
import json
from pathlib import Path

import pytest

from policy_apply_gate import PolicyRefused, evaluate, sha256_bytes

SAFE = {
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "ReadOneBucketPrefix",
        "Effect": "Allow",
        "Action": ["s3:GetObject"],
        "Resource": "arn:aws:s3:::logs-nonprod/app/*",
    }],
}


def write_pair(tmp: Path, policy: dict, freeze_extra: dict) -> tuple[Path, Path]:
    policy_path = tmp / "policy.json"
    raw = json.dumps(policy, indent=2).encode("utf-8")
    policy_path.write_bytes(raw)
    freeze = {
        "policy_sha256": sha256_bytes(raw),
        "reviewer": "sec@example.com",
        "reviewed_at": "2026-09-18T12:00:00Z",
        "expires_at": "2026-09-19T00:00:00Z",
        "ticket": "SEC-1",
        "source": "human-authored",
    }
    freeze.update(freeze_extra)
    freeze_path = tmp / "POLICY_REVIEW.json"
    freeze_path.write_text(json.dumps(freeze), encoding="utf-8")
    return policy_path, freeze_path


def test_wildcard_action_refused(tmp_path):
    policy = json.loads(json.dumps(SAFE))
    policy["Statement"][0]["Action"] = "s3:*"
    policy_file, freeze_file = write_pair(tmp_path, policy, {})
    with pytest.raises(PolicyRefused, match="forbidden action"):
        evaluate(policy_file, freeze_file)


def test_model_source_refused(tmp_path):
    policy_file, freeze_file = write_pair(
        tmp_path, SAFE, {"source": "free-inference-draft"}
    )
    with pytest.raises(PolicyRefused, match="human-authored"):
        evaluate(policy_file, freeze_file)


def test_hash_mismatch_refused(tmp_path):
    policy_file, freeze_file = write_pair(
        tmp_path, SAFE, {"policy_sha256": "00" * 32}
    )
    with pytest.raises(PolicyRefused, match="hash"):
        evaluate(policy_file, freeze_file)


def test_star_resource_refused(tmp_path):
    policy = json.loads(json.dumps(SAFE))
    policy["Statement"][0]["Resource"] = "*"
    policy_file, freeze_file = write_pair(tmp_path, policy, {})
    with pytest.raises(PolicyRefused, match="wildcard resource"):
        evaluate(policy_file, freeze_file)


def test_narrow_human_policy_passes(tmp_path):
    policy_file, freeze_file = write_pair(tmp_path, SAFE, {})
    evaluate(policy_file, freeze_file)

Commands stay short so the ritual is harder to skip than to run.

python -m pytest -q test_policy_apply_gate.py
python policy_apply_gate.py policy.json POLICY_REVIEW.json
# only then, and only by a human:
# aws iam put-role-policy --role-name app-nonprod \
#   --policy-name read-logs --policy-document file://policy.json

A generated draft can still exist. It lives in draft.md and is treated like a rubber duck that types. The duck does not receive an AWS credential. If the draft is useful, a human retypes the intent into the allowlist the tests already understand, then signs the freeze.

Where free inference still belongs

Drafting an explanation of a deny, comparing two already-written statements, or generating extra unit cases for a known-safe action list are read-mostly jobs. They fail open in the worst case: a bad explanation wastes time. They do not mint access. That is the only loop this article is willing to put on a free endpoint.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that read-mostly loop on a machine that has no cloud admin keys. The gate above does not require the product. If every mention of it were removed, the freeze file and the tests would still be the method.

Paste of live account IDs, production ARNs, or session tokens into any remote draft endpoint remains a bad idea, free or not. A sandbox that cannot reach iam.amazonaws.com is the right shape. A sandbox that can apply is just a slower production console.

Red flags that mean stop, not prompt again

The first red flag is a single shell history that contains both a model CLI and put-role-policy, kubectl apply, terraform apply, or gcloud projects add-iam-policy-binding. Adjacent commands are a coupled system. Split them across users if the team cannot keep them apart on one laptop.

The second is a policy without Sid values, or with Sid values that change every generation. Unstable Sids make later revocation a scavenger hunt. A grant that cannot be named cannot be ended on purpose.

The third is any Allow that uses NotAction, *, or a service-star such as iam:*. Models reach for these because they clear the error the human pasted. Clearing an error is not the same as stating intent. The fourth is a freeze that lists the model as reviewer, or a ticket field filled with chat residue. The fifth is asking an endpoint to make the policy work in prod while the working tree already contains account IDs. At that point the prompt is no longer a design aid. It is an unsanctioned change window.

When those flags appear, the exit is mechanical. Delete the generated file. Rotate any credential that sat in the same environment. Rewrite the policy from an allowlist the team already uses in nonprod. Run the tests. Only then open a review that a human can sign. Prompting again is not an exit. It is a second attempt to cut the same key.

Better alternatives than a better prompt

Human-authored templates with substitution parameters beat generated documents. A template that can only fill a bucket name cannot invent iam:CreateUser. Permission boundaries and service control policies then cap what a mistaken inner policy can do even if someone later bypasses the laptop gate.

Policy unit tests, whether this gate, Cedar authorization tests, or OPA/Rego unit tests, beat natural-language assurances. A paragraph that says least privilege is not an oracle. A failing test is. Cloud-native analyzers that read the actual policy after apply are a detection layer, not an author. They belong after a human grant, as a tripwire, not as a substitute for the freeze.

None of those alternatives depend on a free inference quota. They depend on a reviewer who can be named in an incident channel after the context window has closed.

Limitations and who should not use this gate

The gate is syntactic. It will allow a narrow-looking s3:GetObject on the wrong bucket if the reviewer is careless. It will refuse some legitimate break-glass documents that security may still need under a different, slower process. It does not prove semantic least privilege, and it does not watch for identity-based policies attached outside this file.

It also does not make a free endpoint a system of record. Prompt logs are incomplete audit trails. They may be retained by a third party. They may disappear. Neither outcome is acceptable as the history of a privilege grant.

Teams that must generate policy on the apply path in response to incidents should not use this approach. They need a pre-authorized break-glass role with a recorded session, not a model. Teams that cannot name a human reviewer should not use it either. The freeze file would become fiction. Platform groups that already pipe completions into Terraform should not treat this script as a bolt-on blessing. The coupling has to be cut first. Otherwise the gate becomes a sticker on an automated grant factory.

Free inference remains useful as a reader and a critic sitting beside a key cutter, never as the cutter. Privilege is a write. Writes need owners who still work at the company after the session ends. If a keyless sandbox helps a team rehearse deny explanations before review, that loop can live on a free server. The grant itself stays on a hash a human is willing to put a name under.

Live Schema Changes Do Not Belong on Free Inference

Casey Li — Thu, 17 Sep 2026 06:33:48 +0000

Live schema change is a custody problem, not a prompting problem. Free inference can help a team phrase a migration plan, but it should not author the DDL that a runner applies, and it should not hold credentials for migrate up in any database that already stores user rows.

The public argument that models already write code better than most developers does not move that line. Fluency is a sampling process. A migration is a one-way contract with live data. Sampling can be retried and thrown away. Contracts cannot. A free endpoint that stalls, remaps, or drifts in quality mid-call is a poor signer of that contract.

A loading dock is the right picture. Scrap paper is fine for sketching how a crate should be packed. The stamp that lets the crate leave the building is a different instrument. Free-tier inference behaves like scrap paper. Capacity appears and disappears. Output is not a pinned binary. The same prompt can return a conservative ADD COLUMN on one call and a full table rewrite on the next. Stamping a production crate with that output is how a quiet outage is born.

The failure usually arrives dressed as convenience. A workflow asks a free endpoint for SQL that will make a Go struct match production, then a job executes the reply. The same endpoint is later asked whether that SQL is safe, which is a circular verdict rather than a review. A timeout retries the completion while the migrator is not idempotent, so two ALTER statements land. No model snapshot is stored next to the migration file, so a bad deploy cannot be bisected back to the author. The statement includes DROP, a narrowing cast, a column rename that will break readers still in flight, or a rewrite that takes an ACCESS EXCLUSIVE lock on a hot table.

None of those incidents mean the model is uniquely bad at SQL. They mean the pipeline mixed a pencil with a master key. Drafting a plan and applying a plan are different jobs. Free inference can hold the pencil. It should not hold the key.

The control that follows is blunt on purpose. Intent is classified with deterministic rules, not with another model. Asking a free endpoint whether a free endpoint should be allowed to apply DDL repeats the original mistake. If the path is schema apply, any provider marked free is refused. Review notes may still be generated. The apply client talks only to a migrator the team already runs in CI, with the file bytes hashed and a human ACK recorded.

The snippet below is a proposal. It has not been executed against a vendor and is not a claim about any hosted quota or model catalog.

from dataclasses import dataclass
from enum import Enum
import hashlib
import re

class ProviderClass(Enum):
    FREE = "free"
    PINNED = "pinned"
    LOCAL = "local"

DDL_APPLY = re.compile(
    r"\b(alter\s+table|drop\s+table|drop\s+column|create\s+index|create\s+table)\b",
    re.I,
)
DESTRUCTIVE = re.compile(
    r"\b(drop\s+table|drop\s+column|truncate|alter\s+column\s+\w+\s+type)\b",
    re.I,
)

class CustodyError(RuntimeError):
    pass

@dataclass(frozen=True)
class ApplyRequest:
    provider_class: ProviderClass
    sql: str
    human_ack: bool
    environment: str  # prod | staging | dev
    file_bytes: bytes

def migration_digest(file_bytes: bytes) -> str:
    return hashlib.sha256(file_bytes).hexdigest()

def assert_apply_allowed(req: ApplyRequest) -> None:
    if req.provider_class is ProviderClass.FREE:
        raise CustodyError("free inference cannot own DDL apply")
    if req.environment in {"prod", "staging"} and not req.human_ack:
        raise CustodyError("human ACK required outside disposable databases")
    if req.environment == "prod" and DESTRUCTIVE.search(req.sql):
        raise CustodyError("destructive DDL in prod needs an online schema changer")
    if DDL_APPLY.search(req.sql) and not req.file_bytes.strip():
        raise CustodyError("apply path requires hashed migration bytes, not a chat log")

Closed tests should fail closed. Missing provider class is a refusal, not a default to free. A chat transcript is not an input. The file that is about to run has to be the same bytes that a reviewer signed.

import pytest

def test_free_provider_never_applies():
    req = ApplyRequest(
        provider_class=ProviderClass.FREE,
        sql="ALTER TABLE orders ADD COLUMN expires_at timestamptz;",
        human_ack=True,
        environment="prod",
        file_bytes=b"ALTER TABLE orders ADD COLUMN expires_at timestamptz;\n",
    )
    with pytest.raises(CustodyError, match="cannot own DDL apply"):
        assert_apply_allowed(req)

def test_prod_destructive_sql_is_refused_even_when_pinned():
    req = ApplyRequest(
        provider_class=ProviderClass.PINNED,
        sql="ALTER TABLE orders DROP COLUMN notes;",
        human_ack=True,
        environment="prod",
        file_bytes=b"ALTER TABLE orders DROP COLUMN notes;\n",
    )
    with pytest.raises(CustodyError, match="online schema changer"):
        assert_apply_allowed(req)

def test_digest_is_stable_for_the_same_bytes():
    payload = b"-- add expires_at\nALTER TABLE orders ADD COLUMN expires_at timestamptz;\n"
    assert migration_digest(payload) == migration_digest(payload)

Commands around the gate should treat the migration file as the artifact and the model as optional commentary. Piping a completion into psql is the anti-pattern. Hashing the file, linting it, and applying only that file is the pattern.

# proposal workflow: the file is the contract, not the chat log
sha256sum migrations/20260917_add_expires_at.sql
sqlfluff lint migrations/20260917_add_expires_at.sql
atlas schema diff --from file://schema.hcl --to postgres://localhost:5432/app?sslmode=disable
psql --set ON_ERROR_STOP=1 -c "SELECT locktype, relation::regclass, mode FROM pg_locks WHERE NOT granted;"
# never: some_chat_cli | psql

Better alternatives already exist and do not need a model in the apply loop. Expand-contract migrations keep old and new columns serving traffic until readers move, then drop the old column in a later, boring change. Online changers such as gh-ost or pt-online-schema-change avoid a long lock on a busy table. Flyway, Liquibase, or Atlas can record the exact bytes applied, which is a ledger a chat transcript cannot replace. If a model is used at all, it drafts a plan against staging data. It does not emit a statement that a bot will run.

Application-level rollouts are often the real need hiding inside a rushed DDL request. A new column that is only read behind a flag does not require a lock-heavy rewrite. A backfill that can run in chunks does not require a single chat-authored UPDATE. Those shapes keep the irreversible work small and scheduled, which is the opposite of a free completion that arrived in 800 milliseconds and looked confident.

Exit criteria should be written before the first incident, not after. Drafting on free inference stops when the change is not additive, when a rewrite needs a table lock, when rollback is not a forward migration, or when the data class is regulated. Planning stops there as well when a well-meaning intern might paste the output into an apply job. The exit is not a sharper prompt. The exit is taking apply credentials off that path and leaving them on the migrator the team already trusts.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option are still reasonable for restating a dry-run log or drafting a rollback checklist in a sandbox, because those jobs tolerate stalls and do not mutate rows. They are the wrong place to keep psql credentials or an apply token. A team that wants a separate machine for that write-up can keep the sandbox on the free server option, then apply from the existing migrator with hashed files.

The gate has limits. It does not detect semantic damage in an additive column that stores the wrong unit. It does not replace expand-contract discipline, lock measurement on a production-sized copy, or a human who understands the table. Platform groups that already run a schema service should not insert a chat provider into that service in order to look current. Teams without migrations as code should not treat a refusal rule as a substitute for review. Disposable local databases are out of scope. The rule is for data that is shared with users.

Keep the scrap paper. Take the stamp away from the free endpoint. Live schema change belongs to a hashed file, a migrator the team can replay, and a person who can still say no after the model has finished talking.

Canary Verdicts Do Not Belong on Free Inference

Casey Li — Wed, 16 Sep 2026 04:36:17 +0000

A canary is a production contract with a short fuse. Free, best-effort inference can narrate the numbers after the fact. It must not be the function that returns promote or rollback while live traffic is already on the new build.

Merge gates fail in a waiting room. Canaries fail in public. That difference is the whole argument. A blocked pull request wastes a morning. A wrong canary call spends user sessions on a bad binary, then spends the on-call's night undoing it. The verdict path has to stay boring: same snapshot, same thresholds, same exit code, every time.

Think of the canary as a fuse, not a suggestion box. A fuse does not ask a language model whether the smell of ozone is “probably fine.” It opens the circuit on a number. Commentary can come later, in a ticket, in a slack thread, in a postmortem draft. Commentary does not hold the wire.

This article is a field guide for keeping that wire out of unpaid inference. The working example is a tiny collector that already exists in most stacks: two JSON snapshots, a handful of SLIs, and a process exit code that a deploy job can trust. The optional model sits downstream of that code, and it cannot change the code's mind.

Why this boundary is not a merge-gate rerun

Pre-merge review asks whether a change is allowed to exist. A canary asks whether a change that already exists should keep eating traffic. Latency, error ratio, and saturation are not prose. They are counters. Once those counters feed a prompt, three failure modes show up that never appear in a linter.

The first is timeout-as-approval. Free endpoints shed load. If the model call is on the hot path, a 429 or a hung stream can be misread as “no signal,” and the pipeline promotes. Silence is not a green check. The second is log-to-prompt injection. Canary windows scrape traces, user agents, and error strings. Those strings are untrusted input. A model that is allowed to vote can be steered by a payload that was supposed to be telemetry. The third is rerun drift. The same snapshot, sent twice, can come back as rollback then hold. A fuse that flickers is not a fuse.

Rate limits during an incident make the problem worse, not better. The hour a service is on fire is the hour a free pool is busiest. Putting the verdict on that pool couples production recovery to someone else's spare capacity. That coupling is the red flag that ends the design review.

A deterministic verdict, then optional prose

The artifact below is intentionally small. It reads two files, compares a fixed set of SLIs, and exits 0 to promote, 2 to rollback, or 3 to hold for a human. No network call lives inside verdict(). A separate helper may draft a paragraph. The helper's return value is logged and discarded.

Label this as a worked example, not a production SRE platform. Swap the JSON files for Prometheus queries in a real job. Do not swap the exit codes for a model score.

# canary_verdict.py
from __future__ import annotations

import argparse
import json
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Literal

Decision = Literal["promote", "rollback", "hold"]

EXIT = {"promote": 0, "rollback": 2, "hold": 3}


@dataclass(frozen=True)
class Window:
    requests: int
    errors: int
    p99_ms: float
    cpu_pct: float

    @property
    def error_ratio(self) -> float:
        if self.requests <= 0:
            return 1.0
        return self.errors / self.requests


@dataclass(frozen=True)
class Policy:
    min_requests: int = 200
    max_error_delta: float = 0.005
    max_p99_ratio: float = 1.25
    max_cpu_pct: float = 85.0


def load_window(path: Path) -> Window:
    raw = json.loads(path.read_text())
    return Window(
        requests=int(raw["requests"]),
        errors=int(raw["errors"]),
        p99_ms=float(raw["p99_ms"]),
        cpu_pct=float(raw["cpu_pct"]),
    )


def verdict(baseline: Window, canary: Window, policy: Policy) -> Decision:
    # Gray zone: not enough traffic to trust the ratio.
    if canary.requests < policy.min_requests:
        return "hold"
    if canary.error_ratio - baseline.error_ratio > policy.max_error_delta:
        return "rollback"
    if baseline.p99_ms > 0 and (canary.p99_ms / baseline.p99_ms) > policy.max_p99_ratio:
        return "rollback"
    if canary.cpu_pct > policy.max_cpu_pct:
        return "rollback"
    return "promote"


def maybe_comment(decision: Decision, baseline: Window, canary: Window) -> str:
    """Side-channel prose. Must never be consulted by verdict()."""
    return (
        f"decision={decision} baseline_err={baseline.error_ratio:.4f} "
        f"canary_err={canary.error_ratio:.4f} p99 {baseline.p99_ms}->{canary.p99_ms}"
    )


def main() -> int:
    parser = argparse.ArgumentParser(description="Deterministic canary verdict")
    parser.add_argument("--baseline", type=Path, required=True)
    parser.add_argument("--canary", type=Path, required=True)
    parser.add_argument("--comment", action="store_true")
    args = parser.parse_args()
    policy = Policy()
    baseline = load_window(args.baseline)
    canary = load_window(args.canary)
    decision = verdict(baseline, canary, policy)
    print(decision)
    if args.comment:
        print(maybe_comment(decision, baseline, canary), file=sys.stderr)
    return EXIT[decision]


if __name__ == "__main__":
    raise SystemExit(main())

A deploy job should treat the process the way it treats grep or jq: read the integer, ignore stderr. The commands below freeze two snapshots and prove the exit code does not depend on a model being reachable.

cat > baseline.json <<'EOF'
{"requests": 8000, "errors": 16, "p99_ms": 120.0, "cpu_pct": 41.0}
EOF

cat > canary.json <<'EOF'
{"requests": 900, "errors": 22, "p99_ms": 210.0, "cpu_pct": 58.0}
EOF

python canary_verdict.py --baseline baseline.json --canary canary.json
echo "exit=$?"

That canary loses on error delta and p99 ratio, so the process exits 2. Flip errors down to 2 and p99_ms down to 110, and the same binary exits 0. Drop requests to 40, and it exits 3. Three numbers. No temperature parameter.

Tests that refuse to call a model

The test file is the other half of the artifact. If a future refactor smuggles a client into verdict(), these cases still fail closed only if the function stays pure. Keep the tests offline. Do not mark them integration and then skip them when a key is missing.

# test_canary_verdict.py
from canary_verdict import Policy, Window, verdict

POLICY = Policy()
BASE = Window(requests=5000, errors=10, p99_ms=100.0, cpu_pct=40.0)


def test_rollback_on_error_spike():
    canary = Window(requests=2000, errors=40, p99_ms=100.0, cpu_pct=40.0)
    assert verdict(BASE, canary, POLICY) == "rollback"


def test_rollback_on_p99_regression():
    canary = Window(requests=2000, errors=4, p99_ms=160.0, cpu_pct=40.0)
    assert verdict(BASE, canary, POLICY) == "rollback"


def test_hold_when_sample_is_tiny():
    canary = Window(requests=20, errors=0, p99_ms=90.0, cpu_pct=30.0)
    assert verdict(BASE, canary, POLICY) == "hold"


def test_promote_when_inside_budget():
    canary = Window(requests=2000, errors=4, p99_ms=110.0, cpu_pct=44.0)
    assert verdict(BASE, canary, POLICY) == "promote"

python -m pytest -q test_canary_verdict.py

A useful CI check is even narrower: rg -n "openai|anthropic|httpx|requests\.post" canary_verdict.py should be empty except in a file that is not imported by the job. The grep is crude. Crude is the point. The canary path should be greppable by someone half awake.

Red flags, in the order they usually arrive

The first red flag is a prompt that includes the words “decide whether to promote.” That sentence is the design defect, even if the rest of the template looks careful. The second is a retry wrapper around the model call inside the same step that writes the verdict. Retries hide brownouts. Brownouts are when a bad canary most needs a hard rollback. The third is using generated prose as the audit log. Auditors need the snapshot and the policy version, not a paragraph that says the spike was “likely a cold start.”

A fourth flag is quieter. Teams let the model propose new thresholds after each incident, then paste those thresholds back into the policy without a review. That loop is how a fuse becomes a mood. Threshold changes belong in the same review path as the service code. They are production config. They are not chat residue.

Gray traffic is not a loophole. If the canary is at 1% and the model is “only advising,” the pipeline will still grow a habit of trusting the advice. Habits become defaults. Defaults become the path that runs at 03:00. Exit the pattern before the percentage climbs.

Where a free model still earns a seat

None of this says models are useless around deploys. They are useful one layer off the fuse. A model can draft a human-readable summary of a verdict that already exists. It can turn rollback plus two JSON files into a ticket body. It can help an engineer iterate on the collector script in a scratch environment, where a wrong answer costs a re-run rather than a slice of production.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option fit that scratch loop: rewrite the parser, generate extra unit cases, argue with a draft of maybe_comment(). They do not fit the process that Kubernetes, Nomad, or a homegrown deployer will exec as the source of truth. Keep the paid, rate-limited, or local deterministic checker in the job. Keep the free pool in the notebook.

A practical split looks like this in a pipeline. Step one writes snapshots. Step two runs canary_verdict.py and captures $?. Step three, if and only if the job is already finishing, posts a comment. If step three times out, the deploy still did the right thing. That is the only coupling that survives a free-tier outage.

Exit criteria for pulling the model out

Pull the model from the canary path the first time any of these is true. A timeout or HTTP 429 would have changed a promote/rollback. Two reruns of the same snapshot disagreed. A trace line showed up inside a prompt and the output mentioned that line as evidence. The on-call could not explain the decision without quoting the model. The policy file and the prompt drifted, and nobody could say which one the job used.

Any single item is enough. Waiting for a user-facing incident is not an exit criterion. It is an after-action report.

Better replacements are already dull, which is their virtue. A sequential test on error counts. A hard SLO burn alert that pages a person in the gray zone. A hold state that refuses to auto-promote when sample size is small. Those tools do not write fluently. They also do not invent a story in which a 2x p99 is “within normal weekend noise.”

Who should not use a model-shaped canary at all

Skip the pattern entirely if the service handles payments, authn, medical, or anything with a legal retention rule on the logs that would be pasted into a prompt. Skip it if the canary runs in an environment that cannot pin a policy version next to the binary. Skip it if the team cannot test the checker offline. Skip it if the only rollback button is the same chat session that produced the go-live note.

Hobby clusters and internal demo apps can still use a model to explain a graph. They should still keep verdict() pure. The cost of copying a bad habit from a demo into a revenue path is how most of these designs actually ship.

Limitations of the artifact are plain. The windows are files, not live queries. The policy is four numbers, not a full multi-window sequential test. There is no multiple-comparison correction, no warmup exclusion, no per-endpoint breakdown. Those gaps are reasons to grow the checker, not reasons to ask a free endpoint to “use its judgment.” Judgment is what the fuse is there to refuse.

The industry conversation this month keeps collapsing two jobs into one: sketching code and owning production risk. Canaries are the second job. Let a free model talk about the snapshot after the circuit has already opened or closed. Do not hand it the handle.

Red Flags Before Routing a Workload Through a Free Tier

Casey Li — Mon, 14 Sep 2026 21:56:53 +0000

Red Flags Before Routing a Workload Through a Free Tier

Free model access and a free server are probe instruments. They answer one question well — does this idea survive contact with real input — and they answer it cheaply, which is exactly why they get promoted into roles they were never shaped for. The usual failure is not a crash. It is a quiet dependency on capacity nobody promised, wrapped around a code path that a team now treats as production.

This field guide is the inverse of an adoption post. It lists the conditions under which a workload should stay off a free tier, the measurements that decide it, and the criteria that force a migration before the decision gets made for you.

Why free capacity leaks into critical paths

The leak happens for a boring reason. A prototype on free capacity behaves like the real thing during a demo: it returns tokens, it streams, it looks fine. Nothing in the response tells the caller that the model version may move underneath it, that capacity is shared with strangers, or that consumption is drawn from a pool with no contract attached. Only failure surfaces separate the two, and failure surfaces appear after the prototype already has callers.

The distinction worth keeping in the head is accountability. A component is load-bearing when someone can be held to its behavior — a pager, a stored record, an invoice, an audit line. Anything load-bearing belongs somewhere with an owner, a version, and a stated limit.

Four red flags, ordered by how often they bite

The output outlives the request. If a response lands in a database, an object store, a customer inbox, or a compliance archive, the tier that produced it has joined a provenance chain. Free tiers generally do not commit to traceability, and the same prompt replayed next month may not produce the same text.

The caller retries on its own. Retries turn a transient refusal into multiplied consumption, and multiplied consumption into an argument with a ceiling you did not set. Free capacity is most fragile precisely when traffic is spiky, which is when automatic retries fire hardest.

The workload carries secrets. Prompts assembled from internal documents, stack traces, or customer records push sensitive data across a boundary the operator does not control. That is a legal and architectural problem, not a throughput problem.

A human latency promise sits on top. Shared capacity gives a fine median and a jittery tail. The median is what the demo shows; the tail is what the user experiences at 4pm.

Any single flag is survivable with eyes open. Two flags on the same call path is the point where the migration is already late.

A probe harness you can run before committing

The decision should come from numbers, so record them from the first call. The harness below uses only the standard library, runs without a network key in --dry-run mode, and prints a verdict against a policy file. Treat it as a template: swap the synthetic generator for a thin client that wraps your provider call. No performance figures for any provider are claimed here; the dry-run data is deliberately synthetic.

#!/usr/bin/env python3
"""tier_probe.py - decide whether a workload belongs on a free tier.

  python tier_probe.py --dry-run --calls 40 --policy policy.json
  echo $?   # 1 means at least one exit criterion was breached
"""
import argparse, json, random, statistics
from dataclasses import dataclass


@dataclass
class Attempt:
    call_id: int
    attempt: int      # 1 = first try, >1 = retry
    status: str       # ok | rate_limited | timeout | error
    tokens_in: int
    tokens_out: int
    latency_ms: int


def dry_run(calls: int, seed: int = 7):
    rng = random.Random(seed)
    rows = []
    for cid in range(calls):
        attempt = 1
        while True:
            roll = rng.random()
            status = "ok" if roll > 0.18 else rng.choice(
                ["rate_limited", "timeout", "error"])
            rows.append(Attempt(cid, attempt, status,
                                rng.randint(300, 1500), rng.randint(50, 600),
                                int(rng.gauss(900, 250))))
            if status == "ok" or attempt >= 3:
                break
            attempt += 1
    return rows


def measure(rows):
    per_call = {}
    for r in rows:
        per_call.setdefault(r.call_id, []).append(r)
    total = sum(r.tokens_in + r.tokens_out for r in rows)
    wasted = sum(r.tokens_in + r.tokens_out for r in rows if r.attempt > 1)
    ok = [r.latency_ms for r in rows if r.status == "ok"]
    failures = sum(1 for r in rows if r.status != "ok")
    return {
        "calls": len(per_call),
        "attempts_per_call": round(len(rows) / max(len(per_call), 1), 2),
        "failure_rate": round(failures / max(len(rows), 1), 3),
        "retry_token_share": round(wasted / max(total, 1), 3),
        "calls_never_succeeded": sum(
            1 for rs in per_call.values() if all(r.status != "ok" for r in rs)),
        "median_ok_latency_ms": int(statistics.median(ok)) if ok else 0,
    }


def breaches(m, policy):
    return {k: (m[k], limit) for k, limit in policy.items() if m[k] > limit}


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--calls", type=int, default=40)
    ap.add_argument("--policy", default="policy.json")
    ap.add_argument("--dry-run", action="store_true")
    a = ap.parse_args()
    if not a.dry_run:
        raise SystemExit("wire your own client, then drop --dry-run")
    m = measure(dry_run(a.calls))
    with open(a.policy) as fh:
        policy = json.load(fh)
    hit = breaches(m, policy)
    print(json.dumps(m, indent=2))
    print("breaches:", hit or "none")
    raise SystemExit(1 if hit else 0)

A policy file turns engineering taste into a gate. The defaults below are proposed starting points for a backlogged team, not measured properties of any service, and they should be tightened for user-facing paths and loosened for overnight batch work.

{
  "failure_rate": 0.10,
  "retry_token_share": 0.20,
  "attempts_per_call": 2.00,
  "calls_never_succeeded": 0
}

Run it on day one, save the JSON next to the prototype, and re-run it whenever the workload changes shape. retry_token_share is the metric most teams never look at, and it is the one that pre-announces a wall: above roughly a fifth of consumption, the workload is not being served, it is gambling.

Routing by workload shape

Once the numbers are in hand, routing becomes a lookup rather than a debate.

Workload shape	Route	Reason
Scratch classification, output read once	free tier	the loss on failure is a discarded row
Demo, workshop, throwaway preview	free tier plus free server	reset is the feature, not a gap
Batch enrichment into a durable store	paid or self-hosted	provenance and version pinning
Anything inside a CI gate	deterministic check, or pinned paid model	a flaky gate teaches people to ignore it
Long-lived agent with memory	self-hosted with checkpointing	state must survive a capacity change
Regulated or secret-bearing input	self-hosted or contracted vendor	boundary control

Where MonkeyCode's free access fits

The operator of MonkeyCode states that the project provides free model access, described as up to ten million tokens, along with a free server option. Those figures are operator-supplied; I have not independently verified them, and this article claims no benchmark, uptime, quota-per-window, hardware detail, or permanence for either offer.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What that shape of offer is genuinely good for is the first two rows of the table above: probe work, throwaway demos, teaching material, and experiments whose output nobody will be held to. The free server option is a reasonable home for a preview environment you are happy to rebuild from scratch, since the interesting property of a sandbox is that deleting it costs nothing. The same reasoning makes it a poor home for the bottom four rows: the moment output persists, feeds a gate, or carries data with a boundary around it, an owner, a version, and a contract stop being optional.

If the experiment in mind is discardable by design, the free access and free server are worth an afternoon — but run the probe first, so the decision rests on a number rather than a feeling.

Exit criteria and who should not use this

The exit plan should exist before the first call. Keep exactly one seam in the code:

import os

ROUTE = os.environ.get("INFERENCE_ROUTE", "free")
ENDPOINTS = {
    "free":  (FREE_BASE_URL, FREE_MODEL_ALIAS),
    "paid":  (PAID_BASE_URL, PINNED_MODEL_VERSION),
    "local": ("http://127.0.0.1:8080/v1", LOCAL_ALIAS),
}
base_url, model = ENDPOINTS[ROUTE]

With that seam in place, migration is a config change rather than a refactor, and a reviewer can see which environment is hitting which endpoint. The trigger to flip it is the policy file above, not a hunch: two breached criteria on a user-facing path, one breach plus a red flag, or any change that makes the output auditable.

Three groups should skip the free tier entirely for their main path. Teams handling regulated data, because the boundary question has one acceptable answer. Teams that owe an uptime commitment to anyone, because shared capacity cannot be paged. And anyone whose artifact must be reproducible in ninety days, because a moving alias makes that promise unkeepable.

Nobody has to be talked out of using free capacity. They have to be given the moment to leave it, and that moment is cheaper to define with a probe script than to discover during an incident.

When Free Inference Should Not Own the Retry Loop

Casey Li — Sun, 13 Sep 2026 20:45:25 +0000

A retry loop that consults free inference before it fires again is not resilience. It is a second outage, delayed by a model call that can stall, drift, or disagree with itself. Status codes, idempotency keys, and a hard budget already decide whether work is safe to repeat. A model does not improve that decision when the model is the least reliable hop on the path.

This is not an argument against using a language model to study failures after the fact. It is a boundary. Live retries are a control-plane act: they duplicate side effects, burn downstream quotas, and stretch user-visible latency. Free inference is a best-effort text generator. Those two jobs do not share a service-level objective, and pretending they do is how a brief 503 becomes a thundering herd with a narrative attached.

Timeouts, 429s, and connection resets already carry well-understood codes. Mapping those codes through a completion endpoint adds tail latency, non-determinism, and a dependency that fails in the same class of ways as the original call. The loop then retries the work and retries the advisor. Two clocks start, and only one of them belongs to the customer request.

The analogy is a smoke alarm that phones a remote panel to ask whether the smoke is interesting. The building does not get safer while the panel thinks. It gets more smoke. Retry logic that waits on a completion is the same delay, except the panel can also change its mind between attempts.

Red flags show up as design choices, not as slogans. The first is a loop whose break condition is a sentence. If the path reads a model reply and then selects retry, give_up, or page_oncall, the classifier has become the circuit breaker. Model output is not breaker state. Breaker state is a counter, a clock, and a policy a human can read in a diff.

The second flag is a retry allowed to touch anything that is not idempotent. Charging a card, sending mail, rotating a secret, or applying a migration cannot wait on a paraphrase of the error body. Free inference will, on some days, describe a unique-constraint violation as a transient blip. That description is not a permission slip.

The third flag is coupling the retry budget to the inference budget. When the model is slow, the loop waits. When the model is rate-limited, the loop guesses. When a free endpoint swaps a backend, the loop's so-called intelligence silently changes shape. None of those events are properties of the original failure. They are properties of a volunteer dependency.

The fourth flag is using the model to invent severity. A 429 is a 429. A checksum mismatch is not probably fine if retried from another region. Teams that let free inference rewrite severity discover that pages arrive late, or not at all, because a completion once chose a calmer adjective.

Better alternatives are boring on purpose. Classify the error with the transport: HTTP status, gRPC code, SQLSTATE, syscall errno. Map that class onto a tiny policy object a test can exhaust. Cap attempts. Cap wall-clock time. Jitter the sleep. Honor Retry-After when the server sends it. Treat unknown codes as terminal unless a written policy says otherwise. Persist an idempotency key before the first attempt so a replay cannot double-apply.

The following example is a proposal, not a measured production benchmark. It keeps inference off the hot path. The loop never constructs a prompt.

# retry_policy.py
from __future__ import annotations

from dataclasses import dataclass
from enum import Enum
import random
import time
from typing import Callable, TypeVar

T = TypeVar("T")


class ErrorClass(Enum):
    TRANSIENT = "transient"
    RATE_LIMIT = "rate_limit"
    CONFLICT = "conflict"
    TERMINAL = "terminal"


@dataclass(frozen=True)
class RetryPolicy:
    max_attempts: int = 4
    max_sleep_s: float = 8.0
    base_sleep_s: float = 0.25
    deadline_s: float = 12.0


@dataclass
class AttemptRecord:
    attempt: int
    error_class: str | None
    slept_s: float
    elapsed_s: float
    outcome: str


def classify_status(status: int) -> ErrorClass:
    if status in {408, 502, 503, 504}:
        return ErrorClass.TRANSIENT
    if status == 429:
        return ErrorClass.RATE_LIMIT
    if status in {409, 412}:
        return ErrorClass.CONFLICT
    return ErrorClass.TERMINAL


def next_sleep(policy: RetryPolicy, attempt: int, retry_after: float | None) -> float:
    if retry_after is not None:
        return min(retry_after, policy.max_sleep_s)
    exp = policy.base_sleep_s * (2 ** max(0, attempt - 1))
    jitter = random.random() * policy.base_sleep_s
    return min(exp + jitter, policy.max_sleep_s)


def run_with_budget(
    op: Callable[[], tuple[int, T]],
    policy: RetryPolicy,
    clock: Callable[[], float] = time.monotonic,
    sleeper: Callable[[float], None] = time.sleep,
) -> tuple[T, list[AttemptRecord]]:
    started = clock()
    records: list[AttemptRecord] = []
    last_error: str | None = None

    for attempt in range(1, policy.max_attempts + 1):
        elapsed = clock() - started
        if elapsed >= policy.deadline_s:
            records.append(AttemptRecord(attempt, last_error, 0.0, elapsed, "deadline"))
            raise TimeoutError("retry deadline exceeded")

        status, value = op()
        kind = classify_status(status)
        elapsed = clock() - started
        if status < 400:
            records.append(AttemptRecord(attempt, None, 0.0, elapsed, "ok"))
            return value, records
        if kind is ErrorClass.TERMINAL or kind is ErrorClass.CONFLICT:
            records.append(AttemptRecord(attempt, kind.value, 0.0, elapsed, "stop"))
            if kind is ErrorClass.CONFLICT:
                raise RuntimeError("conflict is not retryable without a new idempotency key")
            raise RuntimeError(f"terminal status {status}")

        sleep_for = next_sleep(policy, attempt, retry_after=None)
        remaining = policy.deadline_s - (clock() - started)
        sleep_for = max(0.0, min(sleep_for, remaining))
        records.append(AttemptRecord(attempt, kind.value, sleep_for, elapsed, "retry"))
        last_error = kind.value
        if attempt == policy.max_attempts:
            break
        sleeper(sleep_for)

    raise TimeoutError("retry attempts exhausted")

A test can freeze time and the dice. That is the point. Retry policy that depends on a model cannot freeze the model's vocabulary, and a suite that cannot freeze its oracle is not a suite.

# test_retry_policy.py
from retry_policy import RetryPolicy, run_with_budget


def test_does_not_retry_conflict():
    calls = {"n": 0}

    def op():
        calls["n"] += 1
        return 409, None

    try:
        run_with_budget(op, RetryPolicy(), sleeper=lambda _s: None)
        raise AssertionError("expected conflict to stop the loop")
    except RuntimeError as exc:
        assert "conflict" in str(exc)
    assert calls["n"] == 1


def test_transient_respects_max_attempts():
    calls = {"n": 0}

    def op():
        calls["n"] += 1
        return 503, None

    sleeps: list[float] = []
    try:
        run_with_budget(
            op,
            RetryPolicy(max_attempts=3, deadline_s=60),
            sleeper=lambda s: sleeps.append(s),
        )
        raise AssertionError("expected exhaustion")
    except TimeoutError:
        pass
    assert calls["n"] == 3
    assert len(sleeps) == 2

Run the example locally with a clock the suite controls:

python -m pytest test_retry_policy.py -q

Offline analysis is a different job. After the loop stops, a team may dump the AttemptRecord list, the raw response body, and the idempotency key into JSONL and ask a model to summarize clusters for humans. That summary must not write back into the policy. It may open a ticket. It may suggest a new status mapping. A person still merges the mapping.

The snippet below is a sketch, not a networked client. It reads JSONL and prints a prompt. It does not call anything during the retry.

# offline_cluster.py
import json
import sys


def build_prompt(path: str) -> str:
    rows = [json.loads(line) for line in open(path, encoding="utf-8")]
    compact = [
        {
            "error_class": r.get("error_class"),
            "outcome": r.get("outcome"),
            "status_hint": r.get("status_hint"),
        }
        for r in rows[:200]
    ]
    return (
        "Cluster these retry records by error_class and outcome. "
        "Do not propose live retry rules. "
        "List candidate status mappings for a human to review.\n\n"
        + json.dumps(compact, indent=2)
    )


if __name__ == "__main__":
    sys.stdout.write(build_prompt(sys.argv[1]))

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, which is a reasonable desk for iterating on that offline prompt and the JSONL shape. It is not a component of the live retry path. A free completion that is useful at a desk is still the wrong process to block a payment capture or a deploy rollback.

A compact field guide fits in a table because the rule is mechanical.

Signal on the live path	Treat as	Do not treat as
408 / 502 / 503 / 504	transient, budgeted retry	a prompt about whether the API is sad
429 plus Retry-After	sleep bounded by policy	a model-chosen backoff story
409 / 412	stop, new key required	try a slightly different payload
4xx validation	terminal	a rewrite of the request body
timeout of the model hop	proof the hop should not be there	extra retries of the original work

Exit criteria are equally concrete. Leave this design when the failure is not a transport class. If the only way to know whether a document is complete is to read its language, the problem is not a retry loop. It is a parser, a schema, or a human queue. Move that work off the request path, version the classifier, give it an SLO, and still refuse to let it replay side effects. Leave this design when a regulator requires a logged, deterministic reason for every replay; a free model cannot sign that reason. Leave it when p95 of the downstream already sits near the user timeout, because even a fast completion steals budget that jitter needs.

Limitations follow from the same boundary. Deterministic classification misses semantic failures: a 200 that contains an error string, a partial write that returned 201, a vendor that uses HTTP 500 for user-not-found. Those bugs need contract tests and vendor-specific adapters, not a model stuffed into except Exception. The example policy also assumes the caller can supply an idempotency key and that op() returns a single status-plus-value pair. Streaming RPCs, multi-step sagas, and exactly-once queues need their own ledgers.

Who should not take this as a blanket ban: researchers comparing retry policies in a simulator, and products whose whole job is semantic recovery on sandbox data with no customer side effects. They can put a model in the loop because a wrong retry there does not duplicate a charge. Production traffic is not that sandbox.

The practical workflow stays split. The live loop remains stupid, tested, and budgeted. The model reads a file after the fire is out. The retry budget still ships as code.

When Free Inference Should Not Back an MCP Tool

Casey Li — Sat, 12 Sep 2026 18:45:57 +0000

Free inference is a scratch pad. It is not a backend for an MCP tool that other agents, editors, or CI jobs will call by name. The instant a team publishes a tool over the Model Context Protocol, callers freeze the name, the argument schema, and the implied side effects into their own loops. A free backend that can stall, drop context, or change shape without a ticket then stops being a convenience and starts being a shared outage.

MCP makes that freeze easy to miss. A local function wrapped as a tool looks like a polite helper in a sidebar. After a second client generates a stub against the schema, the helper is an unofficial API. Free inference behind that API inherits none of the properties those clients quietly assume: bounded latency, a stable output shape, a durable session, or an operator who can be paged.

Think of MCP as a hallway directory. The directory does not merely suggest a room. It tells every visitor which door to knock on, and visitors start leaving notes on that door. If the room is a borrowed lounge that closes without warning, the notes become a pile of incident tickets. The directory was honest. The room was never owned.

That failure mode is not the same as hosting an agent on free inference, and it is not the same as letting an agent hold tools. Here the free model is the tool. Other programs treat its JSON as ground truth. That reversal is the red flag that should stop registration before a socket opens.

Writing is the first eject condition. A tool that creates branches, opens issues, patches files, or talks to a webhook is exercising authority. Authority needs an owner and a rollback path. Free inference supplies neither. A hallway that offers a “fix it” button will be pressed by loops that do not read the fine print.

Persistence is the second. If the returned text will be committed, cached, embedded, or used as an identity, the backend has become a source of record. Free inference is a poor source of record. Cache keys derived from model prose will thrash. Identifiers derived from model prose will collide. Release notes derived from model prose will leak into customer channels with no one sure who approved the wording.

Hot paths are the third. A tool invoked on every keystroke, every pull request, or every deploy turns tail latency into a build outage. Callers then add retries. Retries amplify load on a free backend and turn a slow answer into a stampedes of duplicate side effects. The directory still points at the same door. The lounge is now full of the same visitor knocking twice.

Trust boundaries are the fourth. A tool shared with another team, another tenant, or an unattended bot is no longer a scratch pad. Arguments start to contain repository fragments, log lines, and the occasional secret that someone pasted “just this once.” Cross-session confusion is not theoretical once two clients share a name. The safe assumption is that a free backend does not isolate callers the way a product contract would.

Schema freeze is the fifth. Yesterday’s tool returned a string. Today’s free model returns a prose apology wrapped in Markdown. Typed clients break. Onboarding docs lie. Compatibility is not a styling issue. Free inference has no built-in compatibility window, so a published schema is a promise the backend cannot keep.

The artifact below is a registration gate, not a server. It scores a proposed tool against those conditions and refuses to bind a listener when the score says eject. The code is a labeled proposal. It is meant to be read and tested locally, not dropped onto a public port.

# tool_proposal.yaml — declarations the gate trusts more than model prose
name: repo_summarize
mutates: false
output_is_persisted: true
hot_path: false
crosses_trust_boundary: true
callers_auto_retry: true
schema_in_docs: true
arguments_may_contain_secrets: true
binds_shared_socket: true
uses_free_inference: true

# proposal: mcp_free_gate.py — refuse MCP registration, do not implement a server
from __future__ import annotations

from dataclasses import dataclass
from enum import Enum
from pathlib import Path
from typing import List
import json
import sys


class Verdict(str, Enum):
    PASS = "pass"
    EJECT = "eject"


@dataclass(frozen=True)
class ToolProposal:
    name: str
    mutates: bool
    output_is_persisted: bool
    hot_path: bool
    crosses_trust_boundary: bool
    callers_auto_retry: bool
    schema_in_docs: bool
    arguments_may_contain_secrets: bool
    binds_shared_socket: bool
    uses_free_inference: bool

    @classmethod
    def from_mapping(cls, data: dict) -> "ToolProposal":
        required = cls.__dataclass_fields__.keys()
        missing = [k for k in required if k not in data]
        if missing:
            raise ValueError(f"proposal missing fields: {missing}")
        return cls(**{k: data[k] for k in required})


RED_FLAGS = (
    "mutates",
    "output_is_persisted",
    "hot_path",
    "crosses_trust_boundary",
    "callers_auto_retry",
    "schema_in_docs",
    "arguments_may_contain_secrets",
    "binds_shared_socket",
)


def score(proposal: ToolProposal) -> tuple[Verdict, List[str]]:
    reasons: List[str] = []
    if not proposal.uses_free_inference:
        return Verdict.PASS, ["pinned or local backend; gate does not apply"]

    for flag in RED_FLAGS:
        if getattr(proposal, flag):
            reasons.append(flag)

    if reasons:
        return Verdict.EJECT, reasons
    return Verdict.PASS, ["free inference stays private; no shared MCP surface"]


def refuse_bind(proposal: ToolProposal) -> None:
    verdict, reasons = score(proposal)
    report = {"tool": proposal.name, "verdict": verdict.value, "reasons": reasons}
    print(json.dumps(report, indent=2))
    if verdict is Verdict.EJECT:
        raise SystemExit(
            f"refusing to register {proposal.name!r} as an MCP tool on free inference"
        )


def main(argv: List[str]) -> None:
    if len(argv) != 2:
        raise SystemExit("usage: python mcp_free_gate.py path/to/tool_proposal.yaml")
    raw = Path(argv[1]).read_text(encoding="utf-8")
    try:
        import yaml  # type: ignore
        data = yaml.safe_load(raw)
    except ImportError:
        data = json.loads(raw)
    refuse_bind(ToolProposal.from_mapping(data))


if __name__ == "__main__":
    main(sys.argv)

A dry run should fail closed on the sample proposal. The command below must print eject and a non-zero status. Anything else means the gate was edited into a courtesy warning, which callers will ignore.

python mcp_free_gate.py tool_proposal.yaml; echo "exit: $?"
# expected: JSON with verdict eject, then a non-zero exit

# proposal: test_mcp_free_gate.py — unexecuted until the operator runs pytest
from mcp_free_gate import ToolProposal, Verdict, score


def _base(**overrides) -> ToolProposal:
    payload = dict(
        name="scratch_echo",
        mutates=False,
        output_is_persisted=False,
        hot_path=False,
        crosses_trust_boundary=False,
        callers_auto_retry=False,
        schema_in_docs=False,
        arguments_may_contain_secrets=False,
        binds_shared_socket=False,
        uses_free_inference=True,
    )
    payload.update(overrides)
    return ToolProposal(**payload)


def test_private_scratch_may_pass():
    verdict, reasons = score(_base())
    assert verdict is Verdict.PASS
    assert "shared MCP surface" in reasons[0]


def test_any_red_flag_ejects_free_backend():
    verdict, reasons = score(_base(binds_shared_socket=True, schema_in_docs=True))
    assert verdict is Verdict.EJECT
    assert "binds_shared_socket" in reasons
    assert "schema_in_docs" in reasons


def test_pinned_backend_skips_gate():
    verdict, _ = score(_base(uses_free_inference=False, mutates=True))
    assert verdict is Verdict.PASS

Better alternatives follow the grain of any other interface. Deterministic local functions belong on MCP: path existence, grep over a working tree, a calculator, a linter already installed on the machine. A pinned, billed endpoint with an owner belongs on MCP if the tool must generate language. A human approval step belongs in front of anything that mutates. Free inference belongs in a private scratch session that never receives a public tool name, never binds to a shared port, and never accepts arguments from another process.

Binding advice is similarly dull, and that is the point. Local rehearsal, if it happens at all, should listen on a loopback address and die when the terminal closes. Advertising 0.0.0.0 or a container port “so the team can try it” is how a scratch pad becomes a service. The gate treats binds_shared_socket as an eject on purpose.

# refuse the shared-socket habit; loopback only, and only after score() returns pass
# nc -l 127.0.0.1 8765
# do not: python -m http.server 0.0.0.0:8765

Exit criteria should be mechanical so a tired on-call does not have to invent policy at 2 a.m. Unregister on the first page about latency or content. Unregister when a caller commits model output. Unregister when two clients disagree about the schema. Unregister when the tool name appears in a runbook or an onboarding doc. Unregister when a secret, a production URL, or a customer identifier shows up in a trace. After eject, leave the name retired. Reusing a burned tool name teaches callers that the hallway directory lies.

Who should ignore this gate? Teams that already treat MCP tools as versioned APIs with pinned inference and an on-call owner do not need a free-backend veto; they need change control. Researchers probing a single local client, with no shared socket and no secrets in the prompt, can skip the ceremony. People who want a creative partner in an editor sidebar are not the audience. The audience is the engineer about to add a tools/list entry that some other loop will trust.

Limitations are blunt. The gate reads declarations, not runtime behavior. A tool marked read-only can still wrap a prompt that asks a model to summarize a private repository. A classifier that used free inference to score the proposal would reintroduce the same drift the constants were written to avoid, which is why the red flags are local booleans. MCP client behavior varies. This article does not claim a particular SDK, quota, hardware shape, or uptime story.

A team can still use free inference to rehearse the scoring step, as long as rehearsal never becomes registration. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are enough to drive a throwaway client that prints PASS or EJECT from the proposal file, provided that client never registers the scored tool on an interface other processes can reach.

The directory should only point at rooms the team actually owns. Free inference can help draft the sign. It should not live behind the door.

Free Inference Should Not Gate a Merge

Casey Li — Fri, 11 Sep 2026 16:19:22 +0000

A free-tier completion can annotate a pull request. It cannot be the required status that lets that pull request land. The moment an unpaid, uncontracted model holds a merge vote, the CI graph stops being a control plane and starts being a suggestion box with a green checkmark glued on.

This is not an argument about model quality. Quality varies by day, by queue, and by the hidden routing behind a free pool. The failure is architectural. A merge gate needs a stable principal, a fail-closed policy, and an audit trail that still makes sense after the endpoint disappears. Free inference offers none of those as a guarantee.

Think of the required check as a deadbolt. Chat is a hallway conversation. Mixing them is how teams discover, after the fact, that a lockfile change shipped because a model typed approval into a comment. The analogy is blunt because the failure mode is blunt. Git hosts already know how to distinguish a non-blocking comment from a required context. Policy should use that distinction.

A free pool is a crowd, not a reviewer. Completions may be served by different underlying systems, different safety filters, and different rate limits. The same prompt is not the same signer. Status checks, by contrast, are names in a protection rule. If the name is llm-review and the body is a free completion, the protection rule has been bound to weather.

The pull request itself is hostile input. Titles, descriptions, commit messages, and diffs are attacker-controlled documents. Prompt injection in a PR body is not exotic. It is the default shape of the artifact under review. A model asked to approve if the tests look fine will often read the story in the description instead of the hunk that swaps a dependency. That is not a reason to ban drafts. It is a reason to keep drafts off the deadbolt.

Latency makes the problem operational, not only security-shaped. Merge queues punish tail latency. Free endpoints throttle, stall, and return empty. A job that retries until the model emits approval is no longer reviewing. It is sampling. Sampling does not belong in a required check.

The first red flag is a required context whose only implementation is an HTTP call to a free inference URL. If that URL returns 429, the branch cannot merge, or worse, the workflow treats timeout as success. Either interpretation is a policy bug. Timeouts on an unpaid pool are normal weather. They are not a signal about the code.

The second red flag is a bot account that can dismiss reviews or write success to a required check from a free-model JSON blob. Dismissal is a privileged act. It should not be a side effect of a completion. If the workflow can both comment and approve, split those permissions. The comment token should not be the approval token.

The third red flag is using the model output as the audit log. The model said it was fine is not evidence. Evidence is a test report, a signed provenance document, a human review, or a scanner with a pinned version. When the free endpoint changes behavior, yesterday's approval cannot be replayed. An auditor will ask who signed. The answer cannot be whichever replica answered.

The fourth red flag is tool access from the review job: git push, gh pr merge, package publish, or cloud credentials. A draft generator that can mutate the repo is not a draft generator. Earlier articles on this DEV account treated tool access and persistence as separate boundaries. The merge gate is another boundary. It is not a place to collapse them.

Exit criteria follow from those flags. Remove free inference from the merge path when the job is required to merge, when it can write a success status, when it can dismiss a review, when the completion is stored as the official rationale, or when the job can call a tool that mutates shared state. If even one item is true, the model is holding a key. Take the key back.

The artifact below is a small policy module plus tests. It does not call a model. It decides whether a CI job is allowed to sign a required status, given how the completion was produced. Treat it as executable policy, not as a benchmark of any vendor. The tests are a local harness. They are not reported results from a production org.

# merge_gate_policy.py
from __future__ import annotations

import json
import os
import sys
from dataclasses import dataclass

ALLOWED_REQUIRED = {"tests", "license-scan", "typecheck", "human-review"}
BLOCKED_PROVIDERS = {"free-inference", "unmetered-pool", "anonymous"}
MUTATING_TOOLS = {"git_push", "gh_merge", "publish", "apply_tf"}


@dataclass(frozen=True)
class ReviewJob:
    context: str
    required: bool
    provider: str
    can_write_status: bool
    can_dismiss_review: bool
    tools: tuple[str, ...]


def may_sign_status(job: ReviewJob) -> tuple[bool, str]:
    if job.required and job.provider in BLOCKED_PROVIDERS:
        return False, "free inference cannot be a required context"
    if job.can_write_status and job.provider in BLOCKED_PROVIDERS:
        return False, "free inference cannot write check status"
    if job.can_dismiss_review:
        return False, "review dismissal is not a model action"
    if MUTATING_TOOLS.intersection(job.tools):
        return False, "mutating tools are outside the draft path"
    if job.required and job.context not in ALLOWED_REQUIRED:
        return False, f"unknown required context: {job.context}"
    return True, "ok"


def sidecar_comment_allowed(job: ReviewJob) -> bool:
    if job.required or job.can_write_status or job.can_dismiss_review:
        return False
    return job.tools == ("post_comment",)


def job_from_env() -> ReviewJob | None:
    raw = os.environ.get("REVIEW_JOB_JSON")
    if not raw:
        return None
    data = json.loads(raw)
    return ReviewJob(
        context=data["context"],
        required=bool(data["required"]),
        provider=data["provider"],
        can_write_status=bool(data["can_write_status"]),
        can_dismiss_review=bool(data["can_dismiss_review"]),
        tools=tuple(data.get("tools", ())),
    )


if __name__ == "__main__":
    job = job_from_env()
    if job is None:
        sys.exit(0)
    ok, reason = may_sign_status(job)
    print(reason)
    sys.exit(0 if ok else 1)

The function returns a reason string so CI logs stay readable. A boolean without a reason becomes another silent green check. Tests pin the intended failures.

# test_merge_gate_policy.py
from merge_gate_policy import ReviewJob, may_sign_status, sidecar_comment_allowed


def test_free_model_cannot_be_required():
    job = ReviewJob(
        context="llm-review",
        required=True,
        provider="free-inference",
        can_write_status=True,
        can_dismiss_review=False,
        tools=("post_comment",),
    )
    ok, reason = may_sign_status(job)
    assert ok is False
    assert "required context" in reason


def test_sidecar_comment_on_free_pool_is_allowed():
    job = ReviewJob(
        context="llm-draft-notes",
        required=False,
        provider="free-inference",
        can_write_status=False,
        can_dismiss_review=False,
        tools=("post_comment",),
    )
    assert sidecar_comment_allowed(job) is True
    ok, reason = may_sign_status(job)
    assert ok is True
    assert reason == "ok"


def test_mutating_tool_is_rejected_even_if_optional():
    job = ReviewJob(
        context="llm-draft-notes",
        required=False,
        provider="free-inference",
        can_write_status=False,
        can_dismiss_review=False,
        tools=("post_comment", "gh_merge"),
    )
    ok, reason = may_sign_status(job)
    assert ok is False
    assert "mutating" in reason


def test_unknown_required_context_fails_closed():
    job = ReviewJob(
        context="model-lgtm",
        required=True,
        provider="paid-contracted",
        can_write_status=True,
        can_dismiss_review=False,
        tools=(),
    )
    ok, reason = may_sign_status(job)
    assert ok is False
    assert "unknown required context" in reason

Run the tests with a pinned interpreter. The command is ordinary on purpose. Policy should not depend on a model's availability.

python -m pip install pytest
python -m pytest -q test_merge_gate_policy.py

A workflow file should enforce the same split at the permission layer. The draft job gets pull-requests: write and nothing else. The required jobs never receive a model provider. The environment blob is assembled from branch protection inventory, not from the completion text.

# .github/workflows/review-sidecar.yml
name: llm-draft-notes
on:
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  draft-notes:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
      checks: read
    env:
      REVIEW_JOB_JSON: >-
        {"context":"llm-draft-notes","required":false,
         "provider":"free-inference","can_write_status":false,
         "can_dismiss_review":false,"tools":["post_comment"]}
    steps:
      - uses: actions/checkout@v4
      - name: fail closed if this job could sign
        run: python merge_gate_policy.py

Wire the required field from the repository ruleset, not from the model. The model does not get to declare itself optional. Humans do that in the branch rules. A merge queue that cannot see this job as a required context is the correct outcome, even if the comment is insightful.

Static checks already cover a large fraction of what teams hope a model will notice. Compilers, linters, license scanners, secret detectors, and the test suite are signers with pinned versions. They belong on the deadbolt. A language model does not replace them, paid or free.

Human review remains the correct signer for intent. Intent lives in product language, threat models, and the reason a dependency changed. No completion, free or otherwise, should be the only reader of that layer on a production repo. A contracted inference endpoint can be a second, non-required opinion if the team wants one. Contract here means an identifiable vendor, an account, a bill, and a place to file an incident. That is a different principal from a free pool. It is still not a substitute for tests.

Draft generation can stay on a free path. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are a reasonable place to render those non-blocking notes, because the job described above only posts a comment. The policy module does not import that product. Delete the name and the gate still fails closed.

Hobby branches with no users, no secrets, and no required checks are not merge gates. A personal experiment that auto-merges into a private sandbox is a different shape of risk. This guide would only add friction there. Teams that have already banned generated text from the repository can skip the sidecar as well.

The policy is for groups that want the draft and need a hard line in front of main. It is not a claim that free inference is useless. It is a claim that uselessness and danger are different, and the merge path is where they get confused. Do not treat the tests above as a measurement of model accuracy. They never call a model. They encode an admission-control decision. If a team needs accuracy numbers, that is a separate eval harness with pinned prompts, pinned dates, and a dataset that is not the open PR queue.

A loop that asks a free model whether the loop should continue is the same bug in a different jacket. Recent discussion of agent loops keeps rediscovering that the useful part is still a conditional. The merge gate is already that conditional. Adding a free completion does not give it judgment. It gives the conditional a noisy coin and a tempting green icon.

The closing move is administrative, not rhetorical. Inventory every required context on the default branch. If any context is implemented by a free inference call, demote it to a comment job the same day. Keep the free server for drafts that cannot open the lock. Readers who want that scratch path can try MonkeyCode's free model access and free server on a workflow that is forbidden from signing status.

Free Inference Should Not Grant Tool Access

Casey Li — Wed, 09 Sep 2026 16:34:17 +0000

Free-tier inference can draft a plan, a schema comment, even a chain of thought that looks like judgment. It cannot be the component that decides whether a tool is allowed to run. The moment a model reply is parsed into an allow-or-deny bit in front of a shell, a filesystem, a payment call, or an MCP-style tool, the hop has left the sandbox and entered a permission layer the free tier does not own.

That distinction is easy to blur because the two hops share a socket. A drafting call returns English. A permission call returns English too. The difference is not the syntax of the prompt. The difference is whether some other process treats those tokens as a key.

Think of a hotel front desk. A clerk who prints a map of the building is useful. A clerk who programs the elevator so the penthouse button lights up is doing a different job, with a different failure mode. Free inference is the map printer. Tool dispatch is the elevator controller. Mixing them is how a timeout, a recycled session, or a politely worded jailbreak becomes an unlock.

The rest of this field guide is about catching that mix before it ships. The artifact is a local checker. It does not call a model. It inspects a hop description and refuses configurations where free inference sits on the grant path.

What a grant path actually is

A grant path is any edge where model output is interpreted as authorization rather than as text. The classic shape is a router: classify intent, then invoke a tool. The router looks harmless in a demo. It is an if-statement wearing a trench coat, except the condition is no longer a Boolean the team wrote. It is a string from a model that may be rate-limited, interrupted, or served from a shared free pool.

When that string is mapped to run_tool(), the model has become IAM. Free-tier IAM is not a module a production process should import. Three properties make the grant path radioactive. Side effects exist outside the transcript. Failure is ambiguous: a 429, an empty choice, or a truncated JSON blob is not the same as deny. Attribution is weak: later, nobody can prove which weights, which tenant neighbor, or which retry produced the allow.

None of those properties depend on a particular vendor name. They show up whenever inference is unpaid, shared, or allowed to vanish mid-sentence. The correct response is not a longer system prompt. The correct response is to take the grant off the model.

Red flags that already mean stop

The first red flag is a parser that looks for assent. Code that scans for yes, allow, or {"authorized": true} before calling subprocess has already appointed the model as a guard. A guard that can be sweet-talked is a doorstop.

The second is retry-as-policy. A free server that times out on the first pass and returns a confident allow on the third is not resilient. It is a coin flip with extra latency. Authorization retries must be deterministic. Free inference retries are not.

The third is secret-bearing tool descriptions. MCP-style catalogs often include endpoint URLs, header names, or example payloads. Sending that catalog to a free model so the model can choose the right tool leaks the map of the building to a clerk who does not work for the hotel.

The fourth is letting the model pick the tool server, not just the argument. Binding files versus shell versus payments is a deployment decision. It belongs in a checked-in allowlist. A free model that returns a server name is performing change management without a review.

The fifth is using the same free hop to write the policy that later constrains it. Self-authored guardrails are circular. The model that wants to call rm should not be the author of the regex that claims to forbid rm. If any one of those shows up in a design review, the free inference hop is already on the wrong side of the door.

A checker that fails closed

The following module is a proposal for a local gate. It never opens a network socket. It treats a hop as data and exits non-zero if free inference would be asked to grant.

# permission_hop_check.py
# Proposal: local static check. Not executed against a live agent in this article.

from __future__ import annotations

from dataclasses import dataclass
from typing import Iterable

SIDE_EFFECT_TOOLS = frozenset({
    "shell",
    "fs_write",
    "http_post",
    "payments",
    "email_send",
    "mcp_bind",
})

GRANT_ROLES = frozenset({
    "authorizer",
    "router",
    "tool_picker",
    "policy_writer",
})


@dataclass(frozen=True)
class Hop:
    name: str
    inference_class: str  # "free" | "dedicated" | "none"
    role: str
    tools_downstream: tuple[str, ...]
    retries_on_timeout: int = 0
    sends_tool_catalog: bool = False
    parses_assent: bool = False
    writes_own_policy: bool = False


@dataclass(frozen=True)
class Finding:
    hop: str
    code: str
    detail: str


def scan(hops: Iterable[Hop]) -> list[Finding]:
    findings: list[Finding] = []
    for hop in hops:
        if hop.inference_class != "free":
            continue
        if hop.role in GRANT_ROLES:
            findings.append(Finding(
                hop.name,
                "FREE_ON_GRANT_PATH",
                f"role {hop.role!r} treats model text as authorization",
            ))
        if hop.parses_assent:
            findings.append(Finding(
                hop.name,
                "ASSENT_PARSER",
                "yes/allow JSON is being used as a lock",
            ))
        if hop.retries_on_timeout > 0 and hop.role in GRANT_ROLES:
            findings.append(Finding(
                hop.name,
                "RETRY_AS_POLICY",
                "timeout retry can flip deny into allow",
            ))
        if hop.sends_tool_catalog:
            findings.append(Finding(
                hop.name,
                "CATALOG_EXFIL_SURFACE",
                "tool map is leaving the process toward a free model",
            ))
        if hop.writes_own_policy:
            findings.append(Finding(
                hop.name,
                "SELF_AUTHORED_GUARD",
                "the constrained model is authoring its constraint",
            ))
        dangerous = tuple(t for t in hop.tools_downstream if t in SIDE_EFFECT_TOOLS)
        if dangerous and hop.role in GRANT_ROLES:
            findings.append(Finding(
                hop.name,
                "SIDE_EFFECT_BEHIND_FREE",
                f"downstream tools {dangerous} require a non-model grant",
            ))
    return findings


def assert_no_free_grants(hops: Iterable[Hop]) -> None:
    bad = scan(hops)
    if not bad:
        return
    lines = [f"{f.code}: {f.hop}: {f.detail}" for f in bad]
    raise SystemExit("free inference on grant path\n" + "\n".join(lines))


if __name__ == "__main__":
    proposed = [
        Hop(
            name="intent_router",
            inference_class="free",
            role="router",
            tools_downstream=("shell", "fs_write"),
            retries_on_timeout=3,
            sends_tool_catalog=True,
            parses_assent=True,
        ),
        Hop(
            name="draft_plan",
            inference_class="free",
            role="drafter",
            tools_downstream=(),
        ),
    ]
    assert_no_free_grants(proposed)

A drafting hop named draft_plan passes. The router does not. That is the whole policy, encoded as data rather than as hope. The sample command is blunt on purpose:

python permission_hop_check.py
# expected: non-zero exit, FREE_ON_GRANT_PATH first

A tiny test makes the refusal mechanical so a later refactor cannot quietly put the free hop back in front of shell.

# test_permission_hop_check.py
import unittest
from permission_hop_check import Hop, scan


class FreeGrantTests(unittest.TestCase):
    def test_drafter_without_tools_is_quiet(self):
        hops = [Hop("draft", "free", "drafter", ())]
        self.assertEqual(scan(hops), [])

    def test_free_router_in_front_of_shell_fails(self):
        hops = [Hop("r", "free", "router", ("shell",), parses_assent=True)]
        codes = {f.code for f in scan(hops)}
        self.assertIn("FREE_ON_GRANT_PATH", codes)
        self.assertIn("ASSENT_PARSER", codes)
        self.assertIn("SIDE_EFFECT_BEHIND_FREE", codes)

    def test_dedicated_authorizer_is_out_of_scope(self):
        hops = [Hop("authz", "dedicated", "authorizer", ("payments",))]
        self.assertEqual(scan(hops), [])


if __name__ == "__main__":
    unittest.main()

python -m unittest test_permission_hop_check.py

The dedicated authorizer is not blessed. It is simply not this checker's job. Dedicated inference still needs audit logs, allowlists, and a human-owned policy. The checker only answers whether a free hop has wandered onto the grant path.

Hops can live next to the service config the team already reviews. A YAML list keeps the grant decision out of prompt text.

# hops.yaml — proposal, loaded by CI, never sent to a model
hops:
  - name: draft_plan
    inference_class: free
    role: drafter
    tools_downstream: []
  - name: intent_router
    inference_class: free
    role: router
    tools_downstream: [shell, fs_write]
    retries_on_timeout: 3
    sends_tool_catalog: true
    parses_assent: true

Better alternatives than a nicer prompt

Replace the model-shaped if-statement with an actual if-statement. Intent that can be named in a ticket can usually be named in code: path prefixes, authenticated user ids, change type, environment. Those predicates belong in the repo, next to the tests that freeze them.

Where language is genuinely required, keep the model on the drafting side of a queue. Let it propose a tool name and arguments as a patch. A separate, non-model component compares that proposal to a static allowlist and either drops it or waits for a reviewer. The allowlist is the lock. The model is the comment on the lock.

Timeouts must deny. Empty choices must deny. Malformed JSON must deny. That is not pessimism. That is how locks work when the other side of the socket can disappear. A small wrapper makes the fail-closed rule visible in the call site rather than buried in retry middleware.

# fail_closed.py — proposal for any grant-adjacent call, free or not

class GrantDenied(Exception):
    pass


def deny_on_empty_or_timeout(payload, timed_out: bool) -> dict:
    if timed_out:
        raise GrantDenied("timeout is deny")
    if not payload:
        raise GrantDenied("empty choice is deny")
    if not isinstance(payload, dict):
        raise GrantDenied("non-object payload is deny")
    if "tool" in payload or "authorized" in payload:
        raise GrantDenied("model text is not a grant")
    return payload

For MCP-style catalogs, send the model a redacted index: tool names and one-line purposes, never live URLs, never example auth headers. Bind the real server in config the model cannot see. The catalog that authorizes a call and the catalog that explains a call are different documents. Only the second one belongs near a free model.

Exit criteria

Leave the free hop the moment any of the following becomes true. A tool has a side effect outside the chat transcript. An operator would be uncomfortable pasting the prompt plus tool catalog into a public gist. A retry could change the authorization outcome. The model is asked to choose among servers, not among phrasings. The output is stored as an audit decision.

Those exits are cheap if they happen in design. They are expensive if they happen after a shell tool has already run. A free server is a reasonable place to practice the drafting side. It is an unreasonable place to keep the only copy of a policy, a catalog, or a grant.

Teams that still want a cheap room for that practice can park routing language on MonkeyCode's free model access and free server option, then run permission_hop_check.py on the hop list before anything is wired to tools. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The checker does not depend on that sandbox. If the room is unavailable, the same scan still runs on a laptop.

Who should ignore this guide

Throwaway chat experiments with no tools can ignore it. So can notebooks that never leave stdout. The guide is aimed at the hour when a demo sprouting shell and http_post starts to look like a product.

Do not use the checker as a substitute for real authorization. A green scan means only that free inference is not pretending to be IAM. It does not mean the dedicated path is safe, that MCP servers are scoped, or that secrets are absent from prompts. Do not treat SIDE_EFFECT_TOOLS as complete. Add the names the team actually ships. A payments wrapper called settle will sail through until someone writes it down.

The conservative reading is the point. Free inference is abundant at the map printer. It is the wrong clerk to hand the elevator key. Keep the grant somewhere else.

Free Inference Stops at the Persistence Boundary

Casey Li — Mon, 07 Sep 2026 14:24:57 +0000

Free-tier language models can sketch glue code. They should not author the persistence layer. That boundary is what keeps a cheap draft from becoming two competing truths about the same table.

A generator that has never paid for a bad migration will still volunteer one. It adds a column, guesses a type, skips the backfill, and renames a cache key as if old clients were a rumor. The tokens look free. The on-call is not.

A spare hotel key is a useful object. It opens a door. It does not prove the holder knows which walls are load-bearing. Schema files, migrations, lockfiles, and cache identity are load-bearing walls. Free inference can walk the hallway. It should not redraw the floor plan.

This article proposes a small, testable refusal gate that runs before any free model endpoint or free server process is invoked. The gate is boring on purpose. Boring is the isolation.

Persistence fails on a delay

Application glue usually fails loudly. A wrong helper throws. A wrong unique index waits. Two rows that were supposed to be one customer become a ticket with no stack trace and a spreadsheet of guesses.

Cache identity fails in a cousin pattern. A model asked to fix a stale payload often invents a new key prefix instead of invalidating the old one. The brownfield client still reads the previous document. Two serializers now describe one entity. The page looks randomly wrong, which is worse than a clean 500.

Lockfiles belong in the same class. A model that helpfully rewrites a constraint file is not reviewing a supply chain. It is shuffling names it does not own. That work wants the package manager that created the lock, a human reading the diff, and a change someone can actually revert.

None of this needs a glossary of agent jargon. It needs a refusal that can be tested.

A free workspace is a side canal

Some teams park draft generation on a workspace that offers free model access and a free server option. MonkeyCode is one such option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The useful role is narrow. Draft a parser. Sketch a retry wrapper. Propose a log line. Keep that work away from canonical schema. A free server changes where the generator process sleeps. It does not shrink the blast radius of ALTER TABLE.

Treat the free path as a side canal, not the river. Tasks arrive as JSON. A local gate inspects paths, intent, and a few cheap patterns. Allowed jobs may continue to whatever free endpoint the team already uses. Refused jobs print a reason and exit nonzero. No remote call. No prompt that says be careful.

A proposed refusal gate

The following module is a proposal, not a control plane. It is meant for a laptop pre-hook or a CI job. Matches are conservative. A false positive is cheaper than a silent migration.

#!/usr/bin/env python3
"""Refuse free-tier generation for persistence-shaped work.

Proposed example. Unexecuted against any vendor API in this article.
"""
from __future__ import annotations

import json
import re
import sys
from dataclasses import dataclass
from pathlib import Path

PERSISTENCE_PATHS = (
    re.compile(r"(^|/)(migrations?|alembic|prisma|liquibase)(/|$)", re.I),
    re.compile(r"(^|/)(schema|models?|entities)(/|$)", re.I),
    re.compile(r"(docker-compose|compose)\.ya?ml$", re.I),
    re.compile(
        r"(package-lock|pnpm-lock|yarn\.lock|poetry\.lock|"
        r"Pipfile\.lock|go\.sum|Cargo\.lock)$"
    ),
)

PERSISTENCE_INTENT = re.compile(
    r"\b(migrat(e|ion)|alter\s+table|add\s+column|drop\s+index|"
    r"create\s+table|foreign\s+key|cache[-_ ]?key|ttl|invalidate|"
    r"unique\s+constraint|lockfile|pin\s+version)\b",
    re.I,
)

DANGEROUS_WRITE = re.compile(
    r"\b(apply|run|execute|deploy|push)\b.*\b(migrat|schema|prod)",
    re.I,
)


@dataclass(frozen=True)
class Verdict:
    allow: bool
    reason: str
    code: int  # 0 allow, 2 refuse, 3 malformed


def inspect_task(task: dict) -> Verdict:
    if not isinstance(task, dict):
        return Verdict(False, "task must be an object", 3)

    paths = task.get("paths") or []
    intent = str(task.get("intent") or "")
    target = str(task.get("target") or "")

    if target.lower() in {"prod", "production", "primary-db"}:
        return Verdict(False, "production target is a hard stop", 2)

    for raw in paths:
        text = str(raw)
        for rx in PERSISTENCE_PATHS:
            if rx.search(text):
                return Verdict(False, f"path looks like persistence: {text}", 2)

    blob = f"{intent}\n{target}\n" + "\n".join(str(p) for p in paths)
    if PERSISTENCE_INTENT.search(blob):
        return Verdict(False, "intent names schema, cache, or lockfile work", 2)
    if DANGEROUS_WRITE.search(blob):
        return Verdict(False, "intent asks to apply schema work", 2)

    return Verdict(True, "no persistence markers", 0)


def main(argv: list[str]) -> int:
    if len(argv) != 2:
        sys.stderr.write("usage: free_tier_gate.py TASK.json\n")
        return 3
    payload = json.loads(Path(argv[1]).read_text(encoding="utf-8"))
    verdict = inspect_task(payload)
    sys.stdout.write(json.dumps(verdict.__dict__, indent=2) + "\n")
    return verdict.code


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

A refusing fixture can be this small.

{
  "id": "task-0412",
  "target": "staging",
  "intent": "Add a column for display name and invalidate the user cache key",
  "paths": [
    "app/models/user.py",
    "alembic/versions/20260907_display_name.py"
  ]
}

Run it without touching a network.

python3 free_tier_gate.py task-0412.json
echo $?

Stdout should include "allow": false. The process should exit 2. An allowed neighbor might be a CSV preview helper under tools/ whose intent never names tables, TTLs, or locks. That job can travel to a free model. The fixture above should not.

A thin wrapper keeps the rule visible in shell history.

#!/usr/bin/env bash
set -euo pipefail
python3 free_tier_gate.py "$1"
status=$?
if [[ "$status" -eq 2 ]]; then
  echo "refused: keep this off free inference and off any free server" >&2
fi
exit "$status"

Tests that keep the boundary from rotting

A gate without tests becomes folklore. The pytest module below is still a proposal. It pins three refusals and one allowance so a later edit cannot loosen the patterns in the name of convenience.

# test_free_tier_gate.py
from free_tier_gate import inspect_task


def test_refuses_alembic_path():
    v = inspect_task({
        "intent": "small cleanup",
        "paths": ["alembic/versions/20260907_init.py"],
        "target": "dev",
    })
    assert v.allow is False
    assert v.code == 2


def test_refuses_cache_intent_without_schema_path():
    v = inspect_task({
        "intent": "rotate the cache key for the profile payload",
        "paths": ["services/profile.py"],
        "target": "dev",
    })
    assert v.allow is False


def test_refuses_production_even_for_docs():
    v = inspect_task({
        "intent": "update README",
        "paths": ["README.md"],
        "target": "production",
    })
    assert v.allow is False


def test_allows_isolated_parser_sketch():
    v = inspect_task({
        "intent": "draft a CSV to dict helper with tests",
        "paths": ["tools/csv_preview.py", "tests/test_csv_preview.py"],
        "target": "dev",
    })
    assert v.allow is True
    assert v.code == 0

pytest -q test_free_tier_gate.py

If those four cases do not hold, the team does not have a boundary. It has a comment that used to be a script.

Red flags that mean the job already crossed

A request that names both a column and a cache prefix is not a coding task. It is a compatibility task. Free inference will optimize for the sentence that looks complete, not for the client that still hashes the old key.

A request that includes apply, migrate, and a hostname in one paragraph is operations work. An idle free server is not a migration runner with backups. The idle machine will still execute whatever DDL the model invented.

A request to make the model match production without a dumped schema is fiction. The generator will invent a plausible Postgres. Plausible is not the database sitting in front of the app.

A request to regenerate a lockfile because the build is red is supply-chain work. The package manager that wrote the lock is the alternative. A chat window is not an inventory of what the company may ship.

When those shapes appear, the exit is immediate. Stop the free-tier job. Copy the intent into a ticket. Hand it to someone who can see the live schema. Do not relax the regex. Do not add a system prompt about being careful with migrations. Prompts are not isolation.

Cheaper alternatives that stay honest

Draft the helper on the free path. Keep the migration in a checked-in template the team already trusts. Fill column names by hand. Generate backfill SQL from information_schema, not from a transcript.

For cache identity, keep one module of constants. Humans rename constants. Tests fail when a constant vanishes. A model that invents user:v3:profile beside user:profile will not run those tests unless the gate never lets it start.

# proposed cache identity module, not generated per request
PROFILE_KEY = "user:profile:v2"


def profile_cache_key(user_id: str) -> str:
    return f"{PROFILE_KEY}:{user_id}"

For architecture the operator does not yet understand, stop generation entirely. Cheap code that encodes the wrong aggregate boundary is how technical debt arrives already compiled. Read the existing tables. Draw the foreign keys on paper. Then write the migration without a model in the loop.

Exit criteria for the whole habit

Abandon the free path for a given service when a refused job was smuggled around the gate. Abandon it when a cache incident traces to a generated prefix. Abandon it when a migration lands without a down step or a backfill. Abandon it when nobody can say which process is allowed to touch alembic/.

At that point the gate is theater. Move schema work to a reviewed, paid, or fully local toolchain. Leave free inference on parsers, fixtures, and docs.

Who should not use this

This refusal style is the wrong medicine for a throwaway SQLite file on a laptop with no users. It is also the wrong souvenir for a platform team that already has expand-contract migrations, backup-checked runners, and a DBA rotation. Those teams need their existing controls.

Do not treat an allow verdict as a skip-review card. A CSV parser can still read a path it should not see. The artifact only claims to keep persistence-shaped work off a free model and off a free server. It does not claim those environments are safe for secrets, production snapshots, or unsupervised agents.

Regexes will miss a migration hidden in a generic db/update.py. JSON can be laundered by renaming paths. This article does not measure latency, token volume, hardware, or uptime on any free server. Those figures move and are not claimed here. The claim is smaller. Persistence is a hard stop. A stop that cannot be tested is not a stop.

Operators who already route sketches through a MonkeyCode workspace can hang this script on the same local queue they use for free-model drafts, then keep schema work somewhere a human still has to type yes.

When Not to Host an Agent on Free Inference

Casey Li — Mon, 07 Sep 2026 03:43:34 +0000

Free inference is a sandbox, not a substrate. An agent job that can change production state, retain customer text, or invent missing architecture should be refused before the first token is requested. The cheapest host is still the wrong host when the failure mode is irreversible.

Teams keep parking agent loops on complimentary model endpoints because the queue looks empty and the invoice looks like zero. That habit collapses the moment the loop is allowed to assume a schema, a secret, or a deployment target. The rest of this article is a refusal protocol: a job card, a local preflight gate, and exit criteria that fire even after a run has started.

Complimentary stacks exist. MonkeyCode currently advertises free model access and a free server option for experimental work. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability claims are the only product facts used here. They do not license a production role, a data-processing agreement, or a promise that a remote session is memoryless.

The protocol still applies if a different vendor is hosting the sandbox. The gate inspects the job, not the logo on the endpoint.

A free-tier agent run should begin as a file, not as a chat. The card below is deliberately boring. Boring cards are auditable. Flashy prompts are not.

# jobcard.yaml — a job this article would allow only as a sandbox draft
id: draft-readme-only
host_class: free_inference
allowed_tools: [read_file, list_dir]
forbidden_tools: [git_push, kubectl, send_email, http_post]
data_classes: [public_docs]
output_sink: local_drafts/
sla: none
human_merge: required
max_wall_clock_s: 180
max_tool_calls: 12
assumption_policy: refuse_if_unspecified

The card is a contract with the operator, not with the model. Models do not honor YAML. Operators can.

The script that follows never calls a model. It only decides whether a model is allowed to be called. That distinction is the whole point. If the gate cannot run offline, it is not a gate.

#!/usr/bin/env python3
"""preflight_gate.py — refuse free-inference jobs that do not belong there."""
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("pip install pyyaml\n")
    sys.exit(2)

FORBIDDEN_DATA = {"customer_pii", "prod_logs", "secrets", "auth_tokens"}
FORBIDDEN_TOOLS = {"git_push", "kubectl", "terraform_apply", "send_email", "http_post"}
FORBIDDEN_SINKS = {"origin/main", "production_db", "customer_ticket"}


def load_card(path: Path) -> dict:
    data = yaml.safe_load(path.read_text())
    if not isinstance(data, dict):
        raise ValueError("job card must be a mapping")
    return data


def refusals(card: dict) -> list[str]:
    reasons: list[str] = []
    data_classes = set(card.get("data_classes") or [])
    tools = set(card.get("allowed_tools") or [])
    sink = card.get("output_sink") or ""
    sla = (card.get("sla") or "none").lower()

    overlap = data_classes & FORBIDDEN_DATA
    if overlap:
        reasons.append(f"data class not allowed on free inference: {sorted(overlap)}")

    bad_tools = tools & FORBIDDEN_TOOLS
    if bad_tools:
        reasons.append(f"tool can mutate the outside world: {sorted(bad_tools)}")

    if any(tag in sink for tag in FORBIDDEN_SINKS):
        reasons.append(f"output sink leaves the sandbox: {sink}")

    if sla not in {"none", "best_effort"}:
        reasons.append(f"SLA {sla!r} requires a contracted endpoint")

    if card.get("human_merge") != "required":
        reasons.append("free-inference output must stay off any unattended merge")

    if int(card.get("max_wall_clock_s") or 0) <= 0:
        reasons.append("missing wall-clock budget owned outside the model")

    if int(card.get("max_tool_calls") or 0) <= 0:
        reasons.append("missing tool-call budget owned outside the model")

    if card.get("assumption_policy") != "refuse_if_unspecified":
        reasons.append("agent may fill unspecified architecture with guesses")

    if card.get("host_class") != "free_inference":
        reasons.append("card is not claiming free inference; refuse to guess the host")

    return reasons


def main(argv: list[str]) -> int:
    if len(argv) != 2:
        sys.stderr.write("usage: preflight_gate.py jobcard.yaml\n")
        return 2
    card = load_card(Path(argv[1]))
    reasons = refusals(card)
    if reasons:
        sys.stderr.write("REFUSE\n")
        for line in reasons:
            sys.stderr.write(f"- {line}\n")
        return 1
    sys.stdout.write("ALLOW_SANDBOX\n")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv))

Run it before any client is pointed at a free endpoint.

python3 preflight_gate.py jobcard.yaml
echo $?

A non-zero exit is the success case for this article. The gate did its job when it blocked a run, not when it blessed one.

The first red flag is architectural silence. Agent loops are fluent at filling gaps. A missing service boundary becomes a guessed queue. A missing auth story becomes a hardcoded header. That fluency is useful on a whiteboard and poisonous on a complimentary endpoint, because the host has no duty to record why the guess was made. If the job card cannot name the components the agent is allowed to mention, the job does not belong on free inference. It belongs in a design review, or in a deterministic scaffold that fails closed.

The second red flag is residual data. Unredacted production logs look like perfect few-shot material. They also look like an accidental export. A hallway whiteboard is a fair analogy for a free endpoint: anyone who can see the board can copy it, and nobody issues a retention certificate for the eraser. Customer identifiers, session cookies, and stack traces with hostnames should never ride that board. Paid endpoints with a written processing term are the alternative. Local offline models are another. Silence is a third.

The third red flag is an unbounded outside world. Read-only tools still exfiltrate. Write tools mutate. If http_post, kubectl, or git_push appears on the allow list, the sandbox has already been redefined as production-adjacent. A free server that holds the only copy of agent state is the same mistake in a different coat. State that matters needs a disk the operator controls, not a session the operator cannot inventory.

The fourth red flag is a service promise the host never made. On-call copy, customer replies, and release-blocking summaries all imply an SLA, even when nobody wrote the letters S, L, and A. Complimentary inference is best-effort by nature. Treating it as capacity planning is a category error. The alternative is a contracted API, a human-authored template, or a script that does not speak.

A second card shows the refuse path without theatre.

# jobcard.bad.yaml — expected REFUSE
id: page-the-customer
host_class: free_inference
allowed_tools: [read_file, http_post, send_email]
data_classes: [customer_pii, prod_logs]
output_sink: customer_ticket
sla: p1_15m
human_merge: optional
max_wall_clock_s: 0
max_tool_calls: 0
assumption_policy: fill_gaps

python3 preflight_gate.py jobcard.bad.yaml
# REFUSE
# - data class not allowed on free inference: ['customer_pii', 'prod_logs']
# - tool can mutate the outside world: ['http_post', 'send_email']
# - output sink leaves the sandbox: customer_ticket
# - SLA 'p1_15m' requires a contracted endpoint
# ...

Preflight is not enough once a run has started. The model can still request a tool that was never on the card, or keep answering after the wall clock has expired. The wrapper below is a local proxy. It does not improve model quality. It only preserves the refusal.

# tool_proxy.py — labeled example; wire this in front of any tool runner
import time

class FreeTierExit(RuntimeError):
    pass

class ToolProxy:
    def __init__(self, card: dict):
        self.allowed = set(card["allowed_tools"])
        self.max_calls = int(card["max_tool_calls"])
        self.deadline = time.monotonic() + int(card["max_wall_clock_s"])
        self.calls = 0
        self.schema_misses = 0

    def check(self, tool_name: str, args_ok: bool) -> None:
        if time.monotonic() > self.deadline:
            raise FreeTierExit("wall-clock budget exhausted; abandon the host")
        if tool_name not in self.allowed:
            raise FreeTierExit(f"tool {tool_name!r} is not on the card")
        self.calls += 1
        if self.calls > self.max_calls:
            raise FreeTierExit("tool-call budget exhausted; abandon the host")
        if not args_ok:
            self.schema_misses += 1
            if self.schema_misses >= 2:
                raise FreeTierExit("repeated schema failure; model is guessing")

Exit criteria should be dull enough to automate. Two schema misses. One disallowed tool name. One request to disable the gate. One inability to replay the job card from disk. Any of those is sufficient. Charming explanations from the model are not a counter-argument. They are the failure mode.

Better alternatives are usually less conversational. Version bumps and changelog formatting belong in scripts. Incident notes belong in a runbook a human already trusted last quarter. Private drafts of public docs can sit on a local model that never accepts a network route. Customer email belongs on an endpoint with retention controls, or it belongs unwritten until a person writes it. Free inference is what remains after those jobs have been removed.

The approach has limits. The gate believes the card. A human who labels production logs as public_docs will receive ALLOW_SANDBOX and a problem. The gate cannot prove that a remote free server is stateless, isolated, or free of cross-session residue. It cannot measure model quality, latency, or quota, and this article invents none of those figures. It also cannot stop a later process from pasting sandbox output into a merge. Refusal is a local habit, not a cryptographic seal.

Who should not use this pattern is clearer than who should. Regulated workloads should not. On-call should not. Anything with a customer on the other end of the sink should not. Solo experiments that cannot name a second reviewer should not treat ALLOW_SANDBOX as review. The pattern is for operators who already suspect the complimentary host is the wrong place and want that suspicion encoded as a non-zero exit code.

If the gate stays red, leave the complimentary host idle. That idle result is the useful one. If it prints ALLOW_SANDBOX and the work is a disposable draft with a human still sitting on the merge, the free model access and free server option named above is one place that draft can live. Run the gate until it refuses something real. A sandbox that never refuses is not a sandbox.