Five Fail-Closed Gates, One Script: Rehearse Your AI Feature on a Free Tier First

My readiness checklist for small AI features is a Markdown file. It has nine boxes and zero exit codes. Nobody has opened it since August.

That is the whole problem. A checklist you read is a wish. A checklist you run is a gate.

So I rewrote mine as one script. It runs in about two seconds. It exits non-zero when anything is wrong. And it runs first against a deliberately broken fake model, not against a real one.

Here is the harness, the evidence it writes, and the limits of rehearsing on a free tier.

The gate is not the model call

Most teams test the model. Few test the guard around the model.

Ask yourself: if the model returned truncated JSON right now, what happens? Does your service crash, show a blank card, or quietly bill you twice? If you cannot answer in one sentence, you have a doc, not a gate.

A gate needs three parts. A deterministic check. A written evidence record. A fail-closed action that runs without a human.

Gate 0: prove the gates actually fail

This is the step everyone skips. You cannot trust a green run until you have seen a red one.

So the harness ships with fixtures that are already broken. Empty string. Garbage prose. Truncated JSON. Oversized output.

Run the suite against those fixtures first. If anything passes, your gate is decoration. Fix the gate before you touch a real provider.

That single habit has saved me more deploy nerves than any prompt tweak.

The harness

Pure standard library. No dependencies. Drop it in tools/gates.py.

#!/usr/bin/env python3
"""gates.py - five fail-closed gates for a small AI feature."""
import argparse
import json
import os
import time
from dataclasses import dataclass, asdict, field

MAX_OUT_CHARS = 4000
MAX_LATENCY_MS = 4000
REQUIRED_KEYS = {"summary", "confidence"}

FIXTURES = {
    "empty": "",
    "garbage": "lorem ipsum dolor sit amet " * 40,
    "truncated": '{"summary": "refunds take five days", "confidence": 0.',
    "oversize": json.dumps({"summary": "x" * 9000, "confidence": 0.9}),
    "good": json.dumps({"summary": "refunds take five days", "confidence": 0.9}),
}

@dataclass
class Result:
    gate: str
    ok: bool
    detail: str
    evidence: dict = field(default_factory=dict)

def call_fake(fixture: str) -> str:
    return FIXTURES[fixture]

def call_live(prompt: str) -> str:
    # Wire this to your provider. Read the key from os.environ only.
    raise SystemExit("call_live() is not wired yet")

def gate_schema(text: str) -> Result:
    try:
        obj = json.loads(text)
    except json.JSONDecodeError as e:
        return Result("schema", False, f"unparseable: {e}", {"chars": len(text)})
    missing = REQUIRED_KEYS - obj.keys()
    return Result("schema", not missing, f"missing: {sorted(missing)}", {"keys": sorted(obj)})

def gate_size(text: str) -> Result:
    ok = 0 < len(text) <= MAX_OUT_CHARS
    return Result("size", ok, f"{len(text)} chars, cap {MAX_OUT_CHARS}")

def gate_latency(ms: int) -> Result:
    return Result("latency", ms <= MAX_LATENCY_MS, f"{ms} ms, ceiling {MAX_LATENCY_MS} ms")

def gate_budget(used: int, allowance: int) -> Result:
    ok = used < allowance * 0.2
    return Result("budget", ok, f"{used} of {allowance} tokens, 20% cap")

def gate_rollback(flag: str) -> Result:
    ok = os.path.exists(flag)
    return Result("rollback", ok, f"kill switch at {flag}")

def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--model", choices=["fake", "live"], default="fake")
    ap.add_argument("--fixture", default="garbage", choices=sorted(FIXTURES))
    ap.add_argument("--out", default="evidence/gates.json")
    args = ap.parse_args()

    allowance = int(os.environ.get("TOKEN_ALLOWANCE", "0"))
    if allowance <= 0:
        raise SystemExit("set TOKEN_ALLOWANCE to your own plan's number")

    start = time.monotonic()
    if args.model == "fake":
        text = call_fake(args.fixture)
    else:
        text = call_live("summarize the support fixture")
    ms = int((time.monotonic() - start) * 1000)

    results = [
        gate_schema(text),
        gate_size(text),
        gate_latency(ms),
        gate_budget(len(text.split()), allowance),
        gate_rollback(os.environ.get("KILL_SWITCH", ".rollback/feature.flag")),
    ]

    failed = [r.gate for r in results if not r.ok]
    payload = {
        "when": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "model": args.model,
        "fixture": args.fixture,
        "passed": len(results) - len(failed),
        "failed": failed,
        "results": [asdict(r) for r in results],
    }
    os.makedirs(os.path.dirname(args.out), exist_ok=True)
    with open(args.out, "w") as f:
        json.dump(payload, f, indent=2)
    print("failed: " + ", ".join(failed) if failed else "all gates passed")
    raise SystemExit(1 if failed else 0)

if __name__ == "__main__":
    main()

Run it in four commands

mkdir -p .rollback && touch .rollback/feature.flag
export TOKEN_ALLOWANCE=100000
python3 tools/gates.py --model fake --fixture garbage; echo "exit=$?"
python3 tools/gates.py --model fake --fixture good; echo "exit=$?"

The first run must print failed: schema and exit=1. The second must print all gates passed and exit=0.

If both runs look identical, stop. You have a logging script, not a gate.

Then open evidence/gates.json. That file is the thing you attach to a release, not a screenshot of a green checkbox.

What each gate owes you

Gate	Evidence written	Fail-closed action
schema	raw output plus parse error	serve cached result or a plain error card
size	character count vs cap	re-ask once, then error
latency	measured milliseconds	fall back to the non-AI path
budget	tokens used vs allowance	stop calls for the day, no silent retry
rollback	kill-switch path and mtime	refuse to deploy if the flag is missing

Five rows. Five verdicts. No adjectives.

Rehearsing on a free tier

Here is where a free option earns its place. Failing paths cost calls. Empty output, truncation, retries, oversized responses — those are the expensive ones to discover in production.

I rehearse the whole red path against MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free allowance lets me burn calls on intentionally broken fixtures without a card on file.

Two rules keep this honest.

1. Read the current allowance from the product page yourself. Copy the number into TOKEN_ALLOWANCE. Never hard-code a figure you read in a blog post, including this one.
2. Treat every green run on a free tier as a rehearsal, not a benchmark. It proves your gates fire. It says nothing about paid throughput.

A free server is also a fine place to host the kill switch check. If curl cannot reach your flag, your rollback is a rumour.

What this does not buy you

Be blunt with yourself here.

• A free tier is not a load test. Latency under one caller tells you nothing about ten.
• The fake model tests your code, not your prompt. Quality is a separate harness.
• A disposable server is not production rollback. It proves the mechanism, not the blast radius.
• Cost ratios do not transfer. Your 20% cap is a policy, not a measurement.

Who should not use this

Skip this if your feature already moves money. Skip it if you have a pager rotation and real SLAs; you need runbooks, not scripts. Skip it if there is nobody who can own a red exit code at 2am.

Also skip it if you cannot name your kill switch. That is a design gap, and no harness fixes it.

When to abandon the approach

Give it two hours. If the fake run still cannot go red, the harness is wrong. Delete it and start over rather than keeping a fuzzy gate that always passes.

And if a gate cannot produce a deterministic verdict, remove it. A checklist with nine soft boxes is how we got here.

One question for my next iteration: what is the one input your feature cannot survive losing? I want to turn that answer into the next fixture.