Build a Two-Host Canary for a Small AI CLI

Last Tuesday I shipped a 90-line ticket CLI. It tagged inbound mail as bug, billing, or noise. On my laptop the canary exited zero. On a free remote box it exited two. Same repo. Same flags. A different host. Would you ship that?

I did not. I wanted a receipt I could diff. Not a vibe. Not a screenshot. A fail-closed gate sheet a small team can copy.

The real constraint

I had forty-five minutes and a zero-dollar budget. The CLI calls a free model for one classification. Then it prints one JSON object. Local runs looked fine. Remote runs looked “fine” too, until I checked the exit code.

What changed? The model? The Python hash seed? The timezone? An agent-filled env var? I needed evidence, not a theory.

Abandon rule, set up front: if two receipts still drift after I pin TZ, PYTHONHASHSEED, and strip volatile JSON keys, I stop using that remote host for this CLI. No extra flags. No “it is probably fine.”

Why a second host at all?

I needed a box that did not add an invoice. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode’s free model access and free server option as that second host. The receipt format does not depend on it. Any SSH box works. The product just removed the “I will do it later” excuse.

A free remote run is useful. A free remote run that lies is expensive. Quiet drift burns a weekend.

Copy this gate sheet

Print it. Stick it next to the README. Fail closed on any red row.

1. Interpreter gate. Same Python minor version on both hosts. Evidence: python3 -c "import sys; print(sys.version)". Fail if majors differ. Fail if minors differ without a written note.
2. Cwd gate. Same repo commit. Same dirty state. Evidence: git rev-parse HEAD and git status --porcelain. Fail if either host is dirty in a different way.
3. Env allowlist gate. Only four keys may differ by design: none. Evidence: a sliced env dict. Fail if a new key appears. Fail if a required key is missing.
4. Clock gate. Pin TZ=UTC on both hosts. Evidence: the receipt’s tz field. Fail if timestamps remain in stdout after the strip step.
5. Canary gate. Run one frozen argv. Evidence: exit code, stdout sha256, stderr line count. Fail on any mismatch after stripping volatile keys.
6. Model-call gate. The CLI must record model_called: true|false itself. Evidence: that boolean in stdout. Fail if remote prints JSON without a call you expected. Fail if local called the model and remote did not.
7. Rollback gate. One env file must restore the last green pair. Evidence: receipts/last-green.json. Fail the ship if that file is missing.

No row is optional. A skipped row is a red row.

Time and cost boundary

Budget: forty-five minutes. Money: zero. Tooling: Python 3.11+, git, and SSH. If the sheet takes longer than forty-five minutes, the CLI is not ready for a second host. Shrink the canary. Do not grow the sheet.

The receipt schema

Keep it boring. One JSON object per host. No prose.

{
  "host": "local",
  "python": "3.11.9",
  "commit": "a1b2c3d",
  "dirty": false,
  "tz": "UTC",
  "env": {
    "PYTHONHASHSEED": "0",
    "TZ": "UTC",
    "CI": "1"
  },
  "argv": ["python3", "ticket_cli.py", "--canary", "fixtures/mail_001.txt"],
  "exit_code": 0,
  "stdout_sha256": "REPLACE_ME",
  "stderr_lines": 0,
  "model_called": true,
  "stripped_keys": ["generated_at", "request_id", "latency_ms"]
}

Name files receipts/local.json and receipts/remote.json. Diff them as data. Do not diff them as feelings.

The canary script

Save this as host_receipt.py. It is the whole artifact. Run it on both hosts with the same argv.

#!/usr/bin/env python3
"""Capture a fail-closed host receipt for one CLI canary."""
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path

ALLOW_ENV = ("PYTHONHASHSEED", "TZ", "CI", "MODEL_ENDPOINT")
STRIP_KEYS = ("generated_at", "request_id", "latency_ms")


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def env_slice() -> dict[str, str]:
    return {k: os.environ[k] for k in ALLOW_ENV if k in os.environ}


def strip_volatile(stdout: str) -> str:
    stdout = stdout.strip()
    if not stdout:
        return stdout
    try:
        payload = json.loads(stdout)
    except json.JSONDecodeError:
        return stdout
    if isinstance(payload, dict):
        for key in STRIP_KEYS:
            payload.pop(key, None)
        return json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return stdout


def python_version() -> str:
    proc = subprocess.run(
        [sys.executable, "-c", "import sys; print(sys.version.split()[0])"],
        capture_output=True,
        text=True,
        check=True,
    )
    return proc.stdout.strip()


def git_commit() -> str:
    proc = subprocess.run(
        ["git", "rev-parse", "HEAD"],
        capture_output=True,
        text=True,
        check=True,
    )
    return proc.stdout.strip()


def git_dirty() -> bool:
    proc = subprocess.run(
        ["git", "status", "--porcelain"],
        capture_output=True,
        text=True,
        check=True,
    )
    return bool(proc.stdout.strip())


def main() -> int:
    if len(sys.argv) < 3:
        print("usage: host_receipt.py <host-label> <command...>", file=sys.stderr)
        return 2
    host = sys.argv[1]
    argv = sys.argv[2:]
    proc = subprocess.run(argv, capture_output=True, text=True)
    stripped = strip_volatile(proc.stdout)
    receipt = {
        "host": host,
        "python": python_version(),
        "commit": git_commit(),
        "dirty": git_dirty(),
        "tz": os.environ.get("TZ", ""),
        "env": env_slice(),
        "argv": argv,
        "exit_code": proc.returncode,
        "stdout_sha256": sha256_text(stripped),
        "stderr_lines": len(proc.stderr.splitlines()),
        "model_called": None,
        "stripped_keys": list(STRIP_KEYS),
        "stderr_preview": proc.stderr.splitlines()[:8],
    }
    try:
        payload = json.loads(stripped) if stripped else {}
        if isinstance(payload, dict) and "model_called" in payload:
            receipt["model_called"] = payload["model_called"]
    except json.JSONDecodeError:
        pass
    Path("receipts").mkdir(exist_ok=True)
    out = Path("receipts") / f"{host}.json"
    out.write_text(json.dumps(receipt, indent=2) + "\n", encoding="utf-8")
    print(out)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Commands I actually run:

export TZ=UTC PYTHONHASHSEED=0 CI=1
mkdir -p receipts fixtures
# fixture input is one short mail file, committed, not generated live
python3 host_receipt.py local python3 ticket_cli.py --canary fixtures/mail_001.txt
ssh free-server 'cd ~/ticket-cli && TZ=UTC PYTHONHASHSEED=0 CI=1 python3 host_receipt.py remote python3 ticket_cli.py --canary fixtures/mail_001.txt'
scp free-server:~/ticket-cli/receipts/remote.json receipts/remote.json
python3 compare_receipts.py receipts/local.json receipts/remote.json

The compare step

Save this as compare_receipts.py. Exit two means do not ship.

#!/usr/bin/env python3
import json, sys
from pathlib import Path

COMPARE = ("python", "commit", "dirty", "tz", "env", "argv",
           "exit_code", "stdout_sha256", "stderr_lines", "model_called")

def main() -> int:
    if len(sys.argv) != 3:
        print("usage: compare_receipts.py local.json remote.json", file=sys.stderr)
        return 2
    left = json.loads(Path(sys.argv[1]).read_text())
    right = json.loads(Path(sys.argv[2]).read_text())
    drifted = []
    for key in COMPARE:
        if left.get(key) != right.get(key):
            drifted.append({"key": key, "local": left.get(key), "remote": right.get(key)})
    report = {"ok": not drifted, "drifted": drifted}
    print(json.dumps(report, indent=2))
    return 0 if report["ok"] else 2

if __name__ == "__main__":
    raise SystemExit(main())

Green means both hosts tell the same story. Red means you still have a laptop CLI, not a shipped CLI.

Failure fixture

This fixture is labeled. I keep it under fixtures/failure_tz.json so the gate has a known-red case.

Local receipt had tz: "UTC" and a stripped stdout hash of 9c1e…. Remote receipt had tz: "" and a different hash. The CLI had printed generated_at in local time. I had not stripped it yet. Compare exited two. Good. That is the point.

Fix order I use:

1. Export TZ=UTC on both hosts.
2. Add generated_at to STRIP_KEYS.
3. Pin PYTHONHASHSEED=0.
4. Re-run both receipts.
5. If model_called flipped, stop. That is not host drift. That is a different program.

After the strip, both hashes matched. Exit codes matched. I copied receipts/local.json to receipts/last-green.json. That file is the rollback evidence. Lose it and the next drift has no baseline.

What if stderr line counts still differ? I do not hash stderr. I count lines, then read the preview. A progress spinner on one host is enough to fail the gate. Silence the spinner in --canary mode. Do not “accept” noisy stderr.

What the CLI must print

Your canary command should print one JSON object. Include model_called. Include the class label. Include nothing with a clock in it. A tiny contract beats a rich log.

{
  "label": "billing",
  "model_called": true,
  "input_sha256": "b77a"
}

If an agent later adds confidence or renames label, the stdout hash breaks. That is wanted. The canary is a lock, not a welcome mat.

Who should not use this

Skip this sheet if you ship regulated data to a shared box. Skip it if you need an SLA. Skip it if the CLI has no frozen fixture. Skip it if two hosts are actually two products. A free remote server is a second computer. It is not a staging program.

Also skip it if you cannot pin the argv. Live mail as canary input will drift forever. Commit one file. Classify that file. Nothing else.

What I will not claim

I am not publishing model names. I am not publishing quotas. I am not publishing hardware. Those numbers go stale by lunch. The gate sheet stays useful when the vendor page changes. That is why the receipt never stores a marketing string.

MonkeyCode is open source. The free model access and free server option were the cheap second host for this canary. If you already have SSH somewhere, run the same two commands there. The sheet does not care about the brand on the prompt.

Ship rule

I only tag a release when compare_receipts.py exits zero. I only keep the remote host when last-green.json exists. I only rerun the model path when model_called stays true on both sides. Everything else is a laptop demo.

Did the remote host save time? Yes, once the receipts matched. Before that it only saved confidence I did not deserve.

What field still drifts on your second host after you pin TZ and PYTHONHASHSEED?