A canary is a production contract with a short fuse. Free, best-effort inference can narrate the numbers after the fact. It must not be the function that returns promote or rollback while live traffic is already on the new build.
Merge gates fail in a waiting room. Canaries fail in public. That difference is the whole argument. A blocked pull request wastes a morning. A wrong canary call spends user sessions on a bad binary, then spends the on-call's night undoing it. The verdict path has to stay boring: same snapshot, same thresholds, same exit code, every time.
Think of the canary as a fuse, not a suggestion box. A fuse does not ask a language model whether the smell of ozone is “probably fine.” It opens the circuit on a number. Commentary can come later, in a ticket, in a slack thread, in a postmortem draft. Commentary does not hold the wire.
This article is a field guide for keeping that wire out of unpaid inference. The working example is a tiny collector that already exists in most stacks: two JSON snapshots, a handful of SLIs, and a process exit code that a deploy job can trust. The optional model sits downstream of that code, and it cannot change the code's mind.
Why this boundary is not a merge-gate rerun
Pre-merge review asks whether a change is allowed to exist. A canary asks whether a change that already exists should keep eating traffic. Latency, error ratio, and saturation are not prose. They are counters. Once those counters feed a prompt, three failure modes show up that never appear in a linter.
The first is timeout-as-approval. Free endpoints shed load. If the model call is on the hot path, a 429 or a hung stream can be misread as “no signal,” and the pipeline promotes. Silence is not a green check. The second is log-to-prompt injection. Canary windows scrape traces, user agents, and error strings. Those strings are untrusted input. A model that is allowed to vote can be steered by a payload that was supposed to be telemetry. The third is rerun drift. The same snapshot, sent twice, can come back as rollback then hold. A fuse that flickers is not a fuse.
Rate limits during an incident make the problem worse, not better. The hour a service is on fire is the hour a free pool is busiest. Putting the verdict on that pool couples production recovery to someone else's spare capacity. That coupling is the red flag that ends the design review.
A deterministic verdict, then optional prose
The artifact below is intentionally small. It reads two files, compares a fixed set of SLIs, and exits 0 to promote, 2 to rollback, or 3 to hold for a human. No network call lives inside verdict(). A separate helper may draft a paragraph. The helper's return value is logged and discarded.
Label this as a worked example, not a production SRE platform. Swap the JSON files for Prometheus queries in a real job. Do not swap the exit codes for a model score.
# canary_verdict.py
from __future__ import annotations
import argparse
import json
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Literal
Decision = Literal["promote", "rollback", "hold"]
EXIT = {"promote": 0, "rollback": 2, "hold": 3}
@dataclass(frozen=True)
class Window:
requests: int
errors: int
p99_ms: float
cpu_pct: float
@property
def error_ratio(self) -> float:
if self.requests <= 0:
return 1.0
return self.errors / self.requests
@dataclass(frozen=True)
class Policy:
min_requests: int = 200
max_error_delta: float = 0.005
max_p99_ratio: float = 1.25
max_cpu_pct: float = 85.0
def load_window(path: Path) -> Window:
raw = json.loads(path.read_text())
return Window(
requests=int(raw["requests"]),
errors=int(raw["errors"]),
p99_ms=float(raw["p99_ms"]),
cpu_pct=float(raw["cpu_pct"]),
)
def verdict(baseline: Window, canary: Window, policy: Policy) -> Decision:
# Gray zone: not enough traffic to trust the ratio.
if canary.requests < policy.min_requests:
return "hold"
if canary.error_ratio - baseline.error_ratio > policy.max_error_delta:
return "rollback"
if baseline.p99_ms > 0 and (canary.p99_ms / baseline.p99_ms) > policy.max_p99_ratio:
return "rollback"
if canary.cpu_pct > policy.max_cpu_pct:
return "rollback"
return "promote"
def maybe_comment(decision: Decision, baseline: Window, canary: Window) -> str:
"""Side-channel prose. Must never be consulted by verdict()."""
return (
f"decision={decision} baseline_err={baseline.error_ratio:.4f} "
f"canary_err={canary.error_ratio:.4f} p99 {baseline.p99_ms}->{canary.p99_ms}"
)
def main() -> int:
parser = argparse.ArgumentParser(description="Deterministic canary verdict")
parser.add_argument("--baseline", type=Path, required=True)
parser.add_argument("--canary", type=Path, required=True)
parser.add_argument("--comment", action="store_true")
args = parser.parse_args()
policy = Policy()
baseline = load_window(args.baseline)
canary = load_window(args.canary)
decision = verdict(baseline, canary, policy)
print(decision)
if args.comment:
print(maybe_comment(decision, baseline, canary), file=sys.stderr)
return EXIT[decision]
if __name__ == "__main__":
raise SystemExit(main())
A deploy job should treat the process the way it treats grep or jq: read the integer, ignore stderr. The commands below freeze two snapshots and prove the exit code does not depend on a model being reachable.
cat > baseline.json <<'EOF'
{"requests": 8000, "errors": 16, "p99_ms": 120.0, "cpu_pct": 41.0}
EOF
cat > canary.json <<'EOF'
{"requests": 900, "errors": 22, "p99_ms": 210.0, "cpu_pct": 58.0}
EOF
python canary_verdict.py --baseline baseline.json --canary canary.json
echo "exit=$?"
That canary loses on error delta and p99 ratio, so the process exits 2. Flip errors down to 2 and p99_ms down to 110, and the same binary exits 0. Drop requests to 40, and it exits 3. Three numbers. No temperature parameter.
Tests that refuse to call a model
The test file is the other half of the artifact. If a future refactor smuggles a client into verdict(), these cases still fail closed only if the function stays pure. Keep the tests offline. Do not mark them integration and then skip them when a key is missing.
# test_canary_verdict.py
from canary_verdict import Policy, Window, verdict
POLICY = Policy()
BASE = Window(requests=5000, errors=10, p99_ms=100.0, cpu_pct=40.0)
def test_rollback_on_error_spike():
canary = Window(requests=2000, errors=40, p99_ms=100.0, cpu_pct=40.0)
assert verdict(BASE, canary, POLICY) == "rollback"
def test_rollback_on_p99_regression():
canary = Window(requests=2000, errors=4, p99_ms=160.0, cpu_pct=40.0)
assert verdict(BASE, canary, POLICY) == "rollback"
def test_hold_when_sample_is_tiny():
canary = Window(requests=20, errors=0, p99_ms=90.0, cpu_pct=30.0)
assert verdict(BASE, canary, POLICY) == "hold"
def test_promote_when_inside_budget():
canary = Window(requests=2000, errors=4, p99_ms=110.0, cpu_pct=44.0)
assert verdict(BASE, canary, POLICY) == "promote"
python -m pytest -q test_canary_verdict.py
A useful CI check is even narrower: rg -n "openai|anthropic|httpx|requests\.post" canary_verdict.py should be empty except in a file that is not imported by the job. The grep is crude. Crude is the point. The canary path should be greppable by someone half awake.
Red flags, in the order they usually arrive
The first red flag is a prompt that includes the words “decide whether to promote.” That sentence is the design defect, even if the rest of the template looks careful. The second is a retry wrapper around the model call inside the same step that writes the verdict. Retries hide brownouts. Brownouts are when a bad canary most needs a hard rollback. The third is using generated prose as the audit log. Auditors need the snapshot and the policy version, not a paragraph that says the spike was “likely a cold start.”
A fourth flag is quieter. Teams let the model propose new thresholds after each incident, then paste those thresholds back into the policy without a review. That loop is how a fuse becomes a mood. Threshold changes belong in the same review path as the service code. They are production config. They are not chat residue.
Gray traffic is not a loophole. If the canary is at 1% and the model is “only advising,” the pipeline will still grow a habit of trusting the advice. Habits become defaults. Defaults become the path that runs at 03:00. Exit the pattern before the percentage climbs.
Where a free model still earns a seat
None of this says models are useless around deploys. They are useful one layer off the fuse. A model can draft a human-readable summary of a verdict that already exists. It can turn rollback plus two JSON files into a ticket body. It can help an engineer iterate on the collector script in a scratch environment, where a wrong answer costs a re-run rather than a slice of production.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option fit that scratch loop: rewrite the parser, generate extra unit cases, argue with a draft of maybe_comment(). They do not fit the process that Kubernetes, Nomad, or a homegrown deployer will exec as the source of truth. Keep the paid, rate-limited, or local deterministic checker in the job. Keep the free pool in the notebook.
A practical split looks like this in a pipeline. Step one writes snapshots. Step two runs canary_verdict.py and captures $?. Step three, if and only if the job is already finishing, posts a comment. If step three times out, the deploy still did the right thing. That is the only coupling that survives a free-tier outage.
Exit criteria for pulling the model out
Pull the model from the canary path the first time any of these is true. A timeout or HTTP 429 would have changed a promote/rollback. Two reruns of the same snapshot disagreed. A trace line showed up inside a prompt and the output mentioned that line as evidence. The on-call could not explain the decision without quoting the model. The policy file and the prompt drifted, and nobody could say which one the job used.
Any single item is enough. Waiting for a user-facing incident is not an exit criterion. It is an after-action report.
Better replacements are already dull, which is their virtue. A sequential test on error counts. A hard SLO burn alert that pages a person in the gray zone. A hold state that refuses to auto-promote when sample size is small. Those tools do not write fluently. They also do not invent a story in which a 2x p99 is “within normal weekend noise.”
Who should not use a model-shaped canary at all
Skip the pattern entirely if the service handles payments, authn, medical, or anything with a legal retention rule on the logs that would be pasted into a prompt. Skip it if the canary runs in an environment that cannot pin a policy version next to the binary. Skip it if the team cannot test the checker offline. Skip it if the only rollback button is the same chat session that produced the go-live note.
Hobby clusters and internal demo apps can still use a model to explain a graph. They should still keep verdict() pure. The cost of copying a bad habit from a demo into a revenue path is how most of these designs actually ship.
Limitations of the artifact are plain. The windows are files, not live queries. The policy is four numbers, not a full multi-window sequential test. There is no multiple-comparison correction, no warmup exclusion, no per-endpoint breakdown. Those gaps are reasons to grow the checker, not reasons to ask a free endpoint to “use its judgment.” Judgment is what the fuse is there to refuse.
The industry conversation this month keeps collapsing two jobs into one: sketching code and owning production risk. Canaries are the second job. Let a free model talk about the snapshot after the circuit has already opened or closed. Do not hand it the handle.
Top comments (0)