When the Canary Judge Must Not Be a Free Model

The night shift inherited a half-done production canary. Traffic sat at five percent on the new build. A chat pane printed a green promote line.

No metric file sat beside that chat line. The on-call engineer held the canary rollout.

Service latency still showed a slow unexplained climb. The model had read a cropped dashboard screenshot. Cropped screenshots do not close a canary gate.

Why this pattern keeps showing up

Teams now paste canary graphs into a model. The model then returns fluent confident rollout prose. That prose is still untrusted generated output.

A promote is a write to production traffic. Those writes need a pinned numeric oracle. A free inference lane is not an oracle.

This article is a when-not-to field guide. It lists red flags, a gate, and exits. It does not ban draft canary notes.

Where a free lane still helps

Draft notes are cheap and fully reversible. A human can discard every drafted sentence. The promote bit is not equally reversible.

Scratch analysis can live on best-effort compute. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option.

Those options can host draft summaries and scratch notebooks. They should not host the traffic-shift judge. The judge must read files, not chat tokens.

Use the free lane for these tasks only:

• Plain recap of already exported collector metrics
• A checklist that a human still ticks
• A timeline written after the gate fires
• Suggested graphs to pull, never the verdict

Keep the free lane off these controls:

• The promote and rollback enum
• The remaining error-budget counter
• The sample-size sufficiency check
• The signed deploy receipt

Red flags

Stop and fail closed when any item below appears.

1. The model output is the only promote signal.
2. Numeric thresholds live inside a prompt, not a file.
3. The judge may retry until it says yes.
4. Canary sample size is unknown to the gate.
5. The free server also stores deploy tokens.
6. Rollback copy is generated after traffic already moved.
7. SLO burn is described in words, not computed.
8. A screenshot is treated as a metric source.
9. Branch protection accepts a chat paste as proof.
10. The notes lane can also call the deploy API.

Any single flag is already enough cause. Two flags mean the control plane drifted.

Artifact: a fail-closed canary gate

The gate below reads two local JSON files. This gate process never calls a model. A non-zero process exit blocks the promotion.

Treat the listing as a lab example. Wire it to real collectors before any production use.

policy.json

{
  "min_requests": 5000,
  "max_error_rate": 0.002,
  "max_p95_ms": 180,
  "max_slo_burn": 1.0,
  "min_healthy_hosts": 2,
  "required_window_seconds": 600
}

canary_metrics.json

{
  "requests": 1800,
  "errors": 9,
  "p95_ms": 210,
  "slo_burn": 1.4,
  "healthy_hosts": 2,
  "window_seconds": 240,
  "source": "prometheus",
  "collected_at": "2026-09-18T02:14:00Z"
}

canary_gate.py

#!/usr/bin/env python3
"""Fail-closed canary gate. Model text is never an input."""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

ALLOWED_SOURCES = {"prometheus", "datadog", "cloudwatch", "otel-collector"}


def load_json(path: Path) -> dict:
    with path.open("r", encoding="utf-8") as handle:
        data = json.load(handle)
    if not isinstance(data, dict):
        raise ValueError(f"{path} must be an object")
    return data


def violations(policy: dict, metrics: dict) -> list[str]:
    faults: list[str] = []
    source = metrics.get("source")
    if source not in ALLOWED_SOURCES:
        faults.append(f"untrusted metric source: {source!r}")

    requests = int(metrics["requests"])
    errors = int(metrics["errors"])
    if requests < int(policy["min_requests"]):
        faults.append("sample size below min_requests")

    error_rate = errors / requests if requests else 1.0
    if error_rate > float(policy["max_error_rate"]):
        faults.append(f"error_rate {error_rate:.5f} over cap")

    if float(metrics["p95_ms"]) > float(policy["max_p95_ms"]):
        faults.append("p95 over cap")

    if float(metrics["slo_burn"]) > float(policy["max_slo_burn"]):
        faults.append("slo burn over cap")

    if int(metrics["healthy_hosts"]) < int(policy["min_healthy_hosts"]):
        faults.append("healthy host count too low")

    window = int(metrics["window_seconds"])
    if window < int(policy["required_window_seconds"]):
        faults.append("observation window too short")

    return faults


def main() -> int:
    parser = argparse.ArgumentParser(description="Pinned canary gate")
    parser.add_argument("--policy", required=True)
    parser.add_argument("--metrics", required=True)
    parser.add_argument("--model-verdict", default="")
    args = parser.parse_args()

    if args.model_verdict.strip():
        print("reject: model verdict is not a legal input", file=sys.stderr)
        return 2

    policy = load_json(Path(args.policy))
    metrics = load_json(Path(args.metrics))
    faults = violations(policy, metrics)
    if faults:
        print("reject:")
        for item in faults:
            print(f"- {item}")
        return 1

    print("promote: pinned metrics passed policy")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Commands

python3 -m py_compile canary_gate.py

python3 canary_gate.py \
  --policy policy.json \
  --metrics canary_metrics.json
echo $?

python3 canary_gate.py \
  --policy policy.json \
  --metrics canary_metrics.json \
  --model-verdict promote
echo $?

The first command should compile with no syntax errors. The second command should exit with status one. The third command should exit with status two.

Both of those failing paths remain fully closed. A later green run needs a longer window. It also needs a lower measured error rate.

Always export the metrics from the collector itself. Do not paste model JSON into the metrics flag. Treat any model-shaped payload as hostile gate input.

Expected reject lines for the sample files:

reject:
- sample size below min_requests
- error_rate 0.00500 over cap
- p95 over cap
- slo burn over cap
- observation window too short

Nine errors over 1800 requests yield 0.005. That ratio already exceeds the 0.002 cap. The window is 240 seconds against a 600 second floor.

Passing fixture

Replace the metrics file with this passing object. Use it only after the collector window fills.

{
  "requests": 8200,
  "errors": 6,
  "p95_ms": 142,
  "slo_burn": 0.4,
  "healthy_hosts": 3,
  "window_seconds": 900,
  "source": "prometheus",
  "collected_at": "2026-09-18T02:30:00Z"
}

Then rerun the gate without a model verdict flag. Exit zero is the only promote signal. Raw chat text remains illegal as gate input.

Manual test plan

1. Save the three files in an empty working directory.
2. Run the compiler check on the gate script.
3. Run the gate against the failing metrics file.
4. Confirm the process exits with status one.
5. Pass a model verdict and confirm status two.
6. Swap in the passing fixture and rerun the gate.
7. Confirm the process now exits with status zero.
8. Refuse any change that reads a chat file.

Label every fixture as synthetic lab input. Do not ship the sample timestamps as proof.

Decision table

Signal	Owner	Free lane allowed	Gate action
Request count	Collector	Draft a human recap	Block if below floor
Error ratio	Collector	No	Block if over cap
p95 latency	Collector	No	Block if over cap
SLO burn	Collector	No	Block if over cap
Promote enum	Gate script	No	File and metrics only
Rollback copy	Human	Draft only	Publish after rollback
Deploy token	Secret store	No	Never on a free host
Chat looks-good line	Model	Notes only	Ignore as input

Print this table beside the main deploy pipeline. Reviewers should refuse chat screenshots as rollout evidence.

Better alternatives

Use a pinned collector and a file-backed policy. Keep humans on the exception path only.

These numbered options beat a free model judge.

1. Pin Prometheus recording rules and check them with promtool.
2. Run a unit-tested gate script in CI, as above.
3. Drive a feature-flag service only from numeric thresholds.
4. Require a two-person promote for exceptions only.
5. Auto-rollback on burn with no model in the loop.

Example recording rule sketch:

groups:
  - name: canary_gate
    rules:
      - record: canary:error_rate5m
        expr: sum(rate(http_requests_total{canary="1",status=~"5.."}[5m]))
          / sum(rate(http_requests_total{canary="1"}[5m]))

That recording rule stays versioned in git. The model does not edit it during the incident.

A tiny CI wrapper

Keep the gate in the same job that ships. Do not hide it in a chat plugin.

#!/usr/bin/env bash
set -euo pipefail

if [ ! -s canary_metrics.json ]; then
  echo "reject: empty metrics file" >&2
  exit 1
fi

python3 canary_gate.py --policy policy.json --metrics canary_metrics.json
status=$?
if [ "$status" -ne 0 ]; then
  echo "gate closed; draft notes may still be written" >&2
  exit "$status"
fi

./promote.sh

This CI wrapper must refuse empty metrics files. Empty silence is not a healthy canary.

Exit criteria

Leave the free-lane judge pattern when any criterion trips.

• A promote happened from chat text alone.
• Sample size was guessed, not counted.
• Deploy credentials shared a host with scratch notebooks.
• Two canaries in a month lacked a metric file.
• Reviewers accepted a screenshot as the SLO.
• The model retried until it emitted promote.
• Rollback started after the narrative, not the burn.

True exit means the deploy pipeline itself changes. Remove the model tool from the deploy job. Keep the draft notes in a side channel. Require the canary gate script to exit zero.

Limitations

This field guide assumes numeric SLIs already exist. Some products lack a clean error ratio. This gate will refuse all of those rollouts. That hard refusal is the intended gate behavior.

The script does not detect metric spoofing. A writable metrics file is still a trust boundary. Lock the collector identity in a real deployment.

Who should not use this approach:

• Teams with no collector should not adopt this gate yet.
• Groups must not ask a model to invent the SLO.
• Deadline pipelines must not promote on generated vibes.
• Labs must not treat the sample JSON as production truth.

Do not use a free server as the collector either. Best-effort hosts often drop live time series. Dropped series can look like healthy silence.

What to keep on the free lane

Keep drafting the recap after metrics are exported. Keep teaching newcomers from those written recaps. Keep summarizing exported JSON for a human reader.

Drop the habit of asking the model to vote. A short pipeline split looks like this.

python3 export_canary_metrics.py > canary_metrics.json
python3 canary_gate.py --policy policy.json --metrics canary_metrics.json
status=$?
if [ "$status" -ne 0 ]; then
  echo "gate closed; notes may still be drafted"
  exit "$status"
fi
./promote.sh

The draft step can run after a reject. It must not run as the judge.

Closing

A canary promotion is a control-plane write. Control-plane writes still need pinned numeric oracles. Free inference is a notebook, not a judge.

Hold the promote bit in a file and a script. Let models talk about those metric files. Do not let them sign the files.

Scratch notes can use MonkeyCode's free model access. The free server option can host those notebooks. Keep the local canary gate fail closed.