DEV Community: Emery Huang

Split Freeze From Unfreeze, and Map Each Alert to a Read-Only Command

Emery Huang — Sun, 20 Sep 2026 11:23:04 +0000

A paging channel without a freeze rule is just a group chat that happens to be loud. I will not treat an on-call runbook as live until every alert maps to a first command, an escalation tree names a human, and freeze is a different checklist from unfreeze. Can a model write fluent incident prose at two in the morning? Yes, and that is exactly why I keep it away from those fields.

Quiet services still page someone, and that someone still needs a command that will not write. If you skip the human sections, the next page becomes a brainstorm with extra confidence. Have you watched a channel fill with suggested kubectl lines while nobody names a freeze owner? I have, and I will not pretend a generated paragraph is a substitute for that owner.

Why pretty runbooks still fail the first page

Most half-finished runbooks fail in the same places, and none of those places are grammar. The alert does not name a service owner who can answer the phone. The first commands are write-heavy, so the first keystroke mutates production. The escalation path is a Slack handle that nobody pages after midnight. Freeze and unfreeze share one vague sentence, so nobody knows who may lift the gate.

AI-assisted drafting makes those gaps prettier, not smaller. Why would a model know your after-hours phone tree? Why would it know which replica is allowed to restart without a freeze? It will invent confident steps unless you refuse to let it own those fields. The useful work is still naming owners, commands, and gates in daylight, then leaving the model a narrow drafting job.

I still use a drafting box for symptom text, because writing that prose by hand is slow and easy to postpone. Disclosure: This article was prepared as part of MonkeyCode's product outreach. When I need a quiet place to draft non-authoritative sections, MonkeyCode's free model access and free server option are enough to generate candidate paragraphs that I then strip of any write action.

The human sections the pager actually needs

Treat the runbook as a contract the pager can parse, not as a blog post about the service. I keep these sections in a checked-in file, and I reject drafts that leave any of them as prose-only comments.

Alert catalog — each paging alert has an id, a service, a severity, and a human owner.
First-command map — each alert id points at read-only commands, with an explicit deny list for writes.
Escalation tree — primary, secondary, and manager, with a timeout that is a number, not a vibe.
Freeze checklist — who may freeze writes, what evidence they record, and what stays allowed.
Unfreeze checklist — a different owner path, a different evidence set, and a named blast check.

Notice freeze and unfreeze are not one section with a reversible verb. If they share a heading, people will unfreeze with the same confidence they used to freeze. Do you want that person to be a model that never saw your change window? I do not.

Artifact: a runbook file the pager can fail closed

This is a proposed schema, not a war story with fake metrics. Drop it in runbooks/payments-api.yaml and refuse to page off a document that cannot load.

# Proposed template — label every field human-owned or draft-only.
apiVersion: oncall.example/v1
kind: Runbook
metadata:
  service: payments-api
  owners: ["primary-oncall", "payments-tl"]
  timezone: America/Los_Angeles
spec:
  alerts:
    - id: ALRT-PAY-429
      name: checkout_latency_p99
      severity: page
      owner: "payments-primary"
      first_commands:
        - "kubectl --context=prod-ro -n payments get deploy payments-api -o wide"
        - "kubectl --context=prod-ro -n payments get pods -l app=payments-api"
        - "curl -sS https://status.internal/payments/health | jq ."
      deny_commands:
        - "kubectl delete"
        - "kubectl scale"
        - "kubectl apply"
        - "helm upgrade"
      escalate_after_minutes: 10
  escalation:
    primary: { role: "payments-primary", timeout_minutes: 10 }
    secondary: { role: "payments-secondary", timeout_minutes: 15 }
    manager: { role: "payments-tl", timeout_minutes: 20 }
    never: ["@here", "random model chat"]
  freeze:
    required_before_writes: true
    owner_role: "incident-commander"
    evidence:
      - "alert_id"
      - "first_command_output_hash"
      - "blast_radius_service_list"
    allowed_during_freeze: ["read", "page", "capture"]
  unfreeze:
    owner_role: "incident-commander"
    second_signer_role: "service-owner"
    evidence:
      - "alert_id_cleared_or_mitigated"
      - "write_plan_printed"
      - "rollback_command"
    forbidden_if_missing: ["second_signer", "rollback_command"]

Would I let a model fill deny_commands from memory of some other cluster? No, because that list is how you keep the first five minutes read-only. The model may draft the health-check narrative that sits under the YAML. It does not get to invent the owner role or the second signer.

A validator that fails before the rotation starts

I want the runbook to fail in CI, not in the paging channel. The script below is a proposed check you can run locally; it does not talk to production and it does not claim a benchmark.

#!/usr/bin/env python3
"""validate_runbook.py — proposed structural check, not an incident replay."""
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("pip install pyyaml\n")
    sys.exit(2)

WRITE_TOKENS = ("delete", "scale", "apply", "upgrade", "restart", "exec --")


def fail(msg: str) -> None:
    sys.stderr.write(msg + "\n")
    sys.exit(1)


def main(path: str) -> None:
    data = yaml.safe_load(Path(path).read_text())
    spec = data.get("spec") or {}
    alerts = spec.get("alerts") or []
    if not alerts:
        fail("no alerts: a runbook that cannot match a page is a wiki page")

    freeze = spec.get("freeze") or {}
    unfreeze = spec.get("unfreeze") or {}
    if not freeze or not unfreeze:
        fail("freeze and unfreeze must both exist as mappings")
    if freeze.get("owner_role") and freeze.get("owner_role") == unfreeze.get("owner_role"):
        # Same person may hold both hats, but the checklists cannot be identical.
        if freeze.get("evidence") == unfreeze.get("evidence"):
            fail("freeze evidence must not equal unfreeze evidence")
    if not unfreeze.get("second_signer_role"):
        fail("unfreeze needs a second signer role")

    esc = spec.get("escalation") or {}
    for key in ("primary", "secondary", "manager"):
        node = esc.get(key) or {}
        if not node.get("role") or not node.get("timeout_minutes"):
            fail(f"escalation.{key} needs role and timeout_minutes")

    for alert in alerts:
        aid = alert.get("id") or "<missing-id>"
        if not alert.get("owner"):
            fail(f"{aid}: alert has no human owner")
        cmds = alert.get("first_commands") or []
        if not cmds:
            fail(f"{aid}: no first commands")
        for cmd in cmds:
            low = cmd.lower()
            if any(tok in low for tok in WRITE_TOKENS):
                fail(f"{aid}: first command looks like a write: {cmd}")
        if not alert.get("deny_commands"):
            fail(f"{aid}: deny_commands missing")

    print(f"ok: {path} ({len(alerts)} alerts)")


if __name__ == "__main__":
    if len(sys.argv) != 2:
        fail("usage: python3 validate_runbook.py runbooks/payments-api.yaml")
    main(sys.argv[1])

Run it like this, on a laptop, before you accept a week of pages:

python3 validate_runbook.py runbooks/payments-api.yaml
# expected: ok: runbooks/payments-api.yaml (1 alerts)

If the validator prints ok while unfreeze.evidence still equals freeze.evidence, I missed a check and I want you to fail the file by hand. Should a green script be enough to unfreeze production? No. It is only enough to prove the document is not empty.

First commands stay read-only until freeze is named

I keep a tiny wrapper so the tired person at the keyboard cannot “just scale it” from muscle memory. This is a proposed shell gate, not a cluster agent.

#!/usr/bin/env bash
# firstcmd.sh — proposed read-only wrapper. Do not point this at a write kubecontext.
set -euo pipefail
ALERT_ID="${1:?alert id}"
RUNBOOK="${2:-runbooks/payments-api.yaml}"
CONTEXT="${KUBE_RO_CONTEXT:?set KUBE_RO_CONTEXT to a read-only context}"

if [[ "$CONTEXT" == *prod-w* || "$CONTEXT" == *writable* ]]; then
  echo "refusing: context looks writable: $CONTEXT" >&2
  exit 1
fi

echo "alert=$ALERT_ID runbook=$RUNBOOK context=$CONTEXT"
echo "freeze_named=${FREEZE_NAMED:-no}"
if [[ "${FREEZE_NAMED:-no}" != "yes" ]]; then
  echo "writes remain blocked until FREEZE_NAMED=yes and the freeze checklist is signed"
fi

What belongs in first_commands? Gets, describes, logs with a tail limit, and health URLs that cannot mutate. What does not belong? Anything that restarts, scales, applies, deletes, or opens an interactive shell on a prod pod. If the model drafts a “quick restart” as step one, that is a defect in the draft, not a shortcut.

Escalation is a tree with minutes, not a mention

I want numbers on the tree because “ping secondary if needed” is how pages rot in a thread. Primary has ten minutes. Secondary has fifteen. Manager has twenty. After that, you are not brainstorming; you are late.

Do not escalate to @here.
Do not escalate to a model chat that cannot take the phone.
Do not list a person who is already the freeze owner unless a second signer still exists for unfreeze.
Do record the time you moved to the next role, even if the next role is you with a different hat.

If your team is two people, the tree can still exist. It just becomes honest about how thin the bench is. Is a thin bench a reason to skip the tree? It is a reason to write the tree larger than the bench, then staff it later.

Freeze and unfreeze as two checklists

Use a decision table during the incident, on paper or in the channel topic. The table is the artifact that keeps freeze from becoming a mood.

Question	Freeze	Unfreeze
Who signs?	Incident commander	Commander plus service owner
What evidence?	Alert id, first-command output, blast list	Alert mitigated, printed write plan, rollback
Writes allowed?	No, except documented break-glass	Only the printed plan
Model output allowed as proof?	No	No
Can the same human hold both hats?	Yes, if the evidence sets differ	Only with a second signer

I will freeze as soon as a write is tempting and the blast list is still unnamed. I will not unfreeze because a generated summary says the error rate “looks better.” Looks better than what baseline, captured by which command, against which alert id? If you cannot answer that, the freeze stays.

Where a drafting model is allowed, and where it is not

Let the model propose symptom text, dashboard links you already host, and questions to ask the primary. Do not let it propose owners, phone numbers, deny lists, or unfreeze signers. Those fields are boring on purpose, because boring fields are the ones people skip when a page is loud.

If you paste a generated runbook straight into the wiki, you have a document that reads finished and behaves empty. That is the failure mode I am trying to make expensive. The validator is cheap. The second signer on unfreeze is cheaper than a write that nobody can roll back.

Limitations, and who should not use this

This approach assumes you already have paging, a read-only kube context or equivalent, and a place to store YAML that humans review. It does not measure latency, does not replay traffic, and does not prove the first command is the right command for a novel failure. The schema will happily accept a wrong-but-read-only curl if you typed it.

Do not use this if you are a solo hobby project with no pager and no production writes worth freezing. Do not use it as a reason to skip a real staging reproduction when you have time. Do not use a drafting server as the system of record for phone trees or credentials. Do not treat free model access as an on-call teammate; it cannot take the escalation slot, and it cannot sign unfreeze.

If your incident process already requires two-person review for every production write, you may only need the alert-to-command map and the validator. That is fine. The point is not to collect headings. The point is to make the first five minutes of a page boring enough that nobody invents a write.

Fill Four On-Call Lists Before You Accept the Rotation

Emery Huang — Fri, 18 Sep 2026 07:48:12 +0000

On-call is not a conversation with a model while the pager is still firing at you. A rotation is ready only when four lists exist: the alert catalog, the first commands, the escalation packet, and the freeze rule. I will not accept a week of pages that depend on a chat transcript, no matter how fluent that transcript looks. Would you really open a blank prompt at three in the morning and call that a procedure?

Narrative runbooks feel complete in daylight, then collapse when two alerts land together. People scroll, argue about meaning, and invent write commands that nobody reviewed against a named surface. I want files I can grep, scripts that only print state, a clock that forces escalation, and a freeze line that does not depend on mood. Does a five-page story help you when the ID on the page is not even in the catalog?

The useful question this week is not whether assistants can draft English. The question is whether you will call that draft an operating procedure before it survives a checker. I let a model fill comments. I reject the file until the four lists validate. That is the method, and chat is none of the lists.

The four lists, named on purpose

I keep the on-call file as four lists because each list answers a different failure. The catalog answers "what is this page." First commands answer "what do we look at before we speak." Escalation answers "who gets the evidence, and when." Freeze answers "which writes are illegal until a human writes a sentence." If any list is missing, you are improvising, and improvisation is not a runbook.

Here is the order I actually enforce when a rotation starts:

Every page ID in the last month has a catalog row, including a "not this" note.
Every catalog row points at a read-only first-command script that exists on disk.
Escalation names roles, minute bounds, and the three fields in the packet.
Unknown IDs freeze writes immediately; unfreeze is a filled sentence, not a vibe.

Would a generated paragraph satisfy those four gates? Only the parts that survive the YAML and the checker below. Everything else stays a comment in a draft branch.

List 1: the alert catalog is a table, not a blog

Every page needs a stable ID, a meaning, a severity, and a "not this" note. If the ID is missing from the catalog, the incident is already in freeze, and nobody invents a new class on the call. New classes are born in daylight, with a script on disk and a minute bound that a human typed. Why would you let a model mint an ID while customers are already waiting?

Catalog shape I keep in git

# runbook/alerts.yaml — labeled template, not a live service file
alerts:
  - id: "checkout.p95.latency"
    severity: sev2
    means: "p95 checkout latency exceeded the burn threshold for 5 minutes"
    not_this:
      - "batch jobs on the reporting warehouse"
      - "synthetic probes failing from one region"
    first_command_set: "latency-read"
    escalate_after_minutes: 15
    write_allowed: false
  - id: "payments.auth.5xx"
    severity: sev1
    means: "payment auth 5xx rate crossed the error budget page"
    not_this:
      - "client timeouts from a single ISP"
    first_command_set: "payments-read"
    escalate_after_minutes: 8
    write_allowed: false

Notice write_allowed starts false on every row. A catalog row that defaults to writes is a loaded gun with the safety off. Unknown IDs do not get a row invented during the page; they get freeze, then escalation, then a daylight edit. I want the on-call person to grep, not to negotiate meaning in a thread.

List 2: first commands print state and then stop

First commands print state. They do not restart, scale, mute, deploy, or patch. I want the output pasted into the incident doc before anyone discusses a mutation, because the paste is the only evidence the next human can trust. If the command is not in the set named by the alert, it does not run, even when someone swears it is "just a quick look." Would you skip the paste step because the output looked obvious?

Read-only command sets

# first-commands/latency-read.sh
# labeled example: swap in your own read endpoints after review
set -euo pipefail
echo "UTC $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "alert_id=${ALERT_ID:-unset}"
curl -fsS "$STATUS_URL/checkout/p95" | head -c 4096
echo
kubectl get deploy checkout -o wide
kubectl get hpa checkout -o yaml | sed -n '1,80p'

# first-commands/payments-read.sh
set -euo pipefail
echo "UTC $(date -u +%Y-%m-%dT%H:%M:%SZ)"
curl -fsS "$STATUS_URL/payments/auth/5xx" | head -c 4096
echo
kubectl logs deploy/payments-auth --tail=80 --since=10m \
  | grep -E '5[0-9]{2}|timeout' | tail -n 40

I keep these scripts tiny on purpose. A first-command file that contains apply, delete, restart, or scale is not a first command; it is a change that snuck in through naming. The checker later treats those strings as failures, because I have watched "just restart it" show up in a reconnaissance folder.

List 3: escalation is a clock plus a packet

Escalation is not "ping someone if you feel stuck." It is a minute bound, a named role, and a packet of evidence that another human can read without joining your head. The packet is the alert ID, the first-command output path, and the freeze state. If any of those three is missing, you are not escalating; you are venting into a channel. What happens when the named person is asleep and the runbook only lists a nickname?

Escalation decision table

Clock	Condition	Who you page	Packet you must send
T+0	Page fires	primary on-call	alert ID, catalog hit or miss
Catalog minute bound	First commands cannot classify	secondary and service owner	output path, freeze=on
Any time	Alert ID missing from catalog	incident commander	freeze=on, no writes
After classify	A write is proposed	owner of the write surface	command text and revert line

I keep roles as functions, not heroic names. Put the rotation source next to the role so the packet can move when the human changes. A Slack handle with no rotation pointer is a hope, and hope is not an escalation path.

# runbook/escalation.yaml — labeled template
roles:
  primary: "pagerduty://checkout-primary"
  secondary: "pagerduty://checkout-secondary"
  service_owner: "pagerduty://checkout-owner"
  incident_commander: "pagerduty://ic-weekday"
freeze_on_unknown_alert: true
require_packet:
  - alert_id
  - first_command_output_uri
  - freeze_state

List 4: freeze unknown pages; unfreeze with a sentence

Here is the freeze rule I will defend in a postmortem. If the alert ID is absent from the catalog, freeze writes immediately and escalate. If the first-command set cannot classify the symptom inside the catalog minute bound, freeze writes and escalate. Unfreeze requires the catalog ID, a classified symptom, a named write owner, and an explicit unfreeze block in the incident doc. Is that slower than a heroic restart? Yes, and that delay is the control.

Unfreeze is a sentence, not a feeling in the room. I write it like this, and I refuse anything shorter:

UNFREEZE checkout writes
alert_id=checkout.p95.latency
classified_as=db_pool_saturation
write_owner=checkout-owner
command=increase checkout pool from 20 to 40
revert=set pool back to 20
expires=20m

If you cannot fill those fields, you stay frozen. The revert line is not decoration; it is the only way the next on-call person knows how to undo your write without asking the chat. I have no interest in an unfreeze that cannot be read after the call ends.

Artifact: fail the runbook before the rotation starts

I do not argue about completeness in the war room. A checker fails the tree on a laptop, or in CI, before anyone accepts the pager. The script below is a proposal you can run against the YAML shapes above. It does not prove production is healthy. It proves the four lists are present, writes are not smuggled into first commands, and unknown alerts are set to freeze.

#!/usr/bin/env python3
"""runbook_check.py — unlabeled fields fail the runbook."""
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("install pyyaml before running this checker\n")
    sys.exit(2)

REQUIRED_ALERT = {
    "id",
    "severity",
    "means",
    "not_this",
    "first_command_set",
    "escalate_after_minutes",
    "write_allowed",
}
REQUIRED_ROLES = {"primary", "secondary", "service_owner", "incident_commander"}
REQUIRED_PACKET = {"alert_id", "first_command_output_uri", "freeze_state"}
WRITE_MARKERS = ("kubectl delete", "kubectl apply", "restart", "scale ")


def fail(msg: str) -> None:
    print(f"FAIL: {msg}")
    raise SystemExit(1)


def main(root: Path) -> None:
    alerts = yaml.safe_load((root / "runbook/alerts.yaml").read_text())
    esc = yaml.safe_load((root / "runbook/escalation.yaml").read_text())
    rows = alerts.get("alerts") or fail("alerts.yaml missing alerts list")
    seen: set[str] = set()
    for row in rows:
        missing = REQUIRED_ALERT - set(row)
        if missing:
            fail(f"alert {row.get('id')!r} missing {sorted(missing)}")
        if row["id"] in seen:
            fail(f"duplicate alert id {row['id']}")
        seen.add(row["id"])
        if row.get("write_allowed") is True:
            fail(f"{row['id']} must not default write_allowed to true")
        cmd = root / "first-commands" / f"{row['first_command_set']}.sh"
        if not cmd.is_file():
            fail(f"missing first-command script {cmd}")
        text = cmd.read_text()
        for marker in WRITE_MARKERS:
            if marker in text:
                fail(f"{cmd} looks like a write; first commands must be read-only")
        minutes = row["escalate_after_minutes"]
        if not isinstance(minutes, int) or minutes <= 0:
            fail(f"{row['id']} escalate_after_minutes must be a positive int")
    roles = esc.get("roles") or fail("escalation.yaml missing roles")
    missing_roles = REQUIRED_ROLES - set(roles)
    if missing_roles:
        fail(f"missing roles {sorted(missing_roles)}")
    if esc.get("freeze_on_unknown_alert") is not True:
        fail("freeze_on_unknown_alert must be true")
    packet = set(esc.get("require_packet") or [])
    if packet != REQUIRED_PACKET:
        fail(f"require_packet must be {sorted(REQUIRED_PACKET)}")
    print(f"OK: {len(rows)} alerts, freeze-on-unknown enabled")


if __name__ == "__main__":
    main(Path(sys.argv[1]) if len(sys.argv) > 1 else Path("."))

Run it like this on a toy tree before you point it at the real rotation files:

python3 runbook_check.py .
# expected on a complete tree:
# OK: N alerts, freeze-on-unknown enabled

If the checker fails, the rotation does not start. That sounds harsh until you remember the alternative: a page whose first command does not exist, or a freeze rule that lives only in someone's memory. I would rather fail CI on Tuesday than invent YAML while the pager is already open.

How a page should actually move

When the page fires, I want a boring sequence, not a clever one. The primary copies the alert ID, greps the catalog, and runs only the named first-command script. Output goes into the incident doc before any theory gets typed. Classification either happens inside the minute bound or it does not; there is no third state called "we are pretty sure."

If the ID is missing, freeze is already on, and the packet goes to the incident commander with freeze=on and no writes. If classification fails the clock, freeze stays on and secondary plus owner get the output path. Only after those gates does anyone draft a write, and that write still needs an unfreeze block with a revert line. Would you skip a gate because the room feels aligned? The gates exist for the room that feels aligned.

Where a draft assistant fits, and where it does not

I will let a model propose means and not_this text from an exported list of alert names. I will not let it invent IDs, minute bounds, role URIs, or write commands. The draft is a comment. The YAML is the contract. That split is how you use an assistant without pretending the assistant took the page.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Comment drafts for those catalog sentences can sit in MonkeyCode, which has free model access and a free server option, and only fields that survive review get copied into alerts.yaml. The checker does not care where the English came from. The free server is a draft bench, not an incident cluster, and generated kubectl that never sat in first-commands/ stays unread.

Would I point any assistant at a production kubeconfig to "save time"? No. Time saved on typing is worthless if the first write has no revert line. Keep the model on comments. Keep the pager on lists.

Limitations you should not hand-wave

This workflow assumes you already have paging, a place to paste output, and someone who can own a write. The checker is syntactic; it cannot prove a curl target is actually read-only, and it cannot prove a role URI reaches a human who is awake. Minute bounds are guesses until you rehearse them with a game-day page. YAML that passes CI can still describe the wrong service, or a "not this" note that was true last quarter and is false tonight.

Assistants hallucinate confident not_this notes. If you paste those notes without a human who has taken that page, you will freeze the wrong subsystem or unfreeze too early. I treat every generated sentence as untrusted until a person who has carried the pager edits it. The four lists reduce improvisation. They do not replace judgment, and they do not replace a rehearsal.

Who should not use this shape

Do not use this shape if you have no pager, no secondary, and no incident doc. Do not use it to justify running model output against production because the sentences sounded operational. Do not use a shared free server as the system of record for secrets, kubeconfigs, or customer data. Tiny internal tools with one operator may prefer a single shell script over four YAML files, and that is fine.

If your culture rewards the fastest unmute, this freeze rule will feel like friction. That friction is the point of the runbook. I would rather explain a delayed write than explain an untracked restart that nobody can revert. The four lists are the runbook. Everything else is conversation, and conversation does not unfreeze writes.

I Will Not Unfreeze Prod Until the Runbook Hash Matches the Page

Emery Huang — Thu, 17 Sep 2026 05:50:16 +0000

I will not let an assistant invent the incident path while the pager is still screaming. A usable on-call runbook names the alert class, the first command, the escalation owner, and the freeze hash before anyone types. Chat logs are not runbooks, and generated shells are not change tickets, no matter how confident they sound. If those four fields are missing, I keep writes frozen and I keep the assistant on a scratch box.

Why am I this stubborn about a document that looks like YAML? Because the last seven days of AI-coding talk keep confusing fluency with ownership, and on-call work punishes that mix. A model can draft a plausible restart in twenty seconds, then miss the replica that actually holds the lock. Would you rather argue with a chat transcript at 03:00, or execute a hashed runbook you already signed?

The contract I actually page against

I treat every page as a four-field contract, not as a brainstorming session with extra urgency. The contract is boring on purpose, and boring is what I want when the graph is red. If a field cannot be filled from the alert payload and the repo, I do not improvise a fifth field in Slack.

Here is the schema I keep in runbooks/<service>.yml. Copy it, then refuse to run writes until freeze.hash matches the file on disk.

# runbooks/payments-api.yml
apiVersion: oncall.example.com/v1
service: payments-api
owner_oncall: payments-primary
escalation:
  t_plus_15m: payments-secondary
  t_plus_30m: payments-manager
  t_plus_45m: incident-commander
alert_classes:
  - id: PAYMENTS_P99_LATENCY
    severity: page
    observe_commands:
      - id: oc1
        argv: ["kubectl", "-n", "payments", "get", "deploy", "payments-api", "-o", "wide"]
      - id: oc2
        argv: ["kubectl", "-n", "payments", "top", "pod", "-l", "app=payments-api"]
    write_commands: []   # stays empty until unfreeze
freeze:
  state: frozen
  hash_of: runbooks/payments-api.yml
  require:
    - named_blast_radius
    - matching_git_sha
    - human_signature

Notice what is missing on purpose: there is no prompt, no vibe, and no "try whatever the model suggested." Observe commands are argv arrays, not English. Write commands start empty. The freeze block is data, not a pep talk.

Field 1: alert class, not a paragraph

I bind the pager to a stable alert_class id, because humans rename dashboards and models paraphrase titles. PAYMENTS_P99_LATENCY is allowed to wake me; "the site feels slow" is not. If the alert cannot map onto one id in the YAML, I treat it as observe-only noise until a human classifies it.

Page-worthy classes get severity: page and a short observe list.
Ticket-worthy classes get severity: ticket and never unfreeze writes.
Unknown classes inherit freeze and a required escalation note.

Can your current alert rule print that id into the notification body? If it cannot, the runbook is already lying before you open a terminal.

# example Alertmanager annotation, not a prompt
description: class=PAYMENTS_P99_LATENCY ns=payments deploy=payments-api

Field 2: first commands stay read-only

My first commands are inventory, not remediation. I want the replica count, the ready condition, the recent events, and the error budget burn, in that order. Anything that mutates pods, feature flags, or DNS waits behind the freeze hash. Have you ever watched a generated one-liner delete the healthy deployment because the label selector was almost right?

I keep a tiny allowlist checker next to the YAML so the laptop refuses clever extra flags.

# tools/check_first_commands.py
import sys, yaml

ALLOWED = {
    "kubectl": {"get", "describe", "logs", "top", "api-resources"},
    "dig": {"+short"},
    "curl": {"-sS", "-o", "/dev/null", "-w"},
}

def main(path):
    doc = yaml.safe_load(open(path))
    errors = []
    for cls in doc["alert_classes"]:
        for cmd in cls["observe_commands"]:
            argv = cmd["argv"]
            bin_ = argv[0]
            if bin_ not in ALLOWED:
                errors.append(f"{cls['id']}: binary {bin_} not in observe allowlist")
                continue
            if any(tok in argv for tok in ("delete", "apply", "patch", "scale", "drain")):
                errors.append(f"{cls['id']}: mutating token in observe argv {argv}")
        if cls.get("write_commands"):
            errors.append(f"{cls['id']}: write_commands must be empty while frozen")
    if doc.get("freeze", {}).get("state") != "frozen":
        errors.append("freeze.state must start as frozen")
    if errors:
        print("\n".join(errors)); sys.exit(1)
    print(f"ok: {path} observe-only and frozen")

if __name__ == "__main__":
    main(sys.argv[1])

Run it before the page, not during it.

python3 tools/check_first_commands.py runbooks/payments-api.yml
sha256sum runbooks/payments-api.yml | tee runbooks/payments-api.yml.sha256

Field 3: escalation is a clock, not a group chat

I write escalation as timestamps against the first page, because "ask in Slack" is how incidents grow extra owners and zero decisions. Fifteen minutes of observe with no named blast radius promotes to secondary. Thirty minutes promotes to the manager. Forty-five minutes asks for an incident commander who is not also typing kubectl. Who is allowed to unfreeze if the primary is the person who wrote the bad deploy?

T+0 to T+15: primary runs observe commands only, pastes command ids into the incident doc.
T+15: secondary joins, repeats the same observe ids, and challenges any extra argv.
T+30: manager confirms customer impact and names the blast radius in one sentence.
T+45: commander owns communication; primary still cannot unfreeze without the hash check.

I do not escalate because I feel nervous. I escalate because the clock in the runbook elapsed and a field is still blank.

Field 4: freeze hash, then a signed unfreeze

This is the rule that keeps assistants useful instead of dangerous. The freeze hash is sha256 of the runbook file at the git sha I am paging from. Unfreeze is a separate artifact, not a vibes-based "looks good." If the assistant rewrites the YAML during the call, the hash breaks, and writes stay frozen. Is that annoying? Yes. Is it worse than a silent kubectl apply from a regenerated plan? Not even close.

# tools/unfreeze.sh — still a proposal until a human runs it on a signed laptop
set -euo pipefail
RB="${1:?runbook yml}"
INCIDENT="${2:?incident id}"
BLAST="${3:?blast radius sentence}"
SIG="${4:?path to signature file}"

expected=$(cut -d' ' -f1 "${RB}.sha256")
actual=$(sha256sum "$RB" | cut -d' ' -f1)
test "$expected" = "$actual" || { echo "hash mismatch: freeze holds"; exit 2; }
test -s "$SIG" || { echo "no human signature"; exit 3; }
test -n "$BLAST" || { echo "blast radius unnamed"; exit 4; }

printf 'unfrozen\nincident=%s\nblast=%s\n' "$INCIDENT" "$BLAST" > "${RB}.unfreeze"
echo "writes allowed only for commands listed after this file exists"

I still require write commands to be appended as argv arrays after unfreeze, never as free text. The assistant may propose a patch on a scratch clone. It does not get to flip freeze.state.

Where a free coding assistant actually helps

I will use a coding assistant to draft the YAML, generate the allowlist tests, and argue with my own escalation clock, but only on a throwaway box. MonkeyCode is relevant here because free model access and a free server option give me that scratch loop without borrowing a production jumphost. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I paste the schema, the checker, and a redacted alert sample, then I ask for missing observe commands that still fit the allowlist. I do not paste kubeconfigs, customer payloads, or unfreeze signatures. If the model invents a rollout restart, the checker fails, and that failure is the lesson. The free server is for running check_first_commands.py, not for talking to prod.

# labeled as a proposal I would run on a scratch server, not on prod
python3 tools/check_first_commands.py runbooks/payments-api.yml
python3 -c "import yaml,sys; yaml.safe_load(open(sys.argv[1]))" runbooks/payments-api.yml

Would I let the same session open a tunnel to the cluster? No. The runbook is the contract; the assistant is a typist with a linter.

A dry-run I actually practice

I rehearse with a fake page so the first real page is not also the first parse of the YAML. The drill is short, scripted, and mean about extra commands.

export INCIDENT=inc-drill-2026-09-17
export ALERT_CLASS=PAYMENTS_P99_LATENCY
python3 tools/check_first_commands.py runbooks/payments-api.yml
# observe only — replace with your read-only kube context
kubectl --context scratch -n payments get deploy payments-api -o wide
kubectl --context scratch -n payments top pod -l app=payments-api
# stop here unless tools/unfreeze.sh succeeded on this same hash

If the drill needs a command that is not in observe_commands, I update the YAML in git and rehash. I do not "just this once" extend the path in the terminal. That exception is how freeze gates die.

Limitations, and who should skip this

This workflow assumes you can map alerts to ids, keep argv allowlists, and block writes in the default path. It will feel heavy if you are a solo hobby project with no pager, or if your platform cannot distinguish read kubectl from write kubectl. It also will not save you if the runbook hash is computed after the assistant edits the file. I am not claiming latency numbers, model rankings, or token budgets here, because those claims go stale and they are not the point.

Do not use this approach to launder a generated production change behind a decorative freeze file. Do not store signatures, kubeconfigs, or customer data on a shared demo server. Do not skip escalation because the model sounded calm. If you cannot name the blast radius in one sentence, the hash check should keep failing.

The core conclusion does not change when the models get more fluent. Encode the page as alert class, first command, escalation clock, and freeze hash, then make unfreeze a signed break of that hash. Everything else is optional commentary. If you want a scratch box to lint the YAML and the allowlist, MonkeyCode's free server option is enough to practice the checks; it is not an on-call seat.

Copilot Patches Stay Read-Only Until You Sign the Unfreeze

Emery Huang — Wed, 16 Sep 2026 04:04:15 +0000

An AI copilot can draft a remediation in seconds, but it cannot own production blast radius. I still freeze writes until the alert class, first-read commands, and an escalation owner are named. If those three are missing, the patch stays in a scratch shell, not on the cluster. Would you really unfreeze kube-system just because a chat window sounded confident at 03:12?

The wrong debate for a page

Feeds keep asking whether models already outcode most working engineers on ordinary tickets. That question is interesting at lunch, and it is honestly dangerous on a live pager. On-call is not a coding contest; it is a control problem with a clock and a blast radius. A fluent patch that restarts the wrong Deployment is still an incident amplifier, not a win.

I treat every model suggestion as an untrusted diff sitting against a frozen write path. The runbook below is a proposal I keep in the repo, not a story about a specific outage. If your team already pastes chat output into kubectl, this is the gate I wish sat in front of that habit.

What I page on, and what I refuse to page on

I only auto-page when the alert already carries a stable fingerprint, a service owner, and a read-only first command. Everything else can wait for a ticket, because a vague prompt is not an incident. Ask yourself: if the alert cannot name the workload, how can a model name a safe write?

Payload contract

Minimum fields I want on the page:

alertname plus a fingerprint hash, not a prose summary
namespace, workload_kind, and workload_name when Kubernetes is involved
owner as a team alias that actually answers
first_command that is strictly read-only
write_allowed defaulting to false until unfreeze

# proposal: pager payload contract (not executed against a live cluster)
alertname: KubeDeploymentReplicasUnavailable
fingerprint: "a3f1c9e2"
namespace: payments
workload_kind: Deployment
workload_name: checkout-api
owner: payments-oncall
severity: page
write_allowed: false
first_command: >
  kubectl -n payments get deploy checkout-api -o jsonpath='{.status.conditions}'

If any of those keys are empty, I do not run writes, and I do not ask a model for a restart. I escalate with the incomplete payload, because guessing owners at 3 a.m. is how you page the wrong people. Can a copilot invent an owner alias that paging actually honors? It cannot, and I will not pretend otherwise.

First commands are reads, and they are boring on purpose

My first five minutes are observe-only, even when the copilot is already waving a Helm rollback. I want evidence that matches the fingerprint, not a soothing narrative that matches my anxiety. The commands below are the ones I type before anyone on the call talks about unfreeze.

The boring bundle

# proposal: observe-only bundle, labeled unexecuted
export NS=payments APP=checkout-api KIND=deploy NAME=checkout-api
date -u
echo "fingerprint=${FINGERPRINT:-missing}"

kubectl -n "$NS" get deploy,sts,po,ep -l app="$APP" -o wide
kubectl -n "$NS" describe "$KIND" "$NAME" | sed -n '1,80p'
kubectl -n "$NS" logs "deploy/$NAME" --tail=100 --timestamps | tail -n 40

# still read-only: compare live object to last applied
kubectl -n "$NS" get "$KIND" "$NAME" -o yaml > "/tmp/${NAME}.live.yaml"

Notice there is no rollout restart, no scale, and no force delete in that first bundle. If the model prints those, I copy them into a scratch file named candidate.sh, and I leave production frozen. Why would a first command need a write if we still have not proved the alert is real?

Escalation is a lane, not a vibe

I escalate when the fingerprint is real and the owner is not me, or when the read-only evidence disagrees with the model's story. Restarting just to see is more expensive than a two-minute handoff with a complete note. The three questions I send in the escalate note stay small on purpose, and they travel well.

The note I actually send

Which fingerprint and workload did we actually observe?
Which read-only commands already ran, with paste of the output hashes?
What write is proposed, and what blast radius did we name?

# proposal: escalate note
fingerprint: a3f1c9e2
observed: Deployment/checkout-api ns=payments replicas 1/3
reads_run: get,describe,logs (see incident/2026-09-16/reads.log)
proposed_write: kubectl -n payments rollout undo deploy/checkout-api
blast_radius: checkout-api only; payments-worker not in selector
owner_needed: payments-oncall
unfreeze: blocked

If the next engineer cannot answer those, we stay frozen through the next scheduled check-in. A model that cannot fill this note should not get a shell on any cluster. Does that slow you down on a scary page, and should it, when a fast write can still take the site down?

The freeze and unfreeze rule for copilot patches

Here is the rule I want on a sticky note above the keyboard during every page. Production writes stay frozen until four signatures exist together in one unfreeze file on disk. Missing any signature means the copilot patch remains a candidate, even if the diff looks brilliant.

Four signatures

Unfreeze requires all of the following:

Alert fingerprint matches the live object you just read
Owner alias is a human who accepted the page
Blast radius is a named workload list, not "the cluster"
Candidate commands were dry-run in a scratch environment, not on prod

# proposal: unfreeze gate
export INCIDENT_ID=20260916-checkout
export UNFREEZE_FILE="./incident/${INCIDENT_ID}/unfreeze.json"

python3 - <<'PY'
import json, os, sys, pathlib
p = pathlib.Path(os.environ["UNFREEZE_FILE"])
data = json.loads(p.read_text())
required = ["fingerprint", "owner", "blast_radius", "dry_run_ok"]
missing = [k for k in required if not data.get(k)]
if missing or data.get("write_allowed") is not True:
    print("FROZEN: missing", missing or ["write_allowed"])
    sys.exit(2)
print("UNFREEZE ok for", data["blast_radius"])
PY

Example unfreeze.json I will not apply until the checker exits zero:

{
  "fingerprint": "a3f1c9e2",
  "owner": "payments-oncall",
  "blast_radius": ["deploy/checkout-api"],
  "dry_run_ok": true,
  "write_allowed": false
}

I do not source the candidate script until that checker exits zero on the unfreeze file. If you skip the file, you are not using a runbook; you are gambling with autocomplete. Would I make an exception for a Sev-1? Only with a named owner and a named blast radius.

Dry-run the model where it cannot page you twice

I want the copilot to argue with a replica of the failing object, not with production kube-apiserver. A scratch box, a kind cluster, or a disposable VM is enough for most Deployment pages. The goal is simple: does the proposed command even parse, and does it target the named blast radius?

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode, an open source project, only as a scratch place with free model access and a free server option. That keeps the dry run off production credentials, which matters more than chat convenience during a page. I am not claiming a particular model name, a token ceiling, or a hardware spec, because those change and I will not invent them. If you need a disposable box for that dry run, the free server option is enough to keep the experiment off prod.

Static allowlist

# proposal: keep the model off prod kubeconfig
export KUBECONFIG="$HOME/scratch/kind-incident.kubeconfig"
export INCIDENT_ID="20260916-checkout"

# reject any line that is not in a tiny allowlist
grep -E '^(kubectl |helm |echo |#)' candidate.sh >/dev/null

python3 ./tools/assert_blast_radius.py \
  --script candidate.sh \
  --allow namespace=payments \
  --allow name=checkout-api \
  --forbid kube-system \
  --forbid default

# only after static checks: server-side dry-run on the scratch cluster
kubectl -n payments apply --dry-run=server --validate=true -f candidate.yaml
kubectl -n payments diff -f candidate.yaml

# tools/assert_blast_radius.py — proposal checker, unexecuted against prod
import argparse, sys

def tokens(line: str):
    return line.split()

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--script", required=True)
    p.add_argument("--allow", action="append", default=[])
    p.add_argument("--forbid", action="append", default=[])
    args = p.parse_args()
    allow = dict(a.split("=", 1) for a in args.allow)
    text = open(args.script, encoding="utf-8").read().splitlines()
    for i, line in enumerate(text, 1):
        s = line.strip()
        if not s or s.startswith("#"):
            continue
        parts = tokens(s)
        if s.startswith("kubectl") and "-n" in parts:
            ns = parts[parts.index("-n") + 1]
            if ns in args.forbid or ns != allow.get("namespace"):
                print(f"line {i}: namespace {ns} not in blast radius")
                sys.exit(1)
        if any(bad in s for bad in ("--force", "delete ns", "drain ")):
            print(f"line {i}: forbidden verb")
            sys.exit(1)
    print("dry-run static checks passed")

if __name__ == "__main__":
    main()

If that checker fails, I do not unfreeze, and I do not negotiate with the model. I either rewrite the candidate by hand or I escalate with the failing line number. A free scratch server is useful here because I can burn the VM after the page, secrets and all.

Decision table I keep next to the runbook

The matrix below is what I actually glance at when the copilot starts sounding sure. It is deliberately blunt, because a clever exception is how frozen writes become thawed writes. Please read it once before you open a chat window on an active production page.

Signal	First action	Copilot allowed?	Unfreeze?
Fingerprint missing	Reject page / convert to ticket	No	No
Fingerprint matches, reads clean	Capture logs, wait	Comments only	No
Reads confirm bad rollout	Draft undo in scratch	Yes, files only	After signatures
Owner unreachable	Escalate, stay frozen	No writes	No
Candidate touches extra NS	Delete candidate.sh	No	No
Checker exit 0 + owner ack	Run named write once	No further prompts	Yes, time-boxed

The table is the article, if I am honest with the rest of the on-call rotation. Models are optional during a page, and they stay optional after the unfreeze file exists. The freeze file is not optional, and I will argue that in the incident review.

Limitations, and who should not copy this

When I throw the runbook away

This workflow assumes you can freeze writes without making the outage worse, which is not true for every class of alert. Data corruption, certificate expiry in the next few minutes, and disk-full nodes sometimes need a human-owned write with no model in the loop. I also assume you will not paste secrets, kubeconfigs, or customer payloads into any chat, free or otherwise.

Do not use this approach if you lack RBAC that can actually deny your own user. Skip it as well if your so-called scratch cluster still shares any credentials with production. Do not use it as an excuse to skip the owner, either, when the page is loud. A free server does not become a production control plane just because the model answered quickly.

I am not publishing latency numbers, token quotas, or win rates, because I did not measure them for this piece. The only artifact I am defending here is the unfreeze gate, not a vendor benchmark. If your incident commander wants a different freeze policy, follow that policy and keep the checker as a pre-commit hook on runbook PRs.

When the page is quiet again, I file the unfreeze file next to the read logs. That packet is what I want in the review, not a screenshot of a confident chat. Keep the model in the scratch lane, and keep production behind a signature you can explain.

Reproduce the Pager on a Scratch Box Before You Unfreeze Prod

Emery Huang — Sun, 13 Sep 2026 20:08:00 +0000

I freeze production writes the moment the pager fires, and I keep that freeze until a scratch box reproduces the same alert fingerprint. Model-suggested patches do not get a live shell. They get a replica, a failing test, and a human ACK. If the replica cannot fail the same way, I escalate with evidence instead of guessing on prod. Does that feel slower than pasting kubectl at three in the morning?

The live shell is the wrong first destination

When the pager lights up, I want a command that feels like progress, not another pretty graph. I have watched myself paste a confident one-liner into production because a coding model sounded sure of itself. Have you done that, then spent the next hour explaining a second outage to people who were asleep? The postmortem always starts the same way: we treated a suggestion as a runbook.

The useful question is whether I can make this alert fail on a box that is allowed to break. Until that replica exists, production stays read-only for me, including shells that look harmless. The freeze is not a mood I announce in chat; it is a lock file, a fingerprint, and a named unfreeze rule. If I cannot point at those three objects, I am still in observe-only, and that is fine.

An “agent” sitting beside the pager is usually a loop with better copy. Loops are good at drafting probes. Loops are not on-call authority, and I will not pretend otherwise during a SEV.

The replica-first contract

I keep four objects in the incident channel, and I refuse writes until all four exist:

alert_id plus a fingerprint hash of the symptom, never the essay the model wrote
a freeze record that names who holds the write lock
a scratch host that is allowed to die without a customer notice
a failing probe that matches the fingerprint on that host

If any object is missing, the only legal production commands are observe-only. Want a patch? Build it against the replica until the probe goes green. Want to unfreeze? Show me that red-to-green transition, then get a human ACK. Why would I skip the replica just because the prompt included the word urgent?

Freeze and unfreeze rule

Freeze trigger: first page of a mutating class (latency, error-budget, data-loss, authz), or any model-suggested shell that is not a read.
Allowed under freeze: logs, metrics, traces, kubectl get, unlocked SELECT, and copying artifacts onto the scratch box.
Forbidden under freeze: restarts, scale, config apply, schema changes, feature flags, secret rotation, and anything that changes blast radius.
Unfreeze: a human types UNFREEZE <alert_id> <replica_test_id> <ack_name> after the replica matches and the probe is green. No bot may emit that token.

Is the model allowed to draft the unfreeze line for me? Yes. Can it send that token into the lock path? No, and I will not wire that up.

First commands I actually run

I keep a wrapper so the first ten minutes look the same every night. The wrapper refuses to continue when freeze.lock is missing, which surprises people who wanted to “just look around with apply.”

#!/usr/bin/env bash
# replica-gate.sh — labeled example: observe-only on prod while freeze.lock exists
set -euo pipefail
ALERT_ID="${1:?alert_id}"
LOCK="${LOCK:-/var/oncall/freeze.lock}"
REPLICA_HOST="${REPLICA_HOST:?set REPLICA_HOST to a scratch box}"

fingerprint() {
  printf '%s' "$ALERT_ID" | sha256sum | awk '{print $1}'
}

if [[ ! -f "$LOCK" ]]; then
  echo "no freeze.lock; refusing to proceed without an explicit freeze" >&2
  exit 2
fi

if ! grep -q "writes=forbidden" "$LOCK"; then
  echo "freeze.lock does not forbid writes; fix the lock before any shell" >&2
  exit 3
fi

FP="$(fingerprint)"
HOLDER="$(awk -F= '/^holder=/{print $2}' "$LOCK")"
echo "alert_id=$ALERT_ID fingerprint=$FP freeze_holder=$HOLDER"

# legal first commands: reads only
kubectl --context prod get deploy,po,endpoints -A | head
# ship the symptom pack to the replica, never a write to prod
ssh "$REPLICA_HOST" "mkdir -p /incidents/$ALERT_ID && printf '%s\\n' '$FP' > /incidents/$ALERT_ID/fingerprint"

The lock file is equally boring, which is the point. I want something a tired person can grep without interpreting a novel.

# /var/oncall/freeze.lock  (example)
alert_id=ALT-1842
holder=emery
class=latency
frozen_at=2026-09-14T02:11:08Z
writes=forbidden
unfreeze_requires=replica_match,human_ack

If you cannot name the replica host next to that file, you are not ready to unfreeze. Why would a model know your blast radius if you have not even named the box that is allowed to break?

Unfreeze is a separate script, not a second thought in the same shell history.

#!/usr/bin/env bash
# unfreeze.sh — labeled example: humans only, replica test id required
set -euo pipefail
ALERT_ID="${1:?alert_id}"
TEST_ID="${2:?replica_test_id}"
ACK_NAME="${3:?ack_name}"
LOCK="${LOCK:-/var/oncall/freeze.lock}"
PROBE_LOG="${PROBE_LOG:-/incidents/$ALERT_ID/probe.log}"

if [[ ! -f "$LOCK" ]]; then
  echo "nothing frozen" >&2
  exit 1
fi
if [[ "$ACK_NAME" == "model" || "$ACK_NAME" == "bot" ]]; then
  echo "bots cannot unfreeze" >&2
  exit 2
fi
if ! grep -q "PASS $TEST_ID" "$PROBE_LOG" 2>/dev/null; then
  echo "replica test $TEST_ID is not green; stay frozen" >&2
  exit 3
fi

printf 'UNFREEZE %s %s %s\n' "$ALERT_ID" "$TEST_ID" "$ACK_NAME" | tee -a /var/oncall/unfreeze.log
mv "$LOCK" "$LOCK.released.$(date -u +%Y%m%dT%H%M%SZ)"

Scratch lane: draft the probe, do not draft prod

I need a place that can run the failing probe without sharing a kernel with customers. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode's free model access and free server option as that scratch lane: the model drafts the probe, the free server runs it, and production stays frozen.

I do not treat those free options as capacity planning, an SLO, or a benchmark. I treat them as a disposable workbench. If the workbench is slow or missing, I still have the freeze. The incident does not earn mutate rights on prod because the scratch lane hiccuped.

A session on the scratch box gets a tight prompt. Notice the model only sees the fingerprint and a redacted log slice, not cluster-admin.

# inputs I paste into the coding session (redacted example)
alert_id: ALT-1842
fingerprint: 9c1e...
symptom: p99 checkout > 2s, error rate 4% on /pay
forbidden: kubectl apply, helm, terraform, DROP, restart
required_artifact: a probe that exits 1 when the fingerprint matches

#!/usr/bin/env bash
# probes/pay_p99.sh — labeled example, not a production measurement
set -euo pipefail
URL="${1:?base url of the replica app}"
python3 - "$URL" <<'PY'
import sys, time, urllib.request
url = sys.argv[1].rstrip("/") + "/pay"
lat, err = [], 0
for _ in range(30):
    t = time.time()
    try:
        urllib.request.urlopen(url, timeout=3)
        lat.append((time.time() - t) * 1000)
    except Exception:
        err += 1
        lat.append(3000.0)
lat.sort()
p99 = lat[int(0.99 * (len(lat) - 1))]
# fingerprint match: high latency or errors on the replica only
sys.exit(0 if p99 < 2000 and err == 0 else 1)
PY

When that probe exits 1 on the replica, I have a contract instead of a vibe. The model may propose a patch against the replica checkout. I apply the patch on the scratch box only. I rerun the probe. Green on replica is necessary. It is not sufficient. A human still has to unfreeze, using the script above, with a real name.

Escalation that prefers evidence over heroics

If the replica cannot reproduce in fifteen minutes, I do not “just try it on prod to see.” I escalate with a structured note. Reproduction failure is a first-class outcome, not a personal failure, and I say that in the channel so nobody improvises a restart.

Escalation note I paste:

alert_id and fingerprint
freeze holder and freeze start time
replica host and probe path
what matched (logs, metrics) and what did not (cannot trip the probe)
the ask: owner of the service, not a more aggressive command

Who gets the page next? The service owner named in the catalog, not the model, not the loudest person in Slack. If the catalog has no owner, I stay frozen and I say that out loud. Unowned services do not get emergency writes from me, even when the suggested patch looks tiny.

Decision table I keep above the fold

Signal	Replica result	Production action	Next human step
Pager, mutating class	Not started	Freeze writes, observe-only	Name replica host
Probe cannot fail	No match	Stay frozen	Escalate “cannot reproduce”
Probe fails like pager	Match	Stay frozen, patch replica	Review diff, rerun probe
Probe green on replica	Match then green	Still frozen	Human `UNFREEZE` with names
Unfreeze token present	Green	Apply the same diff in a change window	Watch fingerprint for 15 min
Fingerprint returns	Green then red	Re-freeze immediately	Escalate, do not loop patches

Would I let a coding loop walk that table by itself? Not on my pager. The table is the control plane. The model is a clerk that fills in the probe column.

A twenty-minute drill you can run tomorrow

This is a labeled drill, not a war story with fake graphs, and it should feel slightly annoying.

Pick a non-prod service and create a synthetic alert class drill-latency.
Write freeze.lock with your name and the drill alert_id.
Point REPLICA_HOST at a throwaway VM or any scratch server you already control.
Ask a coding model for a probe that exits 1 on the synthetic symptom only.
Break the replica on purpose, confirm the probe is red, patch the replica, confirm green.
Practice the unfreeze line out loud. Then delete the lock. Do not touch prod.

If step 4 wanders into kubectl apply -f for the live cluster, the drill already paid for itself. You found the hole before the pager did. Can you finish the drill without once opening a write to production? That is the only pass condition I care about.

Limitations, and who should skip this

This runbook assumes you can isolate a scratch box and that alert text is allowed to leave the production network. If you are in a sealed environment, keep the replica inside the same trust boundary and do not paste dumps into any external model. If the outage is the data itself, reproduction can be harmful; freeze and escalate instead of cloning customer records onto a free box.

Free model access and a free server are convenience, not an incident SLO. I do not know their quotas, hardware, or uptime, and I will not invent numbers to sound current. If your company already has a staging cluster, use that and keep the same freeze file. If you cannot freeze writes because your platform has no lock, this article will not save you; build the lock first.

Do not use this approach when the page is a simple disk-full on a single node with a documented runbook. Do not use it to delay a well-rehearsed failover. The replica gate is for the messy middle: model-drafted remediations, unclear blast radius, and the urge to type something heroic before anyone has a failing test.

What I want next time I am paged

I want the first artifact to be a red probe on a box that can die, not a green dashboard after I rolled the dice. I want the unfreeze line to include a human name that will still answer email tomorrow. I want the model in the room as a drafter of tests, not as an operator with root.

Steal the freeze file and the decision table. Run the twenty-minute drill on a service nobody will miss. Keep the model off the live shell until a human says UNFREEZE.

Print the Command Plan, Then Freeze Writes Until Blast Radius Is Named

Emery Huang — Sat, 12 Sep 2026 18:16:29 +0000

An on-call page is not a conversation, and it is not a license to improvise a restart. I want every alert to compile into a frozen command plan before anyone opens a shell. The plan names the read-only first commands, the escalate clock, and the freeze latch that blocks writes. If a coding assistant cannot live inside that latch, it does not belong on the pager.

Why chat-shaped runbooks fail at 3 a.m.

Most on-call wikis still read like essays, and most assistants still emit paragraphs that look dangerously executable. Have you ever watched a tired human paste a generated kubectl line because the graph was red? I stop trusting prose as the pager interface when the only artifact is a chat transcript. The page should become a small graph of allowed argv, not another prompt waiting for courage.

A useful plan has three properties that a chat window never actually guarantees at three in the morning. First, every node is either a read, an escalate signal, or a write sitting behind a latch. Second, writes carry a blast-radius field that a human must type, and a model must never fill. Third, the argv is hashed so a small assistant edit cannot sneak into the production shell.

I am not claiming this compiler replaces your paging vendor, your chat bridge, or your existing service catalog. I am proposing a boring gate that sits between alert JSON and a terminal, then refuses anything it cannot name. If that sounds rigid, ask what “flexibility” meant the last time somebody bounced the wrong Deployment because a paragraph looked confident.

Compile the page, do not narrate it

Treat the incoming alert as source code that either type-checks or does not run. If required fields are missing, the compiler fails closed and prints only a read of the raw payload. If the alert class is known, it emits a plan file and refuses any command whose digest is not listed. That is the entire trick, and I want it boring enough that a new on-call engineer can follow it without a pep talk.

Here is the on-disk shape I want committed next to the service, not pasted into a wiki comment at incident time.

# proposed example — unexecuted, not production config
alert_class: checkout_latency
observe_seconds: 180
escalate_after_seconds: 600
read_lane:
  - id: ledger
    argv: ["python3", "incident_ledger.py", "append", "--alert-id", "${ALERT_ID}"]
  - id: p95
    argv: ["python3", "read_slo.py", "--service", "checkout", "--window", "15m"]
  - id: owners
    argv: ["python3", "owners.py", "--service", "checkout"]
write_lane:
  - id: shed_traffic
    argv: ["python3", "shed.py", "--service", "checkout", "--percent", "10"]
    blast_radius: "checkout, 10 percent synthetic and canary, 5 minutes"
freeze: locked

Notice the first command is not a restart, a rollback, or a cluster-wide scale event of any kind. It is an append-only ledger write that lives on a path you already control, so the next person on the bridge can see what already ran. Why start there, instead of another dashboard screenshot that dies in scrollback? Because a ledger survives the chat window, and a screenshot does not explain argv.

Artifact: a local latch that hashes argv

The Python below is a proposed local gate, not a battle-tested control plane, and it does not call exec on purpose. It compiles the YAML, prints the read lane, and blocks writes until a human unfreezes with a blast-radius string that must match the plan. Run it on a laptop or a jump box you already trust, and keep production credentials out of its environment until a dry-run review says otherwise.

#!/usr/bin/env python3
"""proposed_oncall_latch.py — example compiler, not a production pager."""
from __future__ import annotations

import hashlib
import json
import shlex
import sys
import time
from pathlib import Path

import yaml  # proposed example dependency

LEDGER = Path("/tmp/incident_ledger.jsonl")
STATE = Path("/tmp/incident_latch_state.json")


def digest(argv: list[str]) -> str:
    blob = "\0".join(argv).encode("utf-8")
    return hashlib.sha256(blob).hexdigest()[:16]


def load_plan(path: Path) -> dict:
    plan = yaml.safe_load(path.read_text())
    required = ("alert_class", "read_lane", "write_lane", "escalate_after_seconds")
    missing = [key for key in required if key not in plan]
    if missing:
        raise SystemExit(f"plan missing fields: {missing}")
    return plan


def append_ledger(event: dict) -> None:
    event = {**event, "ts": time.time()}
    with LEDGER.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(event) + "\n")


def save_state(state: dict) -> None:
    STATE.write_text(json.dumps(state, indent=2))


def load_state() -> dict:
    if not STATE.exists():
        return {"freeze": "locked", "started_at": time.time(), "ran": []}
    return json.loads(STATE.read_text())


def print_plan(plan: dict) -> None:
    print(f"class={plan['alert_class']} freeze={load_state()['freeze']}")
    print("read_lane:")
    for step in plan["read_lane"]:
        print(f"  [{digest(step['argv'])}] {shlex.join(step['argv'])}")
    print("write_lane (blocked while freeze=locked):")
    for step in plan["write_lane"]:
        print(f"  [{digest(step['argv'])}] {shlex.join(step['argv'])}")
        print(f"    blast_radius: {step['blast_radius']}")


def run_step(plan: dict, step_id: str, freeze_token: str | None) -> None:
    state = load_state()
    lanes = {step["id"]: step for step in plan["read_lane"] + plan["write_lane"]}
    if step_id not in lanes:
        raise SystemExit("unknown step id; I will not guess a command")
    step = lanes[step_id]
    is_write = step_id in {item["id"] for item in plan["write_lane"]}
    if is_write and state["freeze"] != "open":
        raise SystemExit("writes are frozen; unfreeze with a matching blast-radius string")
    if is_write and freeze_token != step["blast_radius"]:
        raise SystemExit("blast radius did not match the plan; still frozen")
    append_ledger({"step": step_id, "argv": step["argv"], "digest": digest(step["argv"])})
    print("LEDGER_OK", shlex.join(step["argv"]))
    print("This example does not exec(). Wire subprocess only after a dry-run review.")
    state["ran"].append(step_id)
    save_state(state)


def unfreeze(reason: str) -> None:
    if len(reason.strip()) < 12:
        raise SystemExit("unfreeze reason is too short to be a blast-radius statement")
    state = load_state()
    state["freeze"] = "open"
    state["unfreeze_reason"] = reason
    save_state(state)
    append_ledger({"event": "unfreeze", "reason": reason})
    print("latch open; writes still must match hashed argv")


def maybe_escalate(plan: dict) -> None:
    state = load_state()
    elapsed = time.time() - state["started_at"]
    if elapsed >= plan["escalate_after_seconds"] and "escalated" not in state:
        print("ESCALATE: clock exceeded; page the service owner, do not restart yet")
        append_ledger({"event": "escalate", "elapsed": elapsed})
        state["escalated"] = True
        save_state(state)


def main(argv: list[str]) -> None:
    if len(argv) < 3:
        raise SystemExit("usage: proposed_oncall_latch.py PLAN.yaml print|run|unfreeze|clock [args]")
    plan = load_plan(Path(argv[1]))
    cmd = argv[2]
    if cmd == "print":
        print_plan(plan)
    elif cmd == "run":
        token = argv[4] if len(argv) > 4 else None
        run_step(plan, argv[3], token)
    elif cmd == "unfreeze":
        unfreeze(" ".join(argv[3:]))
    elif cmd == "clock":
        maybe_escalate(plan)
    else:
        raise SystemExit("unknown verb")


if __name__ == "__main__":
    main(sys.argv)

Would I ship this file tomorrow as the global pager runtime for every service in a company? I would not, because a /tmp ledger is a teaching artifact and not an audit store. I would keep subprocess disabled until the YAML lives in the same repo as the service, and until a teammate can reject a plan in review the same way they reject a bad migration. The point of the artifact is the gate: print, hash, freeze, escalate, then maybe write.

A dry-run you can type without touching prod

Use the verbs in this order, and stop at the first unexpected exit. The failing write is the success case, because it proves the latch still holds.

# proposed dry-run only — do not point this at production kube contexts
python3 proposed_oncall_latch.py checkout.yaml print
python3 proposed_oncall_latch.py checkout.yaml run ledger
python3 proposed_oncall_latch.py checkout.yaml run p95
python3 proposed_oncall_latch.py checkout.yaml clock
# expected: writes are frozen
python3 proposed_oncall_latch.py checkout.yaml run shed_traffic
python3 proposed_oncall_latch.py checkout.yaml unfreeze \
  "checkout, 10 percent synthetic and canary, 5 minutes"
# still requires the blast-radius string as the extra argument
python3 proposed_oncall_latch.py checkout.yaml run shed_traffic \
  "checkout, 10 percent synthetic and canary, 5 minutes"

If print already looks wrong, you do not have an incident yet; you have a bad plan file. Fix the YAML in daylight, not while the error budget is burning.

First commands I want printed, in order

When the page fires, I want the human to run three reads and nothing that mutates customer traffic. Can a model suggest extra cluster soup in the sidebar while those reads run? It can talk, but that soup does not get a digest, so the latch should refuse it. The allowed first commands are the ones already compiled, and they should be dull.

Append the alert identifier, the raw class, and the page time to the local ledger so the bridge does not depend on screenshot memory.
Read the SLO window that the alert claims was breached, using the service name from the plan rather than a random dashboard you like.
Print the service owner and the escalate path, even when you are sure you already know who owns checkout this week.
Look at the clock next, and if you are still inside the observe window, leave the freeze latch locked on purpose.

After those steps, restart is still a write, so it stays behind the latch with every other mutation. If that feels slow, ask whether the last surprise bounce was actually faster once you counted the second incident it created.

Escalation is a command, not a feeling

People escalate when they feel scared, or they never escalate because a restart looks like courage on a status page. I want the compiler to print ESCALATE when the clock says so, and I want that line in the ledger where later readers can find it. The human can still call someone earlier, and that is fine. The bot cannot suppress the clock because a generated paragraph said “looks like GC, just bounce it.”

A short decision table keeps that argument out of the bridge chat, where feelings usually win.

Clock	Freeze	Allowed next step
t < observe_seconds	locked	`read_lane` only
observe_seconds ≤ t < escalate_after_seconds	locked	`read_lane`, draft the unfreeze sentence, still no writes
t ≥ escalate_after_seconds	locked	page the owner, record `ESCALATE`, still no writes
any t	open	`write_lane` whose digest and blast_radius both match

If your team cannot fill that table for an alert class, the class is not ready for automation of any kind. Leave it as a human page, and spend the afternoon naming reads before you name writes. Have you noticed how many “AI on-call agents” skip this table and jump straight to a shell? That skip is the incident.

The freeze / unfreeze rule on a sticky note

Here is the rule I want on paper next to the keyboard, not buried in a tool’s system prompt. Writes stay frozen until a human types the blast-radius sentence that already lives in the plan. The sentence is not a vibe, and it is not lgtm, and it is not a thumbs-up emoji from the bridge. It names the service, the percentage or the blast, and the time box.

Unfreeze does not run the write. Unfreeze only opens the latch, which is a different verb on purpose. The write still has to match the hashed argv, so a rewritten flag is a different command and the gate stays closed. That double door is the difference between a runbook and a chatbot that happens to know kubectl.

If the assistant rewrites a flag, the digest changes, and the latch must refuse the new argv.
If the human cannot name blast radius in the plan’s own words, the latch stays closed and the clock keeps running.
If the alert class has no write_lane at all, there is nothing to unfreeze, and that absence is a feature you should keep.
If someone SSHs around the gate, the ledger will not save you; this compiler is a seatbelt, not a lock on the building.

Where a coding assistant is allowed to sit

I still want help drafting YAML comments and reviewing whether a new alert class forgot a ledger step before the plan is merged. I do not want help emitting production argv from a live page, and I do not want an assistant that can unfreeze itself. That split is the whole workflow: models annotate the plan in daylight, humans hold the latch at night.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

When I am editing the plan offline, MonkeyCode’s free model access and free server option are enough to iterate on the compiler and the YAML without borrowing a production box. The assistant can propose a new read step in a comment, or it can complain that blast_radius is too vague to type under stress. It cannot obtain an unfreeze token, and it cannot skip the digest check, which is exactly why I am willing to put it next to a runbook at all. If you already have a scratch machine, keep using it; the latch does not require that product, and the method still works when every MonkeyCode mention is deleted.

Limitations, and who should not use this

This compiler does not talk to your paging vendor, your cluster, or your secret store, and it will not magically complete an alert that arrived half-empty. It will not stop a human who SSHs around the gate with a personal kubecontext. It will not detect a bad YAML that names the wrong service inside blast_radius while keeping a pretty digest. Garbage in still means fail closed, and fail closed still means you read the raw payload instead of guessing a write.

Do not use this approach if you are already inside an active incident and you have no plan file checked in yet. Do not use it to grant a model production credentials, even on a “temporary” jump host. Do not use it as a compliance control, because a JSON file under /tmp is not an audit system and I will not pretend otherwise. Do not use it if your writes are not enumerable in advance; a free-form “fix it” agent cannot be hashed, and an unhashable write does not belong on this pager.

I would also skip this pattern if the team cannot agree on observe and escalate clocks during a calm review. A latch without a clock becomes another wiki page that nobody opens, and a clock without a latch becomes a restart contest. If both arguments are still open, you need a conversation, not a Python file.

Keep the runbook smaller than the chat window

Print the plan. Hash the argv. Keep writes frozen until a human names blast radius in the plan’s own sentence. Escalate on a clock, not on a generated paragraph that sounds sure of itself. That is the on-call runbook I want sitting beside the pager, and it is intentionally smaller than the assistant window that will try to eat it.

If you try the example, run print and clock long before you ever wire subprocess.run. The first successful night is the one where the ledger shows three reads, one escalate, and a phone call, not a surprise restart that nobody can replay.

Your Pager Should Resolve IDs, Not Prompts

Emery Huang — Fri, 11 Sep 2026 15:49:31 +0000

Your pager should resolve a catalog ID, not a prompt, and that is the whole runbook. If the alert class has no pinned command, you do not improvise inside a production shell. You escalate with the catalog hash, and you keep the mutating path closed until a human pins again. Drafts belong on a laptop at a desk, not in a root shell while error budgets burn.

Why does this keep coming up? Agent-style loops are fashionable, and they look helpful until they invent kubectl flags from a guess. Are you really going to let a completion model choose selectors while customers are paging you? I am not, and I do not want the next on-call to do it either.

The failure I want to make expensive

A typical 3 a.m. path still looks like this, even in teams that own serious monitors. The alert fires, a chat window opens, and someone pastes logs into a model because the wiki is stale. The model returns a confident command, and a tired human runs it because reading diffs feels slower than hoping. Then the command is slightly wrong, and now you have two incidents sharing one pager.

Does that sound like engineering, or like gambling with a shell you already authenticated? I want improvisation to be the expensive action, not escalation to a human. Desk time is for drafting and pinning. Pager time is for executing something you already signed, then stopping.

This article proposes a compile step you can actually rehearse. It is a catalog, a tiny resolver, a freeze bit, and a small escalation packet. Treat the code as a worked example, not as a claim about any production fleet I have not named.

Compile, do not converse

Think of the runbook as a compiler with four inputs you can name out loud. You will recognize them if you have ever shipped a binary you trusted more than a scratch script sitting in /tmp.

Alert class — a stable name from the monitor, not a novel sentence.
Fingerprint — a short hash of the resource identity, so retries do not look new.
Catalog version — the git tag or sha of the pinned command file.
Freeze bit — a lock that means this catalog is untrusted until a human pins again.

The compiler output is either a command ID or an escalation. There is no third path called "just this once, I know this box." If you still need that third path, you are prompting production and calling it judgment.

What a first command is allowed to be

A first command is a pinned, named operation with a dry-run twin sitting beside it. It has a timeout, a mutate flag, and an argv list a human can read without decoding a transcript. It does not take a freeform string from a model at page time. It does not grow extra flags because the completion sounded sure.

Here is a catalog you can copy into catalog.yaml and edit at a desk. Label it a proposal until a human pins the version in git.

# proposal: example catalog — pin this in git, never in a chat log
version: "2026.09.11-1"
freeze: false
commands:
  - id: redis-conn-sat-observe
    alert_class: redis.connections.saturated
    mutate: false
    timeout_sec: 30
    argv: ["redis-cli", "-h", "${TARGET_HOST}", "INFO", "clients"]
    dry_run_argv: ["echo", "observe", "redis-cli", "INFO", "clients"]
  - id: redis-conn-sat-reload-proxy
    alert_class: redis.connections.saturated
    mutate: true
    requires_observe_id: redis-conn-sat-observe
    timeout_sec: 45
    argv: ["systemctl", "reload", "redis-proxy"]
    dry_run_argv: ["systemctl", "show", "redis-proxy", "--property=MainPID"]
  - id: disk-fill-observe
    alert_class: node.disk.fill
    mutate: false
    timeout_sec: 20
    argv: ["df", "-h", "${MOUNT}"]
    dry_run_argv: ["echo", "df", "-h", "${MOUNT}"]
escalation:
  when_no_id: page-secondary
  when_frozen: page-secondary
  when_nonzero_twice: page-secondary
  packet_fields: ["alert_class", "fingerprint", "catalog_version", "last_id", "freeze"]

Would I paste that YAML from a model straight onto the pager host? No. The model can draft it. A human pins it. Git records the pin, and the pager only sees the file that survived review.

The resolver refuses to be interesting

The pager-side tool should be boring on purpose, because clever tools invent argv under pressure. It reads the alert class, refuses unknown classes, and prints the command it would run. Mutating IDs stay blocked until observe succeeded in the same incident directory. If that feels rigid, ask who pays for a flexible reload at 3 a.m.

#!/usr/bin/env python3
"""Proposal: local catalog resolver. Not a production agent."""
from __future__ import annotations

import hashlib
import json
import os
import sys
import time
from pathlib import Path

import yaml  # PyYAML

CATALOG = Path(os.environ.get("RUNBOOK_CATALOG", "catalog.yaml"))
STATE = Path(os.environ.get("INCIDENT_DIR", "/tmp/incident")) / "state.json"


def load_catalog():
    data = yaml.safe_load(CATALOG.read_text())
    if not isinstance(data, dict) or "commands" not in data:
        raise SystemExit("catalog unreadable")
    return data


def fingerprint(alert: dict) -> str:
    raw = "|".join(
        [
            alert["alert_class"],
            alert.get("resource", "unknown"),
            alert.get("region", "unknown"),
        ]
    )
    return hashlib.sha256(raw.encode()).hexdigest()[:16]


def resolve(alert: dict) -> dict:
    cat = load_catalog()
    if cat.get("freeze"):
        return {
            "action": "escalate",
            "reason": "catalog_frozen",
            "version": cat.get("version"),
        }
    matches = [c for c in cat["commands"] if c["alert_class"] == alert["alert_class"]]
    if not matches:
        return {
            "action": "escalate",
            "reason": "no_pinned_id",
            "version": cat.get("version"),
        }
    observe = [c for c in matches if not c.get("mutate")]
    mutate = [c for c in matches if c.get("mutate")]
    state = json.loads(STATE.read_text()) if STATE.exists() else {}
    if observe and not state.get("observe_ok"):
        return {"action": "run", "command": observe[0], "version": cat.get("version")}
    if mutate and state.get("observe_ok"):
        return {"action": "run", "command": mutate[0], "version": cat.get("version")}
    return {
        "action": "escalate",
        "reason": "no_safe_next_id",
        "version": cat.get("version"),
    }


def main() -> None:
    alert = json.loads(sys.argv[1] if len(sys.argv) > 1 else sys.stdin.read())
    alert["fingerprint"] = fingerprint(alert)
    decision = resolve(alert)
    decision["fingerprint"] = alert["fingerprint"]
    decision["ts"] = int(time.time())
    print(json.dumps(decision, indent=2))
    if decision["action"] != "run":
        sys.exit(2)


if __name__ == "__main__":
    main()

Rehearse it like this when you are awake, not when you are guessing under a red Slack channel.

export RUNBOOK_CATALOG="$PWD/catalog.yaml"
export INCIDENT_DIR="$PWD/incidents/demo-1"
mkdir -p "$INCIDENT_DIR"

python3 resolve.py '{"alert_class":"redis.connections.saturated","resource":"redis-2","region":"us-east-1"}'

# expect action=run and id=redis-conn-sat-observe
# after a human marks observe ok:
echo '{"observe_ok": true}' > "$INCIDENT_DIR/state.json"
python3 resolve.py '{"alert_class":"node.disk.fill","resource":"/var","region":"us-east-1"}'

If the second call tries to mutate a class that has no mutate pin, the resolver should escalate. That refusal is the product. Do you want a clever tool, or a tool that will not get creative with systemctl?

Freeze is a catalog lock, not a chat reaction

Freeze, in this runbook, means the pinned file is untrusted. Maybe a command returned noise that no longer matches the host. Maybe a deploy landed, and the argv still names a unit that moved. Maybe someone spotted drift and does not want the next shift to replay a stale ID.

You freeze with a file write, not with a vibe in the incident channel. Unfreeze is not typing "lgtm" into Slack while the pager is still screaming.

# freeze: pager must escalate until a new pin exists
python3 - <<'PY'
from pathlib import Path
import yaml
p = Path("catalog.yaml")
data = yaml.safe_load(p.read_text())
data["freeze"] = True
p.write_text(yaml.safe_dump(data, sort_keys=False))
print("catalog frozen; pager will escalate")
PY

git add catalog.yaml
git commit -m "freeze catalog: redis-proxy reload looked stale"

Unfreeze is pinning a new version string and setting freeze: false in the same commit. If those two edits arrive in different commits, I would still treat the catalog as frozen. Why split the only two bits that decide whether a mutating ID may run?

Escalation is the default compiler output

Escalation should be cheaper to trigger than a guessed mutate, because guessed mutates create the next page. The packet is small on purpose. Secondary on-call should not reverse-engineer your chat history to learn what already ran.

ESCALATION PACKET
alert_class: redis.connections.saturated
fingerprint: a3c1f0e29b77c104
catalog_version: 2026.09.11-1
freeze: true
last_id: redis-conn-sat-observe
last_exit: 1
reason: catalog_frozen
asked_for: page-secondary

If you cannot fill those fields, you do not have a compiled incident yet. You have a mystery, and mysteries do not get systemctl reload. Would you hand a mystery to the next engineer and call that a first command?

Decision table you can print next to the pager

Alert class	Pinned first ID	Mutate ID	Freeze policy	Escalate when
`redis.connections.saturated`	`redis-conn-sat-observe`	`redis-conn-sat-reload-proxy`	Freeze catalog if reload exit is not 0	No ID, freeze, or observe never marked ok
`node.disk.fill`	`disk-fill-observe`	none in v1	Freeze if `df` shows the wrong mount	Any mutate request, because v1 has no mutate pin
unknown / empty class	none	none	Leave freeze unchanged	Always, with reason `no_pinned_id`
any class while `freeze: true`	none	none	Stay frozen until version bump	Always, with reason `catalog_frozen`

Print the table. Tape it near the laptop you actually take to the pager. If the table and the YAML disagree, the YAML is wrong until a human reconciles them. The pager does not get a vote, and the model does not get a vote either.

Draft the catalog where production cannot see you

The drafting environment and the production shell must not be the same window. That split is the safety story, not a branding story. I draft YAML at a desk, with the resolver tests, then I pin. I do not draft inside the SSH session that can reload a proxy.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can hold that drafting split: you iterate on catalog text and resolver tests on a box that is not production. The pager host only receives the pinned file. If a completion suggests a new mutate ID, it stays a draft until a human writes the version bump and clears the freeze bit.

Is a separate drafting box required? No. A local git repo is enough if you keep prod credentials out of the editor. The requirement is the air gap, not a vendor. Use the free server option if you want drafts off the laptop without opening production kubeconfigs. Do not let the model SSH anywhere from that box. Do not paste live secrets into a prompt to "make the YAML more realistic."

Rehearsal plan before you trust a pin

Label this as a rehearsal, because I am not citing a fleet I have not measured. Run the rows in order. If one fails, you do not pin, and you do not page-execute a guess to "just see."

Unknown class — feed alert_class: "totally.new" and expect exit 2 with no_pinned_id.
Frozen catalog — set freeze: true, replay a known class, expect catalog_frozen.
Observe before mutate — without state.json, a saturated Redis alert must return the observe ID only.
Mutate after observe — write observe_ok, replay Redis, expect the reload ID and mutate: true.
Disk fill has no mutate — even with observe_ok, node.disk.fill must escalate rather than invent rm.
Fingerprint stability — same resource twice, same 16-character fingerprint, so retries are visible.

python3 resolve.py '{"alert_class":"totally.new","resource":"x","region":"r"}'; echo exit:$?
# exit:2

If any row fails, that is a compile error at the desk. It is not an incident-skill problem for the person who just woke up. Fix the catalog, bump version, and only then copy the file onto the pager host.

Limitations, and who should not use this

This approach assumes you can name alert classes in advance. If your monitors still emit poetry, the compiler will escalate all night, and you will hate the noise. Good. That pain belongs on the monitor owners, not on the engineer holding the pager.

Do not use this if you have no human pin step. A model that writes catalog.yaml and a bot that executes it is not a compiler. It is a loop with extra files and a false sense of review. Do not use this as a substitute for IAM, sudoers, or change windows. A pinned rm is still rm, and a pinned reload still reloads.

Skip it on toy clusters where one person already types every command and reads every log line. The catalog helps when the next on-call is not the author of the argv. It also helps when you are the author, but you are tired enough to trust a fluent paragraph more than a hashed file.

I am not claiming latency numbers, token quotas, hardware, or model names. I am claiming a control: page time executes IDs you pinned at desk time. If that control feels slow, ask whether speed or correctness is the scarce resource while the alert is still firing.

Close the prompt window

So, will you keep pasting production logs into a chat box and hoping the argv is right? Or will you compile a short catalog, freeze it when it drifts, and escalate when the compiler shrugs? The second path is less cinematic. It is also the only first-command path I will trust when the operation mutates a host.

Keep drafts offline. Keep the pager dull. Pin the version, then go back to sleep if the ID did its job.

If You Cannot Name the Owner, Do Not Run the Command

Emery Huang — Thu, 10 Sep 2026 04:24:20 +0000

I stopped asking the on-call bot for cleverness at two in the morning, and I started asking it for a named owner instead. A restart looks decisive in Slack, but it is just a loud guess when nobody owns the blast radius. My working rule is blunt on purpose: if the packet cannot name a human owner, the bot does not receive mutating tools. Can a language model invent a service owner at three in the morning? Sure it can, and that is exactly why I refuse to let it try.

This write-up is a shadow-mode runbook, not a victory lap with invented graphs or a customer count. I treat last week's pages as the only honest training set I currently trust. Production credentials stay off the classifier host, and the artifact below is a replay harness you can run while you are still awake.

What this runbook is actually for

I keep seeing agent demos jump from a red panel to kubectl rollout restart as if that jump were courage. Is it courage, or is it a missing org chart wearing an incident emoji? Most of my painful nights were not missing a clever command. They were missing a person who could say whether the blast radius was one pod or a whole billing region.

Use this when you already have alerts, a service catalog that is at least half true, and a human who still owns the pager. Do not treat it as a license to auto-remediate a cluster you barely understand. The bot proposes checklists. The owner map decides whether proposing is even allowed tonight.

The packet, minus the poetry

Every alert that enters the shadow lane has to look like a boring JSON object, not a novel. I do not let the model free-type the incident into existence. I fill the fields first, then I ask the model to classify the packet, not to invent an owner, a region, or a restart.

{
  "alert_id": "page-2026-09-09-1842",
  "service": "checkout-api",
  "env": "prod",
  "region": "us-east-1",
  "deploy_sha": "a1b2c3d",
  "symptom": "p99_latency",
  "started_at": "2026-09-09T18:42:11Z",
  "related_services": ["payments-gw", "cart-cache"],
  "page_channel": "#inc-checkout"
}

Notice what is missing on purpose. There is no owner field in the packet, because the owner must come from a map I control. If I let the model fill owner, it will eventually pick a name that looks senior and is currently asleep. Would you want that name in the timeline tomorrow morning?

The owner map is the first command

My first command is not kubectl get and it is not a restart sketch. My first command is lookup_owner, and it reads a committed YAML file that on-call can argue about in daylight. The file is small, mean, and reviewable, which is the point.

# owners.yaml — the only place an unfreeze can happen
services:
  checkout-api:
    owner: "priya"
    backup: "ops-payments"
    blast_budget: 2
  payments-gw:
    owner: "diego"
    backup: "ops-payments"
    blast_budget: 1
  cart-cache:
    owner: "sam"
    backup: "ops-storefront"
    blast_budget: 3

blast_budget is the number of related services the bot may even mention in a proposed command. Why a hard number instead of "be careful"? Because models treat vibes as permission, and vibes do not page anyone. If checkout-api pages with four downstream names attached, a budget of two means inspect checkout-api plus one neighbor, then escalate.

Decision table I actually print

Owner lookup	Related-service blast	Allowed lane	First human action
Hit, budget holds	`<= blast_budget`	Propose read-only checks	Ack in the incident channel
Hit, budget exceeded	`> blast_budget`	Escalate, no cluster talk	Owner confirms the radius
Miss	any	Freeze mutating tools	Merge an `owners.yaml` change
Conflicting owners	any	Freeze, page both backups	Incident commander picks

If the table feels like bureaucracy, ask a better question before you throw it out. Would you rather debate YAML at eleven in the morning, or debate a surprise restart at three?

Replay last week's pages, not last week's prompt

I do not trust a prompt that I wrote while caffeinated and optimistic. I replay a JSONL file of last week's alerts through a classifier that cannot execute anything on a cluster. The output is a receipt, and the receipt is what I read on Monday with the actual owner sitting nearby.

# replay_pager.py — proposed harness, not a production controller
from __future__ import annotations

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any

import yaml

MUTATING_VERBS = ("restart", "rollout", "delete", "scale", "apply", "drain")


@dataclass
class Verdict:
    alert_id: str
    owner: str | None
    lane: str
    proposals: list[str]
    freeze: bool
    reason: str


def load_owners(path: Path) -> dict[str, Any]:
    data = yaml.safe_load(path.read_text())
    return data["services"]


def score_blast(alert: dict[str, Any], owners: dict[str, Any]) -> int:
    related = [alert["service"], *alert.get("related_services", [])]
    known = [name for name in related if name in owners]
    return len(set(known))


def lookup(alert: dict[str, Any], owners: dict[str, Any]) -> dict[str, Any] | None:
    return owners.get(alert["service"])


def propose_readonly(alert: dict[str, Any]) -> list[str]:
    svc = alert["service"]
    return [
        f"metrics.query service={svc} window=15m",
        f"logs.search service={svc} level=error deploy={alert.get('deploy_sha')}",
        f"deploy.info service={svc} sha={alert.get('deploy_sha')}",
        f"channel.post {alert.get('page_channel')} owner-lookup-complete",
    ]


def classify(alert: dict[str, Any], owners: dict[str, Any]) -> Verdict:
    rec = lookup(alert, owners)
    blast = score_blast(alert, owners)
    if rec is None:
        return Verdict(
            alert_id=alert["alert_id"],
            owner=None,
            lane="freeze",
            proposals=[],
            freeze=True,
            reason="owner_map_miss",
        )
    if blast > rec["blast_budget"]:
        return Verdict(
            alert_id=alert["alert_id"],
            owner=rec["owner"],
            lane="escalate",
            proposals=[f"page.owner {rec['owner']}", f"page.backup {rec['backup']}"],
            freeze=True,
            reason="blast_over_budget",
        )
    return Verdict(
        alert_id=alert["alert_id"],
        owner=rec["owner"],
        lane="propose",
        proposals=propose_readonly(alert),
        freeze=False,
        reason="owner_and_budget_ok",
    )


def reject_if_mutating(text: str) -> None:
    lowered = text.lower()
    for verb in MUTATING_VERBS:
        if verb in lowered:
            raise ValueError(f"mutating verb {verb!r} is not allowed in shadow replay")


def replay(alerts_path: Path, owners_path: Path, out_path: Path) -> None:
    owners = load_owners(owners_path)
    receipts: list[dict[str, Any]] = []
    for line in alerts_path.read_text().splitlines():
        if not line.strip():
            continue
        alert = json.loads(line)
        verdict = classify(alert, owners)
        for item in verdict.proposals:
            reject_if_mutating(item)
        receipts.append(verdict.__dict__)
    out_path.write_text(json.dumps(receipts, indent=2))
    print(f"wrote {len(receipts)} receipts to {out_path}")


if __name__ == "__main__":
    replay(Path("alerts.jsonl"), Path("owners.yaml"), Path("receipts.json"))

Keep a tiny fixture next to the script so Monday you is not tempted to improvise.

{"alert_id":"page-001","service":"checkout-api","deploy_sha":"a1b2c3d","related_services":["payments-gw"],"page_channel":"#inc-checkout"}
{"alert_id":"page-002","service":"mystery-batch","deploy_sha":"deadbeef","related_services":["warehouse-etl"],"page_channel":"#inc-unknown"}

Run the harness when you are not in an incident, then read only the frozen rows first.

python3 replay_pager.py
jq '.[] | select(.freeze==true)' receipts.json

If a receipt contains restart, the harness should throw before the file is written. That is the whole point of shadow mode, and it is not optional. The model can still draft a rationale later, but the verbs are a gate in Python, not a wish inside a system prompt.

Freeze and unfreeze, the boring version

Here is the freeze rule for this lane, and it is not a Slack emoji from a tired primary. When owner_map_miss or blast_over_budget fires, mutating tool credentials stay sealed for that service. Unfreeze is a merged change to owners.yaml, or an explicit budget bump reviewed by the backup owner in the same pull request.

A chat message from the bot does not count as unfreeze, and "looks fine" does not count either. Escalation order is equally boring, which I like, because boring is searchable tomorrow.

Page the named owner from the map, not a cluster nickname.
Page the backup alias if there is no ack inside ten minutes.
Page the incident commander only after both of those paths fail.
Never page a node pool, a Terraform path, or "whoever last touched prod."

Would I still restart a stuck deployment after that sequence? Sometimes I would, after a human with a name on the map says the blast radius is theirs. The bot does not get to skip that sentence, and neither do I when I am the one holding the pager.

Where a free model actually belongs

I still want a model in the loop, just not in the loop that holds kubectl or cloud keys. The useful job is classification: given a filled packet and the owner map, draft a short rationale and a propose-only checklist. That work can live on a throwaway box that never sees production tokens.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For that shadow lane I point the classifier at MonkeyCode, which is open source, using its free model access and free server option so the remote box only rewrites redacted packets into receipts. If you try the same split, keep credentials on your side of the wall and let the remote host do language, not actions. Delete that paragraph and the runbook still works, which is how I want the tooling to sit.

A Monday test plan, not a vibe

Dump last week's pages into alerts.jsonl with service, region, and related services filled in.
Commit owners.yaml with budgets that feel uncomfortably small on purpose.
Run replay_pager.py and count freezes versus proposals without changing verbs.
Read every blast_over_budget row with the actual owner, not with the model.
Only then consider exposing a read-only metrics tool, still without mutate verbs.

If the freeze rate is high, your map is lying, not the script. Fix the map in a normal review. Do not loosen MUTATING_VERBS just to make a dashboard look friendly before the next rotation.

Who should not use this

Skip this approach if you have no service catalog, no named humans, and no appetite for YAML arguments during business hours. Skip it if your compliance team has not approved sending even redacted alert text to a hosted model. Skip it if you need sub-minute auto-remediation, because this workflow is deliberately slow. Skip it if you were hoping the agent would replace the pager. It will not replace the pager. It will make the pager more annoying in a way you can audit.

Limitations I will not wave away

Replay is not production, and last week's alert shape will not match the novel failure you get on a holiday weekend. Owner maps rot when people change teams and nobody updates YAML, which is a social failure the script cannot see. Blast budgets are crude, and two related services can still be more dangerous than ten distant ones. The verb denylist is bypassable with creative phrasing, so keep mutating tools out of the process entirely while you are in shadow mode.

I am not giving you latency numbers, pass rates, or a customer story, because those would be invented for this page. The conclusion stays the same when the tools change next quarter. If you cannot name the owner, you do not run the command. Everything else is a receipt you can read when you are awake enough to argue with YAML.

T+0 Is Observe-Only: Bind Alert Classes to a Clock and a Command Budget

Emery Huang — Wed, 09 Sep 2026 16:04:19 +0000

The on-call bot should not improvise while the pager is screaming at three in the morning. I treat every page as a clock, an alert class, and a shrinking command budget. Until a human acknowledges the incident, the write lane stays closed on every mutating command. Why would we let a model negotiate production access during the noisiest fifteen minutes of the week?

This runbook is a state machine I can diff in git, not a prompt I hope the model remembers. Agents that just figure it out will assume a root cause, then reach for restart as the shortest verb. Have you watched an assistant invent confidence while the graphs are still loading in another tab? I would rather ship a boring incident clock tonight than debug a clever shell tomorrow morning.

What this runbook actually decides

I answer four questions before any process on a box is touched tonight.

Which alert class opened the incident, and is that class even in the allowlist?
What time is it on the incident clock, measured from the first accepted page?
Which read-only probes are still inside the remaining command budget?
Is the write lane frozen by clock expiry, budget exhaustion, or a shift handoff?

If any answer is missing, the bot pages a human and waits without improvising a fix. The bot does not get a consolation prize of just one restart against production hosts. Escalation is a first-class action in this runbook file, not a failure of nerve. Would you bet the checkout path on a model that still has not read deploy history?

The incident clock

I open a clock at T+0 when the first well-formed page is accepted, not when a model starts talking. T+0 through T+5 is observe-only, which means metrics, logs, and deploy history, never restarts. T+5 through T+10 allows the remaining read-only probes that the alert class declared in advance. After T+10 without a human ACK, I escalate and freeze writes even if someone unfroze them earlier.

Does a loud P1 severity badge change the incident clock for mutating commands at all? I stay stubborn here, because severity tells me who to wake, not which kill switch to flip during the storm. A screaming pager is a reason to read faster, not a reason to grant kubectl delete. The badge still matters for who gets the phone call, and that mapping lives beside the clock, not inside a prompt.

Clock states I actually encode

observe: T+0 to T+5, read-only probes only, and any unfreeze request is ignored
diagnose: T+5 to T+10, still read-only, while the budget ticks down per command
escalate: T+10 with no ACK, page the secondary, and freeze the write lane
acked: a human owns the incident, so a scoped unfreeze may be requested
frozen: clock expired, budget spent, or the pager changed hands mid-incident
closed: incident ended, receipts archived, and the clock is destroyed on purpose

Bind the page to an alert class first

I refuse to let the model classify the outage in real time under pager noise. Each page must carry alert_class, service, correlation_id, and a timestamp I can parse without guessing. Missing fields do not get imputed from the title of the Slack thread during the storm. If the class is unknown, the bot stops, because unknown is not a synonym for try restart.

Deduping the storm before T+0 starts

Alert storms lie, and they will lie to an agent with a shell faster than they lie to you. I group pages that share correlation_id and alert_class inside a two-minute window into one incident. The clock starts once, on the first accepted member of that group, not on every echo. If a later page arrives with a new class, I open a sibling incident instead of widening the first one.

Why merge a node failure into a checkout latency clock just because they landed in the same minute? Sibling incidents can share a Slack thread and still keep separate budgets, clocks, and write lists. The bot is allowed to mention the sibling in a receipt, not to borrow its commands. That keeps a noisy kube event from unlocking a checkout flag flip.

Here is the YAML I keep next to the service, not inside a chat prompt that can drift.

# runbook/incident_clock.yaml
version: 1
clock:
  observe_until_s: 300
  diagnose_until_s: 600
  auto_escalate_s: 600
  unfreeze_ttl_s: 900
budget:
  max_reads: 6
  max_writes: 1
  max_retries_per_command: 1
shift_handoff:
  freeze_writes: true
  require_incoming_ack: true
alert_classes:
  checkout_latency:
    service: checkout
    first_commands:
      - id: p95
        argv: ["promql", "histogram_quantile(0.95, checkout_latency_bucket)"]
        mutates: false
      - id: deploys
        argv: ["deploylog", "checkout", "--since", "90m"]
        mutates: false
      - id: errors
        argv: ["logql", "count_over_time({app=checkout} |= `5xx` [15m])"]
        mutates: false
    escalate:
      page: ["checkout-secondary", "payments-owner"]
      reason: "clock_expired_or_budget_spent"
    writes_after_ack:
      - id: canary_toggle
        argv: ["featureflag", "checkout", "canary", "off"]
        mutates: true
        requires_unfreeze: true
  kube_node_not_ready:
    service: platform
    first_commands:
      - id: node_status
        argv: ["kubectl", "get", "nodes", "--no-headers"]
        mutates: false
      - id: last_cordon
        argv: ["auditlog", "search", "cordon", "--since", "2h"]
        mutates: false
    writes_after_ack: []

Notice the kube class has an empty write list on purpose, and I want that emptiness to be loud. Not every page deserves a mutation, even after a human finally shows up. If the class cannot name a write, the bot cannot invent one later during the call. That is the whole point of binding classes before the incident starts.

First commands are probes, not fixes

The first commands are boring on purpose, and I like them that way at three. They read p95, recent deploys, error counts, node status, and audit history without touching traffic. They never restart, never cordon, never flip a flag, and never drain a node. If a command is not in first_commands for that class, the executor returns denied and spends nothing.

I also stamp a receipt after every probe, because memory is not an audit log I can hand to the secondary. The receipt carries the clock time, the remaining budget, and the exact argv that ran. When the secondary asks what the bot already did, I paste the receipt file instead of a vibe. Would you rather debug from that file or from a model summarizing its own courage?

# /var/oncall/receipts/inc-20260909-0142.jsonl
{"t_plus_s": 12, "cmd": "p95", "mutates": false, "exit": 0, "budget_reads_left": 5}
{"t_plus_s": 41, "cmd": "deploys", "mutates": false, "exit": 0, "budget_reads_left": 4}
{"t_plus_s": 88, "cmd": "errors", "mutates": false, "exit": 0, "budget_reads_left": 3}

The executor compares argv byte for byte against the YAML, and it rejects paraphrases from the model. Equivalent kubectl is how nodes disappear while everybody is still reading graphs in another window. If the assistant wants a new probe, it opens a pull request against the runbook, not a shell. Can your current bot prove it did not rewrite the command under pressure?

Escalation is cheaper than a clever write

At T+10 without ACK, I page the secondary and freeze writes, even if graphs look obvious. The bot may keep reading if budget remains, but it may not spend the write slot tonight. Escalation is not an insult to the primary sitting in the channel. It is how I keep a tired person from rubber-stamping a restart the model already drafted.

I escalate on three triggers, and I log which one fired in the receipt stream.

Clock hit auto_escalate_s with no human ACK in the incident record.
Read budget hit zero before a coherent picture showed up in receipts.
The pager rotated, because a new human has not accepted write risk yet.

That third trigger is the one people forget when they celebrate handoff GIFs in the channel. Shift change is a freeze event, not a courtesy Slack message in the incident thread. The incoming engineer inherits the clock and the receipts, not an open write lane. Do you currently freeze writes when the rotation bot posts that Bob is up?

Freeze and unfreeze without a chat vote

Unfreeze is not a thumbs-up emoji in the incident channel, and I will not parse emoji. I require a scoped record: incident id, service, single command id, expiry, and the on-call identity. The unfreeze dies when the TTL ends, when the budget's one write is spent, or when the shift hands off. A freeze always wins a race with an unfreeze, and the executor checks freeze first on purpose.

The unfreeze record

{
  "incident_id": "inc-20260909-0142",
  "service": "checkout",
  "command_id": "canary_toggle",
  "expires_at": 1780000000,
  "actor": "emery.oncall"
}

That record cannot say command_id: "*" and it cannot outlive unfreeze_ttl_s in the YAML. If the human wants a second write, they spend a new ACK path and a new budget, which this file refuses. I would rather reopen the discussion than silently refill a kill switch after a canary toggle. Is your unfreeze currently a channel mood, or a record another engineer can replay?

# incident_clock.py — proposal / policy sketch, not production-hardened
from __future__ import annotations
import time
from dataclasses import dataclass
from typing import Literal

State = Literal["observe", "diagnose", "escalate", "acked", "frozen", "closed"]

@dataclass
class Unfreeze:
    incident_id: str
    service: str
    command_id: str
    expires_at: float
    actor: str

@dataclass
class Incident:
    id: str
    alert_class: str
    service: str
    t0: float
    acked_by: str | None
    frozen: bool
    reads_used: int
    writes_used: int
    shift_id: str

def clock_state(inc: Incident, now: float, cfg: dict) -> State:
    if inc.frozen:
        return "frozen"
    elapsed = now - inc.t0
    if inc.acked_by:
        return "acked"
    if elapsed >= cfg["clock"]["auto_escalate_s"]:
        return "escalate"
    if elapsed >= cfg["clock"]["observe_until_s"]:
        return "diagnose"
    return "observe"

def first_ids(cfg: dict, alert_class: str) -> set[str]:
    return {c["id"] for c in cfg["alert_classes"][alert_class]["first_commands"]}

def freeze_reasons(inc: Incident, now: float, cfg: dict, current_shift: str) -> list[str]:
    reasons = []
    st = clock_state(inc, now, cfg)
    if st in {"escalate", "frozen"}:
        reasons.append("clock_or_flag")
    if inc.reads_used >= cfg["budget"]["max_reads"]:
        reasons.append("read_budget")
    if inc.writes_used >= cfg["budget"]["max_writes"]:
        reasons.append("write_budget")
    if current_shift != inc.shift_id:
        reasons.append("shift_handoff")
    return reasons

def unfreeze_ok(inc: Incident, cmd: dict, now: float, u: Unfreeze | None) -> bool:
    if u is None:
        return False
    return (
        u.incident_id == inc.id
        and u.service == inc.service
        and u.command_id == cmd["id"]
        and u.expires_at > now
        and bool(u.actor)
    )

def allow(inc, cmd, now, cfg, unfreeze, current_shift) -> tuple[bool, str]:
    reasons = freeze_reasons(inc, now, cfg, current_shift)
    if reasons and cmd["mutates"]:
        return False, f"frozen:{','.join(reasons)}"
    st = clock_state(inc, now, cfg)
    if not cmd["mutates"]:
        if inc.reads_used >= cfg["budget"]["max_reads"]:
            return False, "read_budget"
        if st == "observe" and cmd["id"] not in first_ids(cfg, inc.alert_class):
            return False, "not_in_first_commands"
        return True, "read_ok"
    if st != "acked":
        return False, "writes_require_ack"
    if not unfreeze_ok(inc, cmd, now, unfreeze):
        return False, "unfreeze_missing_or_expired"
    return True, "write_ok"

Is this glamorous agentic AI with a memory of the company wiki? No, and that is the feature I want. Most of the danger is a model assuming a cause, then picking a verb that sounds like ownership. The clock does not care how confident the summary sounded in Slack at T+3.

CLI I wrap around the same policy

python incident_clock.py accept --file /var/oncall/page.json
python incident_clock.py probe  --incident inc-20260909-0142 --cmd p95
python incident_clock.py status --incident inc-20260909-0142
python incident_clock.py ack    --incident inc-20260909-0142 --actor emery.oncall
python incident_clock.py unfreeze --incident inc-20260909-0142 --cmd canary_toggle --ttl 900
python incident_clock.py handoff --incident inc-20260909-0142 --from emery.oncall --to bob.oncall

Every subcommand prints a receipt line or a deny reason, never a fuzzy paragraph. If handoff runs, writes are frozen until Bob ACKs and issues a fresh unfreeze. I keep kube credentials out of this process, because a policy engine with extra secrets becomes an incident of its own. Can you run status right now and see the clock without asking a model to narrate it?

A decision table I keep in the repo

Clock / condition	Reads	Writes	Escalate	Notes
T+0–T+5, no ACK	allowlisted probes	deny	no	observe-only, ignore unfreeze
T+5–T+10, no ACK	remaining budget	deny	no	still no personality
T+10, no ACK	optional	deny	yes	freeze writes
ACK, unfreeze scoped	yes	one matching id	no	TTL and service must match
Budget spent	deny	deny	yes	freeze wins
Shift handoff	yes	deny	page incoming	new ACK required
Unknown alert class	deny	deny	yes	do not classify live

Test plan I want in the pipeline before the pager is live

I do not need a cluster to prove the policy holds under fake time. I freeze now, feed the executor incidents, and assert deny reasons by string. These tests are a proposal you can copy, not production metrics I pretend already exist. If you skip them, the YAML is just literature sitting next to a dangerous bot.

# test_incident_clock.py — proposal
def test_observe_denies_write_even_with_unfreeze():
    ...

def test_t10_auto_escalates_and_freezes_writes():
    ...

def test_unfreeze_expires_and_cannot_be_reused():
    ...

def test_shift_handoff_freezes_write_lane():
    ...

def test_unknown_class_pages_human():
    ...

def test_empty_write_list_cannot_gain_a_restart():
    ...

def test_sibling_class_does_not_share_write_budget():
    ...

Run them like this, and fail the pipeline if any policy test is skipped.

python -m pytest test_incident_clock.py -q
python -c "import yaml; yaml.safe_load(open('runbook/incident_clock.yaml'))"

If the YAML cannot parse, the bot does not start, and that is intentional friction. A broken runbook is not an invitation to free-style against production with a helpful model. Would you merge a payment service that cannot parse its own config at boot?

Where a coding assistant actually helps

I still want help writing the YAML and the tests, just not help SSHing into prod during the incident. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode, an open-source coding assistant with free model access and a free server option, to draft alert-class stubs and the pytest file above. The assistant proposes probes from my service README, and I keep every mutating argv behind ACK plus unfreeze.

It does not get a production kube context, and I do not paste secrets into the prompt. If you already maintain runbooks in git, try generating one alert class and its negative tests there, then delete anything that is not in your allowlist. The clock stays in your repo either way, which is the part I actually care about during a page.

Limitations, and who should not use this

This design assumes you can identify alert classes before the outage, which baby systems sometimes cannot honestly do. It also assumes your executor is the only path to kubectl and flag flips during an incident. A human with a raw shell can bypass every freeze, and I do not pretend otherwise in this writeup. The clock will look slow if your real ACK time is ninety seconds and you already trust a tiny reviewed catalog.

Do not use this if you want an agent to own the incident end to end without a human ACK. Do not use it if legal requires a two-person unfreeze and this file only stores one actor. Do not use it to hide missing observability behind a budget of six reads that never quite explain the graph. And please do not point a free coding assistant at live credentials just because the runbook is sitting in the same repo.

The core conclusion does not change when the model gets smarter again next month. T+0 stays observe-only, first commands stay probes, escalation stays cheaper than a guessed restart, and freeze still wins at shift change. Can your current bot explain which of those rules it followed, with receipts you could paste? If the answer is a shrug, start with the clock, not with a bigger prompt.

Make Escalation Cheaper Than Restart: A Three-Lane On-Call Runbook

Emery Huang — Tue, 08 Sep 2026 03:35:19 +0000

If a restart is cheaper to type than a page, your on-call runbook is teaching the wrong habit. I want the first twelve minutes after an alert to prefer observation and escalation over mutation, every time. Why should a language model get a kubectl rollout restart before a human even hears the pager? This piece treats the runbook as a three-lane contract: observe, page, then mutate, with a freeze latch that a human must clear.

I am not claiming a war story from last night's incident, and I am not selling a miracle agent. What follows is a proposed contract plus a small guard you can run locally before any bot is allowed to speak. If the guard cannot classify a command, the command does not ship. That is the whole thesis, and it stays useful even if you never touch an assistant.

The problem the runbook usually hides

Most “AI on-call” drafts collapse three different jobs into one chatty paragraph. The model reads a noisy alert, invents a cause, and then reaches for the most dramatic verb it knows. Restart sounds decisive. Scale-up sounds helpful. Delete-the-pod sounds like surgery. None of those verbs are first commands.

Ask yourself a blunt question: after the alert fires, who owns the next twelve minutes? If the answer is “the bot, unless someone interrupts,” you already lost the lane design. A runbook that lists twenty kubectl spells but never names a pager target is not a runbook. It is a mutation menu with a narrative glued on top.

I keep the lanes boring on purpose. Observe commands may run automatically because they cannot change production. Page commands are the default next step when the observe clock expires. Mutate commands exist, but they sit behind a freeze latch and a human token. Does that feel slow? Good. Slow is the point when the alternative is an invented restart.

Three lanes, one clock, one latch

Here is the contract I actually want checked in beside the service.

Observe: read-only commands with a hard wall clock, usually three minutes.
Page: create or raise an incident before any mutating verb is even proposed.
Mutate: a tiny allowlist that requires an unfrozen latch and a human token.
Freeze: after any mutate proposal or execution, further mutate traffic is refused until a human unfreezes.

The clock is not decoration. If observation burns three minutes and the service is still red, escalation is cheaper than another theory. Who benefits from a fourth describe pod while the primary on-call is still asleep? Nobody in the blast radius, that is who.

I also refuse mixed argv. A command is observe, page, or mutate. If your wrapper both tails logs and restarts a Deployment, split it. The guard below is deliberately dumb, because clever classifiers are how agents start assuming things.

Artifact: a versioned runbook contract

Label this YAML as a proposal you can edit, not as production law copied from my cluster. Put it in git next to the service so reviewers can argue about verbs, not vibes.

# runbook.payments-api.yaml
# Proposal / example — unexecuted. Review before any real pager or cluster use.
apiVersion: oncall.local/v1
kind: Runbook
metadata:
  service: payments-api
  owner: platform-oncall
  timezone: UTC
clock:
  observeSeconds: 180
  escalateAfterSeconds: 720
  muteRepeatSeconds: 300
alert:
  requiredFields:
    - fingerprint
    - service
    - severity
    - firedAt
    - runbookRef
  failClosedOnMissing: true
lanes:
  observe:
    - id: rollout-status
      argv: ["kubectl", "rollout", "status", "deploy/payments-api", "-n", "prod", "--timeout=30s"]
    - id: recent-events
      argv: ["kubectl", "get", "events", "-n", "prod", "--field-selector", "involvedObject.name=payments-api"]
    - id: error-rate
      argv: ["promtool", "query", "instant", "sum(rate(http_requests_total{service=\"payments-api\",code=~\"5..\"}[5m]))"]
  page:
    - id: page-primary
      argv: ["pd", "incident", "create", "--service", "payments-api", "--urgency", "high"]
    - id: raise-severity
      argv: ["pd", "incident", "update", "--from-alert", "--urgency", "high"]
  mutate:
    - id: restart-deploy
      argv: ["kubectl", "rollout", "restart", "deploy/payments-api", "-n", "prod"]
      requiresHumanToken: true
      requiresUnfrozen: true
      maxBlastRadius: "one Deployment in prod"
freeze:
  latchFile: "./oncall.freeze"
  reasonFile: "./oncall.freeze.reason"
  rule: "After any mutate proposal or execution, refuse further mutate until a human unfreezes."
unfreeze:
  argv: ["./unfreeze.sh"]
  requiresHumanToken: true

Notice what is missing. There is no “ask the model what to do next” field. There is no free-form shell. The alert schema fails closed if fingerprint or runbookRef is absent, because a bot that cannot name the runbook should not pick a lane. That rule is adjacent to incomplete-alert hygiene, but the load-bearing idea here is the clock plus the page lane, not a packet filter.

Artifact: a guard that refuses the wrong lane

The script is a proposal you can run against recorded argv. It does not talk to Kubernetes. It does not page anyone. It classifies, watches the clock, and honors the freeze latch.

# runbook_guard.py
# Proposal / example — unexecuted. Local classifier only; no cluster calls.
from __future__ import annotations

import json, sys, time
from pathlib import Path

import yaml

LATCH = Path("oncall.freeze")
STATE = Path("oncall.state.json")


def load_runbook(path: str) -> dict:
    data = yaml.safe_load(Path(path).read_text())
    if data.get("kind") != "Runbook":
        raise SystemExit("not a Runbook document")
    return data


def lane_for(runbook: dict, argv: list[str]) -> str | None:
    for lane in ("observe", "page", "mutate"):
        for step in runbook["lanes"][lane]:
            if step["argv"] == argv:
                return lane
    return None


def state() -> dict:
    if STATE.exists():
        return json.loads(STATE.read_text())
    return {"alertFiredAt": None, "pagedAt": None, "lastLane": None}


def save(st: dict) -> None:
    STATE.write_text(json.dumps(st, indent=2))


def decide(runbook: dict, argv: list[str], now: float, human_token: bool) -> dict:
    st = state()
    lane = lane_for(runbook, argv)
    if lane is None:
        return {"allow": False, "reason": "argv is not on any sealed lane"}

    fired = st.get("alertFiredAt") or now
    st["alertFiredAt"] = fired
    elapsed = now - fired
    observe_s = runbook["clock"]["observeSeconds"]
    escalate_s = runbook["clock"]["escalateAfterSeconds"]

    if lane == "observe" and elapsed > observe_s and not st.get("pagedAt"):
        return {
            "allow": False,
            "reason": "observe window closed; issue a page lane command first",
            "elapsedSeconds": elapsed,
        }

    if lane == "page":
        st["pagedAt"] = now
        st["lastLane"] = lane
        save(st)
        return {"allow": True, "lane": lane, "elapsedSeconds": elapsed}

    if elapsed >= escalate_s and not st.get("pagedAt") and lane != "page":
        return {
            "allow": False,
            "reason": "twelve-minute clock elapsed without a page; mutate is refused",
        }

    if lane == "mutate":
        if LATCH.exists():
            return {"allow": False, "reason": "freeze latch is set; human must unfreeze"}
        if not human_token:
            return {"allow": False, "reason": "mutate requires a human token"}
        LATCH.write_text("frozen after mutate proposal\n")
        Path("oncall.freeze.reason").write_text("mutate proposed: " + json.dumps(argv))
        st["lastLane"] = lane
        save(st)
        return {"allow": True, "lane": lane, "note": "latch set; further mutate refused"}

    st["lastLane"] = lane
    save(st)
    return {"allow": True, "lane": lane, "elapsedSeconds": elapsed}


def main() -> None:
    if len(sys.argv) < 3:
        raise SystemExit("usage: runbook_guard.py RUNBOOK.yaml -- argv...")
    runbook = load_runbook(sys.argv[1])
    try:
        sep = sys.argv.index("--")
    except ValueError:
        raise SystemExit("missing -- before argv")
    argv = sys.argv[sep + 1 :]
    human = "--human-token" in sys.argv[:sep]
    print(json.dumps(decide(runbook, argv, time.time(), human), indent=2))


if __name__ == "__main__":
    main()

A few commands show the shape. None of these should be aimed at production until you replace the pager binary and the kube context on purpose.

# Proposal / example commands — local classification only.
python runbook_guard.py runbook.payments-api.yaml -- \
  kubectl rollout status deploy/payments-api -n prod --timeout=30s

# After three minutes without a page, the same observe command should fail closed.
python runbook_guard.py runbook.payments-api.yaml -- \
  pd incident create --service payments-api --urgency high

# Mutate without a human token should fail.
python runbook_guard.py runbook.payments-api.yaml -- \
  kubectl rollout restart deploy/payments-api -n prod

# Mutate with a token should succeed once, then freeze.
python runbook_guard.py runbook.payments-api.yaml --human-token -- \
  kubectl rollout restart deploy/payments-api -n prod

# Second mutate should hit the latch.
python runbook_guard.py runbook.payments-api.yaml --human-token -- \
  kubectl rollout restart deploy/payments-api -n prod

The unfreeze path is intentionally ugly. A human writes a reason and deletes the latch. The bot never deletes the latch, even if the model is very polite about it.

# unfreeze.sh — proposal / example
set -euo pipefail
if [[ -z "${HUMAN_TOKEN:-}" ]]; then
  echo "HUMAN_TOKEN required" >&2
  exit 1
fi
echo "unfrozen by human at $(date -u +%FT%TZ)" >> oncall.freeze.reason
rm -f oncall.freeze

Decision table you can paste into the pull request

Situation	Allowed lane	Guard result
Alert missing `runbookRef`	none	refuse before argv classification
T+0 to T+180s, argv on observe list	observe	allow
T+181s, still no page, observe argv	observe	refuse, demand page
Any time, exact page argv	page	allow, stamp `pagedAt`
T+720s, never paged, mutate argv	mutate	refuse
Paged, mutate argv, no human token	mutate	refuse
Paged, mutate argv, human token, no latch	mutate	allow once, set latch
Latch present, any mutate argv	mutate	refuse until unfreeze
Argv not in YAML	none	refuse

Would I let a model fill the YAML? Only as a draft. The allowlist is a human artifact because the cost of a wrong verb is paid by whoever is asleep. That is also why escalation sits in front of restart: paging is reversible attention, restart is not.

Where a free assistant actually helps

Drafting this YAML by hand is tedious, and tedium is how observe commands quietly grow a --force. I will use an assistant to propose extra read-only argv, to argue about blast radius wording, and to generate the decision table from the file. I will not give that assistant a kubeconfig.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat MonkeyCode here as a drafting surface: free model access plus a free server option, so the runbook text can be iterated without standing up a GPU box. That is the whole product role in this workflow. No model name, quota, or hardware story is implied, because those details change and I will not invent them.

If you try that path, keep the server on the draft. The guard process, the latch file, and the pager token stay on a machine you already trust. Mixing “helpful autocomplete” with cluster-admin is how assume-happy agents get a restart they were never offered.

A test plan you can run without paging the org

Do not rehearse this against production paging policies on a Friday. Use a fixture alert and a fake pd stub that only writes a JSON line.

Feed an alert JSON missing firedAt. The wrapper around the guard should refuse before argv runs.
Run three observe commands inside three minutes. All three should allow.
Fast-forward alertFiredAt in oncall.state.json by 181 seconds. The next observe command should refuse.
Run page-primary. Confirm pagedAt is stamped and the stub file contains one incident.
Attempt restart-deploy without --human-token. Expect refuse.
Attempt it with --human-token. Expect allow, and oncall.freeze must exist afterward.
Attempt a second restart. Expect refuse with the latch reason.
Run unfreeze.sh with HUMAN_TOKEN set. Confirm the latch is gone and the reason file kept history.
Replay a slightly different argv, such as adding --force. Expect refuse because the list is exact.

If step nine ever starts “helpfully” normalizing flags, you broke the guard. Exact argv is a feature. Fuzzy matching is how restart becomes delete.

Limitations, and who should not use this

This contract does not replace IAM, admission control, or a real incident bot. A local latch file is process-local truth, not a distributed lock. Two replicas of the bot can race unless you put the latch on something both of them honor. The Python classifier also cannot see shell wrappers, so bash -c 'kubectl rollout restart ...' will not match and must stay refused.

Who should skip this approach? Anyone without a real human roster, because a page lane with nobody on the other end is theater. Anyone handing cluster-admin to an unsupervised agent, because the YAML will not save you. Anyone whose “runbook” is still a wiki paragraph with no argv. And anyone who needs sub-minute automated remediation with a signed playbook already reviewed by SRE; this design is slower on purpose.

I also will not pretend the twelve-minute constant is science. Change escalateAfterSeconds when your error budget and your pager load demand it. Just do not let the model edit the clock during an incident. The clock is policy. Policy that mutates itself under stress is not policy.

What I want in the merge request

When this lands, I want four files and one argument. The YAML, the guard, the unfreeze script, and a recorded fixture alert. The argument is simple: did we make paging cheaper than restart, and does the latch survive a second mutate? If reviewers cannot answer those two questions from the diff, the runbook is still a blog post.

So, will your next alert still offer a restart in the first breath? If yes, the lanes are fake. Seal observe, put escalation on a clock, and freeze mutate until a human says otherwise. That is the runbook I will actually keep next to production, and it is the one I am willing to let an assistant draft—never execute.

If you want a quiet place to iterate on the YAML before it hits git, MonkeyCode’s free model access and free server option are enough for drafting. Keep kubectl and the pager token somewhere the draft cannot reach.

The Bot Already Acted: Freeze Mutating Commands Until a Human Unfreezes

Emery Huang — Sun, 06 Sep 2026 21:22:30 +0000

The core conclusion is simple, and I will not bury it under another agent prompt. Once an on-call bot has touched production, the next danger is a second mutation, not a missing explanation. You need a freeze flag in the runbook, and an unfreeze path that a human must sign. Why do we keep treating freeze as a Slack vibe when the bot only obeys structured state?

I am writing this as a runbook control, not as a story about a heroic war room. Agentic on-call tools keep assuming the next step is another restart, another scale event, or another config write. A freeze rule is the smallest honest answer I have found: halt mutations after the first accepted action, then require a human unfreeze with evidence. If your bot cannot freeze, it is not on-call help. It is an unsupervised change pipeline wearing a pager hat.

What freeze is, and what it is not

Freeze is not silence, and it is not a polite system prompt that says please stop. Freeze is a durable incident flag that rejects mutating commands until an unfreeze record exists. Unfreeze is not a vibe check in chat. Unfreeze is a signed transition with a reason, a scope, and an expiry.

I keep freeze separate from three other controls people mash together. Allowlists decide which commands can ever exist. Escalation decides which human gets the page. Completeness checks decide whether the alert is even usable. Freeze answers a later question: the incident is already open, something already ran, so what is forbidden now?

Ask yourself the ugly version. If the bot rolled a canary and latency got worse, should it roll again because the model feels confident? If a human is already in the cluster, should the bot still drain nodes because the runbook text still says consider draining? Freeze exists because those answers must be no without another model debate.

The runbook rule I actually encode

I write four fields into the runbook so the bot cannot invent policy from tone. They are boring on purpose, because boring serializes.

freeze_after — which accepted action arms the freeze, usually the first mutating command.
frozen_verbs — the command classes that die during freeze, such as restart, scale, drain, rollback, apply.
read_allow — diagnostics that remain legal, such as logs, metrics, describe, trace.
unfreeze — who may clear the flag, with which evidence, and for how long.

Here is a proposed runbook fragment. Treat it as an example schema, not a production dump from my pager.

# proposed example: incident freeze policy
apiVersion: oncall.example/v1
kind: Runbook
metadata:
  id: checkout-latency
spec:
  alert_match:
    service: checkout
    severity: ["sev1", "sev2"]
  freeze_policy:
    arm_on:
      - class: mutate
        once: true
    frozen_verbs: [restart, scale, drain, rollback, apply, patch]
    read_allow: [logs, metrics, describe, trace, kubectl_get]
    unfreeze:
      actors: ["incident-commander", "service-owner"]
      require:
        - human_ack: true
        - evidence: ["graph_link", "change_ticket"]
        - scope: service  # not cluster-wide
      ttl_seconds: 1800
      rearm_on_mutate: true

Notice the last line. If unfreeze lets one more mutation through, the freeze arms again. Why would you unfreeze forever after a single ack? Humans get interrupted. Models get chatty. The flag should be sticky by default.

A freeze gate you can run locally

The runbook is theater unless a process actually rejects the next command. I want a gate that sits in front of the executor, not inside the prompt. Prompts forget. Files and locks forget less often.

This is proposed Python, labeled because I am not claiming a measured production rollout here. It keeps incident freeze state on disk so two bot workers cannot disagree by accident.

# proposed: freeze_gate.py
from __future__ import annotations

import json
import time
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Literal

VerbClass = Literal["read", "mutate"]

MUTATING = {"restart", "scale", "drain", "rollback", "apply", "patch"}

@dataclass
class FreezeState:
    incident_id: str
    frozen: bool
    armed_at: float | None
    unfreeze_until: float | None
    last_reason: str

class FreezeGate:
    def __init__(self, path: Path):
        self.path = path
        self.path.parent.mkdir(parents=True, exist_ok=True)

    def _load(self, incident_id: str) -> FreezeState:
        if not self.path.exists():
            return FreezeState(incident_id, False, None, None, "init")
        raw = json.loads(self.path.read_text())
        data = raw.get(incident_id) or {}
        return FreezeState(
            incident_id=incident_id,
            frozen=bool(data.get("frozen")),
            armed_at=data.get("armed_at"),
            unfreeze_until=data.get("unfreeze_until"),
            last_reason=data.get("last_reason", ""),
        )

    def _save(self, state: FreezeState) -> None:
        raw = json.loads(self.path.read_text()) if self.path.exists() else {}
        raw[state.incident_id] = asdict(state)
        self.path.write_text(json.dumps(raw, indent=2, sort_keys=True))

    def classify(self, verb: str) -> VerbClass:
        return "mutate" if verb in MUTATING else "read"

    def allow(self, incident_id: str, verb: str) -> tuple[bool, str]:
        state = self._load(incident_id)
        now = time.time()
        if state.unfreeze_until and now > state.unfreeze_until:
            state.frozen = True
            state.unfreeze_until = None
            state.last_reason = "ttl_expired_rearm"
            self._save(state)
        kind = self.classify(verb)
        if kind == "read":
            return True, "read_allow"
        if state.frozen:
            return False, f"frozen:{state.last_reason}"
        return True, "mutate_open"

    def note_success(self, incident_id: str, verb: str) -> None:
        if self.classify(verb) != "mutate":
            return
        state = self._load(incident_id)
        state.frozen = True
        state.armed_at = time.time()
        state.unfreeze_until = None
        state.last_reason = f"armed_after:{verb}"
        self._save(state)

    def unfreeze(
        self,
        incident_id: str,
        actor: str,
        evidence: list[str],
        ttl_seconds: int,
        allowed_actors: set[str],
    ) -> tuple[bool, str]:
        if actor not in allowed_actors:
            return False, "actor_rejected"
        if not {"graph_link", "change_ticket"}.issubset(set(evidence)):
            return False, "evidence_incomplete"
        state = self._load(incident_id)
        now = time.time()
        state.frozen = False
        state.unfreeze_until = now + ttl_seconds
        state.last_reason = f"unfrozen_by:{actor}"
        self._save(state)
        return True, "unfrozen"

Wire it in front of whatever executes shell. The bot may still narrate. Narration is cheap. apply is not.

# proposed: executor.py
FORBIDDEN_EXIT = 78  # EX_CONFIG, useful for pagers

def run_command(gate: FreezeGate, incident_id: str, verb: str, argv: list[str]) -> int:
    ok, reason = gate.allow(incident_id, verb)
    print(f"gate allow={ok} reason={reason} verb={verb}")
    if not ok:
        return FORBIDDEN_EXIT
    # subprocess.run(argv, check=False)  # hook your real executor here
    gate.note_success(incident_id, verb)
    return 0

Human unfreeze is a command, not a model reply. I want that on the timeline.

# proposed operator commands
python -m oncall_freeze unfreeze \
  --incident INC-2026-0917 \
  --actor incident-commander \
  --evidence graph_link --evidence change_ticket \
  --ttl 1800

python -m oncall_freeze status --incident INC-2026-0917

If your chat tool cannot emit that command, the chat tool is not the control plane. Stop asking it to be one.

Decision table for the next five minutes

I use a table during review because tables survive copy-paste better than paragraphs. This is the proposed matrix I want in the runbook footer.

Situation	Verb	Freeze state	Result
Fresh incident, no mutation yet	`describe`	open	allow
Fresh incident, no mutation yet	`restart`	open	allow, then arm freeze
After first restart	`logs`	frozen	allow
After first restart	`scale`	frozen	reject
Human unfreeze with ticket + graph	`rollback`	unfrozen, TTL live	allow once, then rearm
Unfreeze TTL expired	`apply`	frozen again	reject
Random engineer, no role	`unfreeze`	any	reject
Model proposes unfreeze in prose	n/a	any	ignore, not a transition

Would you let the model tick that last row? I would not, because prose is not a signature. If the freeze file did not change, nothing unfroze.

A test plan you can execute without production

I do not want a demo that only works while the author watches the terminal. This is a proposed local test plan. Run it against a temp state file.

Start with an empty freeze store and incident INC-TEST.
Send describe; expect allow and no freeze arm.
Send restart; expect allow, then frozen=true.
Send scale; expect reject with frozen:armed_after:restart.
Send logs; expect allow while still frozen.
Call unfreeze as intern with no evidence; expect actor_rejected or evidence_incomplete.
Call unfreeze as incident-commander with both evidence keys and ttl_seconds=2.
Send rollback immediately; expect allow, then freeze rearm.
Sleep past TTL if you unfroze without a mutate; expect the next mutate to reject.
Crash the process and reload the JSON file; expect the same freeze bit.

# proposed: test_freeze_gate.py
from pathlib import Path
from freeze_gate import FreezeGate

def test_restart_arms_and_blocks_scale(tmp_path: Path):
    gate = FreezeGate(tmp_path / "freeze.json")
    ok, _ = gate.allow("INC-TEST", "restart")
    assert ok
    gate.note_success("INC-TEST", "restart")
    ok, reason = gate.allow("INC-TEST", "scale")
    assert not ok
    assert reason.startswith("frozen:")

If step ten fails, you do not have a freeze rule. You have a variable in RAM, and RAM is not an incident record.

Where a free coding workspace actually helps

I draft freeze predicates and the reject reasons as code first, then I ask a model to nitpick missing verbs. That is a writing aid, not an executor. Disclosure: This article was prepared as part of MonkeyCode's product outreach. When I need a scratch place to iterate on the gate and the YAML without standing up extra infra, MonkeyCode's free model access and free server option are enough to run the checker against sample alerts. The freeze file still lives in my repo. The pager still belongs to a human.

Keep the model on the read path while freeze is armed. Let it summarize logs. Let it propose an unfreeze packet. Do not let it sign the packet. If that split feels inconvenient, good. Inconvenience is the point of a change freeze.

Limitations, and who should not use this

This approach fails in several honest ways, and I want those listed beside the code. Multiple bot replicas need a real lock or a single writer, because a JSON file is a teaching store, not a cluster database. Clock skew can expire unfreeze windows early or late, so NTP is part of the runbook. Freeze does not stop a human with kubectl on a laptop. If your real risk is shadow access, encode IAM, not YAML.

Do not use this if you have no mutating on-call bot. A human-only rotation already has change freezes through tickets, and extra flags just add theater. Do not use this as a substitute for blast-radius limits. A freeze after the first bad restart does not rewind the restart. Do not use this in regulated change windows that already require two-person review unless you wire unfreeze to that same review system. A homemade actor string is not an audit trail.

There is also a product limit I will not paper over. Free model access and a free server do not make the freeze correct. They only make the draft loop cheaper. Incorrect verbs in frozen_verbs will fail open for anything you forgot to name. Review the verb set like you review IAM actions.

Put the flag where the executor can see it

I want the freeze bit next to the incident id, not inside a paragraph the model can paraphrase away. Alerts still fire. First commands still need an allowlist. Escalation still needs an owner. After the first mutation, though, the only kind question is whether the next write is even legal.

If you already run an on-call bot, add the freeze file this week and prove the reject path with the test plan above. If the bot cannot fail that scale call on purpose, it will eventually succeed at the worst time. Do you want that success on your timeline, or do you want a boring exit 78?

Incomplete Alerts Are a Trap: Make the On-Call Bot Fail Closed

Emery Huang — Sat, 05 Sep 2026 19:58:23 +0000

If a paging alert is missing a host, service, or freeze flag, your on-call bot should stop, not improvise. I keep seeing agents fill those holes with guesses that look confident and still page the wrong people. The safer pattern is a typed alert contract, a read-only first command list, and a freeze latch only a human can lift. Everything below is a proposed workflow you can run locally; treat the code as a starting kit, not production gospel.

Have you watched a model invent a hostname because the payload looked almost complete? That is the failure I want to kill this week. Current agent writeups keep circling the same wound: tools assume missing context, then act. On-call is the worst place for that habit, because a missing field is not a creative prompt. It is a reason to refuse the page.

Why incomplete pages hurt more than late pages

A late page still points at a real service, a real host, and a real owner. An incomplete page sends someone into a fog with a critical severity sticker glued on top. I would rather the bot reply INCOMPLETE_ALERT in Slack than SSH into a guessed box. Does that feel slow during a real incident? Yes, and that slowness is the point of a contract.

Here is the rule I want at the top of the runbook, before any model sees the payload:

Required fields must exist, parse, and match an allowlist.
Missing fields fail closed; the bot does not complete them.
First commands are read-only probes, never restarts or deploys.
Freeze is a latch on disk; unfreeze is a human action with a reason.
Escalation follows a table of severity, completeness, and freeze state.

If those five bullets already feel strict, good. On-call bots do not need more imagination. They need fewer chances to lie.

A typed alert contract, not a friendlier prompt

I do not start with a system prompt that says “be careful.” I start with a schema the process can reject before any model call. The contract below is proposed Python, not a war story from a specific outage, and you should swap the enums for your own services.

# alert_contract.py — proposed local validator, not a hosted service
from __future__ import annotations

from datetime import datetime, timezone
from enum import Enum
from typing import Optional

from pydantic import BaseModel, Field, ValidationError, field_validator

class Severity(str, Enum):
    INFO = "info"
    WARN = "warn"
    CRITICAL = "critical"

class Service(str, Enum):
    CHECKOUT = "checkout"
    PAYMENTS = "payments"
    INGRESS = "ingress"

ALLOWED_HOST_SUFFIXES = (".prod.internal", ".staging.internal")

class AlertContract(BaseModel):
    alert_id: str = Field(min_length=8, max_length=64)
    service: Service
    severity: Severity
    host: str
    runbook_id: str
    freeze_hint: bool
    received_at: datetime
    page_owner: str
    extra: Optional[dict] = None

    @field_validator("host")
    @classmethod
    def host_must_be_ours(cls, value: str) -> str:
        if not value.endswith(ALLOWED_HOST_SUFFIXES):
            raise ValueError("host is not in the allowlist")
        if any(ch.isspace() for ch in value):
            raise ValueError("host cannot contain whitespace")
        return value

    @field_validator("received_at")
    @classmethod
    def must_be_aware(cls, value: datetime) -> datetime:
        if value.tzinfo is None:
            raise ValueError("received_at must include a timezone")
        return value


def parse_alert(payload: dict) -> AlertContract:
    try:
        return AlertContract.model_validate(payload)
    except ValidationError as exc:
        raise ValueError(f"INCOMPLETE_ALERT: {exc.error_count()} field issues") from exc

Notice what the model never gets to do here. It never invents payments-2.prod.internal because the JSON key was blank. It never upgrades warn to critical because the description sounded scary. If the contract fails, the bot stops. Why would we even call a model before that gate?

First commands that cannot mutate anything

After the contract passes, I still do not want a shell that can restart a fleet. I want a tiny allowlist of read-only probes keyed by service. Anything else is a human decision, including the tempting one-liner that “just bounces the pod.”

# first_commands.py — proposed allowlist
READ_ONLY = {
    "checkout": [
        "kubectl -n checkout get pods -o wide",
        "curl -fsS https://checkout.prod.internal/healthz",
        "journalctl -u checkout --since -10m --no-pager | tail -n 80",
    ],
    "payments": [
        "kubectl -n payments get pods -o wide",
        "curl -fsS https://payments.prod.internal/healthz",
        "redis-cli -h payments-redis.prod.internal ping",
    ],
    "ingress": [
        "kubectl -n ingress get pods -o wide",
        "curl -fsS https://ingress.prod.internal/healthz",
        "ss -ltnp | grep -E ':443|:80'",
    ],
}

FORBIDDEN_TOKENS = ("restart", "delete", "scale", "rollout undo", "drain", "reboot")


def first_commands(service: str) -> list[str]:
    cmds = READ_ONLY[service]
    for cmd in cmds:
        lowered = cmd.lower()
        if any(token in lowered for token in FORBIDDEN_TOKENS):
            raise RuntimeError(f"mutating token in allowlist: {cmd}")
    return cmds

I like that this file can fail CI if somebody sneaks rollout restart into the probe list. Would you rather debate that in a postmortem, or in a unit test on a Tuesday afternoon? I know my answer.

Freeze is a latch; unfreeze is a human act

A freeze window is not a sentence in the prompt. It is a latch the bot can read and cannot lift. I keep it as a small JSON file so a human, or a locked change ticket, is the only writer. The bot may notice freeze, and it may refuse mutating advice, but it does not get to declare the freeze over because error rates dropped.

# freeze_latch.py — proposed local latch
import json
from pathlib import Path

LATCH = Path("/var/oncall/freeze.json")


def freeze_state() -> dict:
    if not LATCH.exists():
        return {"frozen": False, "reason": "no-latch", "by": None}
    data = json.loads(LATCH.read_text())
    return {
        "frozen": bool(data.get("frozen")),
        "reason": data.get("reason") or "unspecified",
        "by": data.get("by"),
    }


def assert_can_page(severity: str) -> None:
    state = freeze_state()
    if state["frozen"] and severity != "critical":
        raise PermissionError(
            f"FROZEN: {state['reason']} (set by {state['by']}); non-critical pages are suppressed"
        )


def human_unfreeze(actor: str, reason: str) -> None:
    if not actor or not reason or len(reason) < 12:
        raise ValueError("unfreeze requires an actor and a real reason")
    LATCH.write_text(json.dumps({
        "frozen": False,
        "reason": reason,
        "by": actor,
    }, indent=2))

Critical alerts can still page during freeze, because a payments outage does not wait for the marketing freeze to end. Everything else waits. Who should hold the unfreeze key? A human on the change calendar, not the same process that wants to look helpful.

Escalation is a table, not a vibe

I do not want the bot to “use its judgment” about who to wake. I want a boring table that a new hire can read at 03:00. Completeness is a first-class input, sitting beside severity and freeze, because an incomplete critical alert is an investigation, not a page storm.

Severity	Contract	Freeze	Action
info	valid	any	ticket only, no page
warn	valid	frozen	ticket only
warn	valid	open	page secondary after 15 minutes
critical	valid	any	page primary now
any	invalid	any	do not page; post `INCOMPLETE_ALERT` and wake the on-call only if it repeats

# escalate.py — proposed decision table
from dataclasses import dataclass

@dataclass
class Decision:
    page_primary: bool
    page_secondary: bool
    ticket_only: bool
    note: str


def decide(severity: str, valid: bool, frozen: bool) -> Decision:
    if not valid:
        return Decision(False, False, True, "INCOMPLETE_ALERT")
    if severity == "info":
        return Decision(False, False, True, "info stays on the ticket board")
    if severity == "warn" and frozen:
        return Decision(False, False, True, "warn suppressed during freeze")
    if severity == "warn":
        return Decision(False, True, True, "warn pages secondary after delay")
    return Decision(True, False, True, "critical pages primary even when frozen")

Ask yourself the uncomfortable question before you wire this to a pager. If the alert is critical and the host field is empty, do you still want a 03:00 phone call? I do not, unless the same incomplete fingerprint repeats twice. Repeats are a signal. One broken exporter is noise.

A reproducible test plan, not a vibe check

I want tests that fail on a laptop, without a cluster and without a model bill. Label these as unexecuted examples until you run them. They encode the contract, not my personal production metrics, because I am not inventing a customer outage to sell the pattern.

# test_oncall_contract.py
from datetime import datetime, timezone
import pytest

from alert_contract import parse_alert
from escalate import decide
from first_commands import first_commands
from freeze_latch import assert_can_page

NOW = datetime.now(timezone.utc)

BASE = {
    "alert_id": "alt-9f3c12ab",
    "service": "payments",
    "severity": "critical",
    "host": "pay-1.prod.internal",
    "runbook_id": "rb-payments-latency",
    "freeze_hint": False,
    "received_at": NOW.isoformat(),
    "page_owner": "payments-primary",
}

def test_rejects_blank_host():
    payload = dict(BASE, host="")
    with pytest.raises(ValueError, match="INCOMPLETE_ALERT"):
        parse_alert(payload)

def test_rejects_unknown_suffix():
    payload = dict(BASE, host="pay-1.random.cloud")
    with pytest.raises(ValueError, match="INCOMPLETE_ALERT"):
        parse_alert(payload)

def test_first_commands_stay_read_only():
    for cmd in first_commands("payments"):
        assert "restart" not in cmd.lower()

def test_incomplete_never_pages_primary():
    d = decide("critical", valid=False, frozen=False)
    assert d.page_primary is False
    assert d.note == "INCOMPLETE_ALERT"

def test_warn_does_not_page_when_frozen(tmp_path, monkeypatch):
    latch = tmp_path / "freeze.json"
    latch.write_text('{"frozen": true, "reason": "release freeze", "by": "sre"}')
    monkeypatch.setattr("freeze_latch.LATCH", latch)
    with pytest.raises(PermissionError, match="FROZEN"):
        assert_can_page("warn")

Run it like this, then paste a broken payload from last week’s alert tool and watch it fail closed:

python -m pip install pydantic pytest
python -m pytest -q test_oncall_contract.py

If a test fails because your real hostnames use a different suffix, that is a gift. Change the allowlist. Do not loosen the parser until a model can “figure it out.”

Where a free coding workspace actually fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a measured speedup, a model name, or a quota. I am saying the boring part of this workflow is rewriting the contract and the tests until they hurt in the right places. MonkeyCode’s free model access and free server option are useful there, because you can iterate the validator on a scratch box without pointing it at production kubeconfigs. Keep secrets off that box. Keep the latch file in a repo your on-call humans already trust.

The product is not the runbook. The contract is the runbook. If you strip every product name out of this article, you should still have a schema, an allowlist, a latch, and a table you can argue about in a review.

Limitations, and who should not do this

This approach is wrong for several honest reasons, and I want those on the same page as the code. A fail-closed contract will drop pages when exporters omit fields, which is safer than guessing and still dangerous if your monitoring is sloppy. Read-only first commands will not restore a dead shard. A file-based freeze latch is not a distributed lock, and two writers can still race if you skip real change control.

Do not use this if you lack paging authority, if you cannot name an allowlist of hosts, or if a regulator expects every action to go through an already approved runbook tool. Do not point a free server at production credentials. Do not let a model write the unfreeze reason after the fact. And do not copy this table into a bank or hospital pager without a human review that I cannot do for you in a blog post.

I also will not pretend this matches every stack. If your alerts arrive as free-form email, you need a parser in front of the contract, and that parser should fail closed too. If your freeze calendar lives in a vendor, read it through an API you already audit, not through a scraped screenshot.

What I want you to try on the next noisy afternoon

Take one real alert JSON from staging, strip the host, and run the tests. Then put the host back, set the freeze latch, and confirm a warn does not page. After that, print the first commands and ask a teammate if any line could mutate state. If the answer is yes, you do not need a smarter agent. You need a smaller allowlist.

If you want a scratch place to hammer on those tests without waking a pager, MonkeyCode’s free server option is enough for a dry run. Keep the disclosure in mind, keep production keys off the box, and let the contract stay stricter than the model. That is the whole lesson, and it still holds if you never touch the product at all.