Split Freeze From Unfreeze, and Map Each Alert to a Read-Only Command

A paging channel without a freeze rule is just a group chat that happens to be loud. I will not treat an on-call runbook as live until every alert maps to a first command, an escalation tree names a human, and freeze is a different checklist from unfreeze. Can a model write fluent incident prose at two in the morning? Yes, and that is exactly why I keep it away from those fields.

Quiet services still page someone, and that someone still needs a command that will not write. If you skip the human sections, the next page becomes a brainstorm with extra confidence. Have you watched a channel fill with suggested kubectl lines while nobody names a freeze owner? I have, and I will not pretend a generated paragraph is a substitute for that owner.

Why pretty runbooks still fail the first page

Most half-finished runbooks fail in the same places, and none of those places are grammar. The alert does not name a service owner who can answer the phone. The first commands are write-heavy, so the first keystroke mutates production. The escalation path is a Slack handle that nobody pages after midnight. Freeze and unfreeze share one vague sentence, so nobody knows who may lift the gate.

AI-assisted drafting makes those gaps prettier, not smaller. Why would a model know your after-hours phone tree? Why would it know which replica is allowed to restart without a freeze? It will invent confident steps unless you refuse to let it own those fields. The useful work is still naming owners, commands, and gates in daylight, then leaving the model a narrow drafting job.

I still use a drafting box for symptom text, because writing that prose by hand is slow and easy to postpone. Disclosure: This article was prepared as part of MonkeyCode's product outreach. When I need a quiet place to draft non-authoritative sections, MonkeyCode's free model access and free server option are enough to generate candidate paragraphs that I then strip of any write action.

The human sections the pager actually needs

Treat the runbook as a contract the pager can parse, not as a blog post about the service. I keep these sections in a checked-in file, and I reject drafts that leave any of them as prose-only comments.

1. Alert catalog — each paging alert has an id, a service, a severity, and a human owner.
2. First-command map — each alert id points at read-only commands, with an explicit deny list for writes.
3. Escalation tree — primary, secondary, and manager, with a timeout that is a number, not a vibe.
4. Freeze checklist — who may freeze writes, what evidence they record, and what stays allowed.
5. Unfreeze checklist — a different owner path, a different evidence set, and a named blast check.

Notice freeze and unfreeze are not one section with a reversible verb. If they share a heading, people will unfreeze with the same confidence they used to freeze. Do you want that person to be a model that never saw your change window? I do not.

Artifact: a runbook file the pager can fail closed

This is a proposed schema, not a war story with fake metrics. Drop it in runbooks/payments-api.yaml and refuse to page off a document that cannot load.

# Proposed template — label every field human-owned or draft-only.
apiVersion: oncall.example/v1
kind: Runbook
metadata:
  service: payments-api
  owners: ["primary-oncall", "payments-tl"]
  timezone: America/Los_Angeles
spec:
  alerts:
    - id: ALRT-PAY-429
      name: checkout_latency_p99
      severity: page
      owner: "payments-primary"
      first_commands:
        - "kubectl --context=prod-ro -n payments get deploy payments-api -o wide"
        - "kubectl --context=prod-ro -n payments get pods -l app=payments-api"
        - "curl -sS https://status.internal/payments/health | jq ."
      deny_commands:
        - "kubectl delete"
        - "kubectl scale"
        - "kubectl apply"
        - "helm upgrade"
      escalate_after_minutes: 10
  escalation:
    primary: { role: "payments-primary", timeout_minutes: 10 }
    secondary: { role: "payments-secondary", timeout_minutes: 15 }
    manager: { role: "payments-tl", timeout_minutes: 20 }
    never: ["@here", "random model chat"]
  freeze:
    required_before_writes: true
    owner_role: "incident-commander"
    evidence:
      - "alert_id"
      - "first_command_output_hash"
      - "blast_radius_service_list"
    allowed_during_freeze: ["read", "page", "capture"]
  unfreeze:
    owner_role: "incident-commander"
    second_signer_role: "service-owner"
    evidence:
      - "alert_id_cleared_or_mitigated"
      - "write_plan_printed"
      - "rollback_command"
    forbidden_if_missing: ["second_signer", "rollback_command"]

Would I let a model fill deny_commands from memory of some other cluster? No, because that list is how you keep the first five minutes read-only. The model may draft the health-check narrative that sits under the YAML. It does not get to invent the owner role or the second signer.

A validator that fails before the rotation starts

I want the runbook to fail in CI, not in the paging channel. The script below is a proposed check you can run locally; it does not talk to production and it does not claim a benchmark.

#!/usr/bin/env python3
"""validate_runbook.py — proposed structural check, not an incident replay."""
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("pip install pyyaml\n")
    sys.exit(2)

WRITE_TOKENS = ("delete", "scale", "apply", "upgrade", "restart", "exec --")


def fail(msg: str) -> None:
    sys.stderr.write(msg + "\n")
    sys.exit(1)


def main(path: str) -> None:
    data = yaml.safe_load(Path(path).read_text())
    spec = data.get("spec") or {}
    alerts = spec.get("alerts") or []
    if not alerts:
        fail("no alerts: a runbook that cannot match a page is a wiki page")

    freeze = spec.get("freeze") or {}
    unfreeze = spec.get("unfreeze") or {}
    if not freeze or not unfreeze:
        fail("freeze and unfreeze must both exist as mappings")
    if freeze.get("owner_role") and freeze.get("owner_role") == unfreeze.get("owner_role"):
        # Same person may hold both hats, but the checklists cannot be identical.
        if freeze.get("evidence") == unfreeze.get("evidence"):
            fail("freeze evidence must not equal unfreeze evidence")
    if not unfreeze.get("second_signer_role"):
        fail("unfreeze needs a second signer role")

    esc = spec.get("escalation") or {}
    for key in ("primary", "secondary", "manager"):
        node = esc.get(key) or {}
        if not node.get("role") or not node.get("timeout_minutes"):
            fail(f"escalation.{key} needs role and timeout_minutes")

    for alert in alerts:
        aid = alert.get("id") or "<missing-id>"
        if not alert.get("owner"):
            fail(f"{aid}: alert has no human owner")
        cmds = alert.get("first_commands") or []
        if not cmds:
            fail(f"{aid}: no first commands")
        for cmd in cmds:
            low = cmd.lower()
            if any(tok in low for tok in WRITE_TOKENS):
                fail(f"{aid}: first command looks like a write: {cmd}")
        if not alert.get("deny_commands"):
            fail(f"{aid}: deny_commands missing")

    print(f"ok: {path} ({len(alerts)} alerts)")


if __name__ == "__main__":
    if len(sys.argv) != 2:
        fail("usage: python3 validate_runbook.py runbooks/payments-api.yaml")
    main(sys.argv[1])

Run it like this, on a laptop, before you accept a week of pages:

python3 validate_runbook.py runbooks/payments-api.yaml
# expected: ok: runbooks/payments-api.yaml (1 alerts)

If the validator prints ok while unfreeze.evidence still equals freeze.evidence, I missed a check and I want you to fail the file by hand. Should a green script be enough to unfreeze production? No. It is only enough to prove the document is not empty.

First commands stay read-only until freeze is named

I keep a tiny wrapper so the tired person at the keyboard cannot “just scale it” from muscle memory. This is a proposed shell gate, not a cluster agent.

#!/usr/bin/env bash
# firstcmd.sh — proposed read-only wrapper. Do not point this at a write kubecontext.
set -euo pipefail
ALERT_ID="${1:?alert id}"
RUNBOOK="${2:-runbooks/payments-api.yaml}"
CONTEXT="${KUBE_RO_CONTEXT:?set KUBE_RO_CONTEXT to a read-only context}"

if [[ "$CONTEXT" == *prod-w* || "$CONTEXT" == *writable* ]]; then
  echo "refusing: context looks writable: $CONTEXT" >&2
  exit 1
fi

echo "alert=$ALERT_ID runbook=$RUNBOOK context=$CONTEXT"
echo "freeze_named=${FREEZE_NAMED:-no}"
if [[ "${FREEZE_NAMED:-no}" != "yes" ]]; then
  echo "writes remain blocked until FREEZE_NAMED=yes and the freeze checklist is signed"
fi

What belongs in first_commands? Gets, describes, logs with a tail limit, and health URLs that cannot mutate. What does not belong? Anything that restarts, scales, applies, deletes, or opens an interactive shell on a prod pod. If the model drafts a “quick restart” as step one, that is a defect in the draft, not a shortcut.

Escalation is a tree with minutes, not a mention

I want numbers on the tree because “ping secondary if needed” is how pages rot in a thread. Primary has ten minutes. Secondary has fifteen. Manager has twenty. After that, you are not brainstorming; you are late.

• Do not escalate to @here.
• Do not escalate to a model chat that cannot take the phone.
• Do not list a person who is already the freeze owner unless a second signer still exists for unfreeze.
• Do record the time you moved to the next role, even if the next role is you with a different hat.

If your team is two people, the tree can still exist. It just becomes honest about how thin the bench is. Is a thin bench a reason to skip the tree? It is a reason to write the tree larger than the bench, then staff it later.

Freeze and unfreeze as two checklists

Use a decision table during the incident, on paper or in the channel topic. The table is the artifact that keeps freeze from becoming a mood.

Question	Freeze	Unfreeze
Who signs?	Incident commander	Commander plus service owner
What evidence?	Alert id, first-command output, blast list	Alert mitigated, printed write plan, rollback
Writes allowed?	No, except documented break-glass	Only the printed plan
Model output allowed as proof?	No	No
Can the same human hold both hats?	Yes, if the evidence sets differ	Only with a second signer

I will freeze as soon as a write is tempting and the blast list is still unnamed. I will not unfreeze because a generated summary says the error rate “looks better.” Looks better than what baseline, captured by which command, against which alert id? If you cannot answer that, the freeze stays.

Where a drafting model is allowed, and where it is not

Let the model propose symptom text, dashboard links you already host, and questions to ask the primary. Do not let it propose owners, phone numbers, deny lists, or unfreeze signers. Those fields are boring on purpose, because boring fields are the ones people skip when a page is loud.

If you paste a generated runbook straight into the wiki, you have a document that reads finished and behaves empty. That is the failure mode I am trying to make expensive. The validator is cheap. The second signer on unfreeze is cheaper than a write that nobody can roll back.

Limitations, and who should not use this

This approach assumes you already have paging, a read-only kube context or equivalent, and a place to store YAML that humans review. It does not measure latency, does not replay traffic, and does not prove the first command is the right command for a novel failure. The schema will happily accept a wrong-but-read-only curl if you typed it.

Do not use this if you are a solo hobby project with no pager and no production writes worth freezing. Do not use it as a reason to skip a real staging reproduction when you have time. Do not use a drafting server as the system of record for phone trees or credentials. Do not treat free model access as an on-call teammate; it cannot take the escalation slot, and it cannot sign unfreeze.

If your incident process already requires two-person review for every production write, you may only need the alert-to-command map and the validator. That is fine. The point is not to collect headings. The point is to make the first five minutes of a page boring enough that nobody invents a write.