I Will Not Unfreeze Prod Until the Runbook Hash Matches the Page

I will not let an assistant invent the incident path while the pager is still screaming. A usable on-call runbook names the alert class, the first command, the escalation owner, and the freeze hash before anyone types. Chat logs are not runbooks, and generated shells are not change tickets, no matter how confident they sound. If those four fields are missing, I keep writes frozen and I keep the assistant on a scratch box.

Why am I this stubborn about a document that looks like YAML? Because the last seven days of AI-coding talk keep confusing fluency with ownership, and on-call work punishes that mix. A model can draft a plausible restart in twenty seconds, then miss the replica that actually holds the lock. Would you rather argue with a chat transcript at 03:00, or execute a hashed runbook you already signed?

The contract I actually page against

I treat every page as a four-field contract, not as a brainstorming session with extra urgency. The contract is boring on purpose, and boring is what I want when the graph is red. If a field cannot be filled from the alert payload and the repo, I do not improvise a fifth field in Slack.

Here is the schema I keep in runbooks/<service>.yml. Copy it, then refuse to run writes until freeze.hash matches the file on disk.

# runbooks/payments-api.yml
apiVersion: oncall.example.com/v1
service: payments-api
owner_oncall: payments-primary
escalation:
  t_plus_15m: payments-secondary
  t_plus_30m: payments-manager
  t_plus_45m: incident-commander
alert_classes:
  - id: PAYMENTS_P99_LATENCY
    severity: page
    observe_commands:
      - id: oc1
        argv: ["kubectl", "-n", "payments", "get", "deploy", "payments-api", "-o", "wide"]
      - id: oc2
        argv: ["kubectl", "-n", "payments", "top", "pod", "-l", "app=payments-api"]
    write_commands: []   # stays empty until unfreeze
freeze:
  state: frozen
  hash_of: runbooks/payments-api.yml
  require:
    - named_blast_radius
    - matching_git_sha
    - human_signature

Notice what is missing on purpose: there is no prompt, no vibe, and no "try whatever the model suggested." Observe commands are argv arrays, not English. Write commands start empty. The freeze block is data, not a pep talk.

Field 1: alert class, not a paragraph

I bind the pager to a stable alert_class id, because humans rename dashboards and models paraphrase titles. PAYMENTS_P99_LATENCY is allowed to wake me; "the site feels slow" is not. If the alert cannot map onto one id in the YAML, I treat it as observe-only noise until a human classifies it.

• Page-worthy classes get severity: page and a short observe list.
• Ticket-worthy classes get severity: ticket and never unfreeze writes.
• Unknown classes inherit freeze and a required escalation note.

Can your current alert rule print that id into the notification body? If it cannot, the runbook is already lying before you open a terminal.

# example Alertmanager annotation, not a prompt
description: class=PAYMENTS_P99_LATENCY ns=payments deploy=payments-api

Field 2: first commands stay read-only

My first commands are inventory, not remediation. I want the replica count, the ready condition, the recent events, and the error budget burn, in that order. Anything that mutates pods, feature flags, or DNS waits behind the freeze hash. Have you ever watched a generated one-liner delete the healthy deployment because the label selector was almost right?

I keep a tiny allowlist checker next to the YAML so the laptop refuses clever extra flags.

# tools/check_first_commands.py
import sys, yaml

ALLOWED = {
    "kubectl": {"get", "describe", "logs", "top", "api-resources"},
    "dig": {"+short"},
    "curl": {"-sS", "-o", "/dev/null", "-w"},
}

def main(path):
    doc = yaml.safe_load(open(path))
    errors = []
    for cls in doc["alert_classes"]:
        for cmd in cls["observe_commands"]:
            argv = cmd["argv"]
            bin_ = argv[0]
            if bin_ not in ALLOWED:
                errors.append(f"{cls['id']}: binary {bin_} not in observe allowlist")
                continue
            if any(tok in argv for tok in ("delete", "apply", "patch", "scale", "drain")):
                errors.append(f"{cls['id']}: mutating token in observe argv {argv}")
        if cls.get("write_commands"):
            errors.append(f"{cls['id']}: write_commands must be empty while frozen")
    if doc.get("freeze", {}).get("state") != "frozen":
        errors.append("freeze.state must start as frozen")
    if errors:
        print("\n".join(errors)); sys.exit(1)
    print(f"ok: {path} observe-only and frozen")

if __name__ == "__main__":
    main(sys.argv[1])

Run it before the page, not during it.

python3 tools/check_first_commands.py runbooks/payments-api.yml
sha256sum runbooks/payments-api.yml | tee runbooks/payments-api.yml.sha256

Field 3: escalation is a clock, not a group chat

I write escalation as timestamps against the first page, because "ask in Slack" is how incidents grow extra owners and zero decisions. Fifteen minutes of observe with no named blast radius promotes to secondary. Thirty minutes promotes to the manager. Forty-five minutes asks for an incident commander who is not also typing kubectl. Who is allowed to unfreeze if the primary is the person who wrote the bad deploy?

1. T+0 to T+15: primary runs observe commands only, pastes command ids into the incident doc.
2. T+15: secondary joins, repeats the same observe ids, and challenges any extra argv.
3. T+30: manager confirms customer impact and names the blast radius in one sentence.
4. T+45: commander owns communication; primary still cannot unfreeze without the hash check.

I do not escalate because I feel nervous. I escalate because the clock in the runbook elapsed and a field is still blank.

Field 4: freeze hash, then a signed unfreeze

This is the rule that keeps assistants useful instead of dangerous. The freeze hash is sha256 of the runbook file at the git sha I am paging from. Unfreeze is a separate artifact, not a vibes-based "looks good." If the assistant rewrites the YAML during the call, the hash breaks, and writes stay frozen. Is that annoying? Yes. Is it worse than a silent kubectl apply from a regenerated plan? Not even close.

# tools/unfreeze.sh — still a proposal until a human runs it on a signed laptop
set -euo pipefail
RB="${1:?runbook yml}"
INCIDENT="${2:?incident id}"
BLAST="${3:?blast radius sentence}"
SIG="${4:?path to signature file}"

expected=$(cut -d' ' -f1 "${RB}.sha256")
actual=$(sha256sum "$RB" | cut -d' ' -f1)
test "$expected" = "$actual" || { echo "hash mismatch: freeze holds"; exit 2; }
test -s "$SIG" || { echo "no human signature"; exit 3; }
test -n "$BLAST" || { echo "blast radius unnamed"; exit 4; }

printf 'unfrozen\nincident=%s\nblast=%s\n' "$INCIDENT" "$BLAST" > "${RB}.unfreeze"
echo "writes allowed only for commands listed after this file exists"

I still require write commands to be appended as argv arrays after unfreeze, never as free text. The assistant may propose a patch on a scratch clone. It does not get to flip freeze.state.

Where a free coding assistant actually helps

I will use a coding assistant to draft the YAML, generate the allowlist tests, and argue with my own escalation clock, but only on a throwaway box. MonkeyCode is relevant here because free model access and a free server option give me that scratch loop without borrowing a production jumphost. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I paste the schema, the checker, and a redacted alert sample, then I ask for missing observe commands that still fit the allowlist. I do not paste kubeconfigs, customer payloads, or unfreeze signatures. If the model invents a rollout restart, the checker fails, and that failure is the lesson. The free server is for running check_first_commands.py, not for talking to prod.

# labeled as a proposal I would run on a scratch server, not on prod
python3 tools/check_first_commands.py runbooks/payments-api.yml
python3 -c "import yaml,sys; yaml.safe_load(open(sys.argv[1]))" runbooks/payments-api.yml

Would I let the same session open a tunnel to the cluster? No. The runbook is the contract; the assistant is a typist with a linter.

A dry-run I actually practice

I rehearse with a fake page so the first real page is not also the first parse of the YAML. The drill is short, scripted, and mean about extra commands.

export INCIDENT=inc-drill-2026-09-17
export ALERT_CLASS=PAYMENTS_P99_LATENCY
python3 tools/check_first_commands.py runbooks/payments-api.yml
# observe only — replace with your read-only kube context
kubectl --context scratch -n payments get deploy payments-api -o wide
kubectl --context scratch -n payments top pod -l app=payments-api
# stop here unless tools/unfreeze.sh succeeded on this same hash

If the drill needs a command that is not in observe_commands, I update the YAML in git and rehash. I do not "just this once" extend the path in the terminal. That exception is how freeze gates die.

Limitations, and who should skip this

This workflow assumes you can map alerts to ids, keep argv allowlists, and block writes in the default path. It will feel heavy if you are a solo hobby project with no pager, or if your platform cannot distinguish read kubectl from write kubectl. It also will not save you if the runbook hash is computed after the assistant edits the file. I am not claiming latency numbers, model rankings, or token budgets here, because those claims go stale and they are not the point.

Do not use this approach to launder a generated production change behind a decorative freeze file. Do not store signatures, kubeconfigs, or customer data on a shared demo server. Do not skip escalation because the model sounded calm. If you cannot name the blast radius in one sentence, the hash check should keep failing.

The core conclusion does not change when the models get more fluent. Encode the page as alert class, first command, escalation clock, and freeze hash, then make unfreeze a signed break of that hash. Everything else is optional commentary. If you want a scratch box to lint the YAML and the allowlist, MonkeyCode's free server option is enough to practice the checks; it is not an on-call seat.