Refuse the Page Until the Runbook Names an Escalation Minute

I refuse every new page until the runbook names an escalation minute, a first command, and a freeze owner. AI drafts can fill comments on a throwaway box, but they cannot sign those three fields for me. A document without a clock is a blog post, and I will not carry it overnight. Would you really take a rotation whose escalation path still says "use your judgment"?

Why the escalation minute comes first

Alerts without a minute mark turn into private investigations that never escalate and never freeze writes. Night threads often show six people typing kubectl, and nobody can say when the next role should be paged. The runbook has to name that minute before I accept the rotation, not after the customer already felt the outage. If that escalation clock is missing from git, what exactly are you pretending to be on call for?

I do not start from a chatbot essay about incident best practices, because essays do not page humans. I start from the alert name, then I write the first read-only command, then I write the minute, then I write who freezes. That strict order keeps generated text inside comments, where it cannot open a production shell tonight. Why would I hand a model the pager when it still cannot name the unfreeze owner?

The signed runbook shape I actually keep

I keep one YAML file per service, and I treat unsigned keys as pages I will not accept. The file has to list alerts, first commands, escalation, and a freeze contract, because those are the only night-safe objects. Generated prose can live under a drafts key, and the validator must fail if that key is still unsigned. Does your repo still store the real procedure in a wiki that deploys independently of git?

Here is a template, labeled as a proposal, not as a dump from some secret production cluster.

# proposal: service runbook, unsigned until owners.sign == true
service: payments-api
owners:
  sign: false
  freeze: "sre-payments"
  unfreeze: "sre-payments-lead"
drafts:
  source: "model-comment-only"
  signed: false
  notes: "AI may propose text here; it must never become a command."
alerts:
  - id: PaymentsP99Latency
    severity: page
    silence_minutes: 15
    first_commands:
      - "kubectl --context=readonly get deploy payments-api -o wide"
      - "kubectl --context=readonly logs deploy/payments-api --tail=80"
    escalation:
      minute: 10
      to: "sre-payments-lead"
      page_back_when: "p99 < 300ms for 10m AND error_rate < 1%"
    freeze:
      on_clock_start: true
      writes: ["deploy", "configmap", "rollout"]
      unfreeze_evidence: "page_back_when cleared in this file"

Fields I will not negotiate

• alerts[].id must match the pager string, including punctuation, because fuzzy titles create the wrong first command.
• first_commands must be read-only invocations, and the wrapper below rejects anything that is not in this list.
• escalation.minute is an integer clock that starts at ack, not a feeling about how bad the graph looks.
• page_back_when is a measurable condition, because "looks better" is how silent failures return at dawn.
• freeze.on_clock_start means writes stop when escalation starts, not after someone remembers the wiki.
• drafts.signed must stay false until a human copies accepted text into the signed keys.

First commands are read-only or they do not ship

I want the first five minutes to be evidence, not mutation, because mutation without a freeze owner is just another outage. Every first command in the file above is a get or a log read, and the allowlist wrapper will refuse a rollout. If a model suggests kubectl delete, that line stays in drafts.notes until a human rewrites it after unfreeze. Are you still pasting generated kubectl into production before the runbook even names the write blast?

I keep a boring wrapper so tired humans cannot "just this once" run a stray apply.

#!/usr/bin/env bash
# proposal: allowlist runner for first commands only
set -euo pipefail
RUNBOOK="${1:?runbook yaml}"
ALERT="${2:?alert id}"
shift 2
CMD="$*"

python3 - "$RUNBOOK" "$ALERT" "$CMD" <<'PY'
import sys, yaml
runbook, alert_id, cmd = sys.argv[1], sys.argv[2], sys.argv[3]
doc = yaml.safe_load(open(runbook))
if not doc.get("owners", {}).get("sign"):
    raise SystemExit("unsigned runbook: refuse to exec")
if doc.get("drafts", {}).get("signed") is True:
    raise SystemExit("drafts cannot be signed; copy text into alerts first")
alert = next(a for a in doc["alerts"] if a["id"] == alert_id)
if cmd not in alert["first_commands"]:
    raise SystemExit("command not in first_commands allowlist")
print("allow")
PY

# reached only after the allowlist printed allow
# first_commands already pin --context=readonly
exec $CMD

What this wrapper is allowed to do

• Read the signed YAML from git, not from a paste in Slack, before any shell starts.
• Refuse the whole night if owners.sign is still false on the branch that can page.
• Compare the exact command string, because a nearby destructive cousin is still a miss.
• Leave writes to the freeze contract, because first-command territory ends when mutation begins.

That wrapper is deliberately rude, and I actually like it that way during a noisy page. It will not run an unsigned file, and it will not run a command the alert did not name. If you need a write, you are no longer in first-command territory, and the freeze contract has to speak. Should a tired operator be able to skip the allowlist because a chatbot sounded confident?

Escalation is a clock with a page-back condition

I write escalation.minute: 10 when ten minutes of read-only evidence is enough to stop being a lone hero. At that minute the named role is paged, freeze stays on, and I stop collecting extra screenshots for a thread nobody will read. The page-back condition has to be something a graph can answer, not a vibe about error budget remaining. Who is supposed to page you back if p99 only recovered because customer traffic already died?

Escalation ladder I paste into the service file

1. Minute 0: ack the page, run only first_commands, and start the freeze clock if the alert is a page.
2. Minute 10: page escalation.to with the command output attached, not with a novel about possible causes.
3. Minute 20: if page_back_when is still false, page the unfreeze owner, who is the only person allowed to lift writes.
4. Any earlier recovery: leave freeze on until page_back_when is recorded as cleared in git, not in chat.

I want those minutes in YAML because chat messages vanish, and memories lie after adrenaline fades. The validator below fails the pull request when minute is missing or when page_back_when is an empty string. That failure is the whole point, because empty escalation is how teams pretend they have a runbook. Can you show me the minute in git, or only in a slide from last quarter?

Freeze starts when the clock starts

I do not wait for a dramatic graph before I freeze writes, because drama arrives after the bad config already rolled. When the escalation clock starts, deploys, configmaps, and rollouts for that service are blocked until the unfreeze owner records evidence. Unfreeze is a different signature from freeze, and the YAML names two different people on purpose. If one person can freeze and unfreeze alone, what stops a panicked apply at minute nine?

This is not a platform design essay; it is a git-enforced habit for a single service file. You can block writes with an admission policy, a deploy queue, or a locked pipeline without buying a new vendor. The runbook only has to name the rule so humans do not argue about it while the pager is screaming. Is your production freeze still just a Slack emoji from whoever happens to be awake?

Draft on a throwaway box, never on the bastion

I will let a coding model propose YAML comments, but I will not let it propose a live shell on the production bastion. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode only as a drafting place, because free model access and a free server keep the skeleton off the pager host. The model fills drafts.notes, I copy surviving lines into signed keys, and then I set owners.sign on my laptop. If the draft still contains writes, why would I promote those lines into first_commands tonight?

The useful artifact remains the signed file in git, not the editor that proposed the comments. Remove the drafting box and the validator still protects you, which is how this workflow should work. Generated confidence is not an escalation path, and a free server is not a production replica. Do not paste a model rollout into the allowlist and call it a runbook.

The validator I run in CI

I run this as a proposal script in CI, and I have not baked it into a particular platform. It fails closed: unsigned owners, signed drafts, missing minutes, empty page-back, or a first command that looks like a write.

#!/usr/bin/env python3
"""Proposal: fail CI unless the on-call runbook is night-safe."""
from __future__ import annotations

import re
import sys
from pathlib import Path

import yaml

WRITE_HINTS = re.compile(
    r"\b(apply|delete|replace|patch|rollout\s+undo|scale|edit)\b",
    re.I,
)


def fail(msg: str) -> None:
    raise SystemExit(f"runbook invalid: {msg}")


def main(path: str) -> None:
    doc = yaml.safe_load(Path(path).read_text())
    owners = doc.get("owners") or {}
    if owners.get("sign") is not True:
        fail("owners.sign must be true before the file can page humans")
    if not owners.get("freeze") or not owners.get("unfreeze"):
        fail("freeze and unfreeze owners must be named")
    if owners.get("freeze") == owners.get("unfreeze"):
        fail("freeze and unfreeze must be different humans or roles")

    drafts = doc.get("drafts") or {}
    if drafts.get("signed") is True:
        fail("drafts.signed must stay false; copy text into alerts first")

    alerts = doc.get("alerts") or []
    if not alerts:
        fail("at least one alert is required")

    for alert in alerts:
        aid = alert.get("id") or "<missing>"
        cmds = alert.get("first_commands") or []
        if not cmds:
            fail(f"{aid} has no first_commands")
        for cmd in cmds:
            if WRITE_HINTS.search(cmd):
                fail(f"{aid} first command looks like a write: {cmd}")
            if "readonly" not in cmd:
                fail(f"{aid} first command must pin a readonly context: {cmd}")
        esc = alert.get("escalation") or {}
        if not isinstance(esc.get("minute"), int) or esc["minute"] <= 0:
            fail(f"{aid} escalation.minute must be a positive integer")
        if not esc.get("to"):
            fail(f"{aid} escalation.to is empty")
        if not esc.get("page_back_when"):
            fail(f"{aid} page_back_when is empty")
        freeze = alert.get("freeze") or {}
        if freeze.get("on_clock_start") is not True:
            fail(f"{aid} must freeze writes when the escalation clock starts")
        if not freeze.get("unfreeze_evidence"):
            fail(f"{aid} unfreeze_evidence is empty")
    print("runbook ok")


if __name__ == "__main__":
    main(sys.argv[1])

Run it like this, and keep the unsigned template out of the branch that can page.

pip install pyyaml
python3 validate_runbook.py payments-api.runbook.yaml
# unsigned files must exit nonzero

I want that nonzero exit more than I want a prettier wiki, because pretty docs do not block a rollout. If CI is green while owners.sign is false, you built a linter that flatters the author. Should a pull request that cannot name a page-back condition be allowed to merge?

Limitations I will say out loud

This file does not detect a bad metric, and it does not replace a real observability stack during an incident. The write hints are regular expressions, so a creative destructive command can still sneak through if you work at it. The YAML cannot freeze a cluster by itself, so you still need an admission policy or a locked pipeline. If your pager string does not match alerts[].id, the allowlist will refuse, and that refusal is correct.

It also does not make generated text trustworthy, which is the whole lesson hiding under the trend around AI-authored work. A model can outline comments quickly on a free server, and that speed is useful only because the validator remains hostile. I will not claim latency numbers, token quotas, or hardware details I cannot verify from here. If you need those numbers, measure them yourself on your own box before you change a night rotation.

Who should skip this approach

Skip it if you are a solo hobby project with no pager, because freeze owners and escalation minutes are ceremony you do not need. Skip it if your compliance team requires a vendor-signed runbook format that this YAML cannot satisfy. Skip it if you expected the drafting box to execute production commands, because that is the opposite of the allowlist. And skip it if you want a model to unfreeze writes, because I will not give any model that signature.

I still want humans on the rotation, and I still want git to be ruder than Slack. Name the escalation minute, keep first commands read-only, and freeze writes when the clock starts. If you remember only one rule tonight, remember that unsigned drafts still do not page people. Would you actually merge this runbook file tonight, or are the escalation minutes still sitting blank?