Fill Four On-Call Lists Before You Accept the Rotation

On-call is not a conversation with a model while the pager is still firing at you. A rotation is ready only when four lists exist: the alert catalog, the first commands, the escalation packet, and the freeze rule. I will not accept a week of pages that depend on a chat transcript, no matter how fluent that transcript looks. Would you really open a blank prompt at three in the morning and call that a procedure?

Narrative runbooks feel complete in daylight, then collapse when two alerts land together. People scroll, argue about meaning, and invent write commands that nobody reviewed against a named surface. I want files I can grep, scripts that only print state, a clock that forces escalation, and a freeze line that does not depend on mood. Does a five-page story help you when the ID on the page is not even in the catalog?

The useful question this week is not whether assistants can draft English. The question is whether you will call that draft an operating procedure before it survives a checker. I let a model fill comments. I reject the file until the four lists validate. That is the method, and chat is none of the lists.

The four lists, named on purpose

I keep the on-call file as four lists because each list answers a different failure. The catalog answers "what is this page." First commands answer "what do we look at before we speak." Escalation answers "who gets the evidence, and when." Freeze answers "which writes are illegal until a human writes a sentence." If any list is missing, you are improvising, and improvisation is not a runbook.

Here is the order I actually enforce when a rotation starts:

1. Every page ID in the last month has a catalog row, including a "not this" note.
2. Every catalog row points at a read-only first-command script that exists on disk.
3. Escalation names roles, minute bounds, and the three fields in the packet.
4. Unknown IDs freeze writes immediately; unfreeze is a filled sentence, not a vibe.

Would a generated paragraph satisfy those four gates? Only the parts that survive the YAML and the checker below. Everything else stays a comment in a draft branch.

List 1: the alert catalog is a table, not a blog

Every page needs a stable ID, a meaning, a severity, and a "not this" note. If the ID is missing from the catalog, the incident is already in freeze, and nobody invents a new class on the call. New classes are born in daylight, with a script on disk and a minute bound that a human typed. Why would you let a model mint an ID while customers are already waiting?

Catalog shape I keep in git

# runbook/alerts.yaml — labeled template, not a live service file
alerts:
  - id: "checkout.p95.latency"
    severity: sev2
    means: "p95 checkout latency exceeded the burn threshold for 5 minutes"
    not_this:
      - "batch jobs on the reporting warehouse"
      - "synthetic probes failing from one region"
    first_command_set: "latency-read"
    escalate_after_minutes: 15
    write_allowed: false
  - id: "payments.auth.5xx"
    severity: sev1
    means: "payment auth 5xx rate crossed the error budget page"
    not_this:
      - "client timeouts from a single ISP"
    first_command_set: "payments-read"
    escalate_after_minutes: 8
    write_allowed: false

Notice write_allowed starts false on every row. A catalog row that defaults to writes is a loaded gun with the safety off. Unknown IDs do not get a row invented during the page; they get freeze, then escalation, then a daylight edit. I want the on-call person to grep, not to negotiate meaning in a thread.

List 2: first commands print state and then stop

First commands print state. They do not restart, scale, mute, deploy, or patch. I want the output pasted into the incident doc before anyone discusses a mutation, because the paste is the only evidence the next human can trust. If the command is not in the set named by the alert, it does not run, even when someone swears it is "just a quick look." Would you skip the paste step because the output looked obvious?

Read-only command sets

# first-commands/latency-read.sh
# labeled example: swap in your own read endpoints after review
set -euo pipefail
echo "UTC $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "alert_id=${ALERT_ID:-unset}"
curl -fsS "$STATUS_URL/checkout/p95" | head -c 4096
echo
kubectl get deploy checkout -o wide
kubectl get hpa checkout -o yaml | sed -n '1,80p'

# first-commands/payments-read.sh
set -euo pipefail
echo "UTC $(date -u +%Y-%m-%dT%H:%M:%SZ)"
curl -fsS "$STATUS_URL/payments/auth/5xx" | head -c 4096
echo
kubectl logs deploy/payments-auth --tail=80 --since=10m \
  | grep -E '5[0-9]{2}|timeout' | tail -n 40

I keep these scripts tiny on purpose. A first-command file that contains apply, delete, restart, or scale is not a first command; it is a change that snuck in through naming. The checker later treats those strings as failures, because I have watched "just restart it" show up in a reconnaissance folder.

List 3: escalation is a clock plus a packet

Escalation is not "ping someone if you feel stuck." It is a minute bound, a named role, and a packet of evidence that another human can read without joining your head. The packet is the alert ID, the first-command output path, and the freeze state. If any of those three is missing, you are not escalating; you are venting into a channel. What happens when the named person is asleep and the runbook only lists a nickname?

Escalation decision table

Clock	Condition	Who you page	Packet you must send
T+0	Page fires	primary on-call	alert ID, catalog hit or miss
Catalog minute bound	First commands cannot classify	secondary and service owner	output path, freeze=on
Any time	Alert ID missing from catalog	incident commander	freeze=on, no writes
After classify	A write is proposed	owner of the write surface	command text and revert line

I keep roles as functions, not heroic names. Put the rotation source next to the role so the packet can move when the human changes. A Slack handle with no rotation pointer is a hope, and hope is not an escalation path.

# runbook/escalation.yaml — labeled template
roles:
  primary: "pagerduty://checkout-primary"
  secondary: "pagerduty://checkout-secondary"
  service_owner: "pagerduty://checkout-owner"
  incident_commander: "pagerduty://ic-weekday"
freeze_on_unknown_alert: true
require_packet:
  - alert_id
  - first_command_output_uri
  - freeze_state

List 4: freeze unknown pages; unfreeze with a sentence

Here is the freeze rule I will defend in a postmortem. If the alert ID is absent from the catalog, freeze writes immediately and escalate. If the first-command set cannot classify the symptom inside the catalog minute bound, freeze writes and escalate. Unfreeze requires the catalog ID, a classified symptom, a named write owner, and an explicit unfreeze block in the incident doc. Is that slower than a heroic restart? Yes, and that delay is the control.

Unfreeze is a sentence, not a feeling in the room. I write it like this, and I refuse anything shorter:

UNFREEZE checkout writes
alert_id=checkout.p95.latency
classified_as=db_pool_saturation
write_owner=checkout-owner
command=increase checkout pool from 20 to 40
revert=set pool back to 20
expires=20m

If you cannot fill those fields, you stay frozen. The revert line is not decoration; it is the only way the next on-call person knows how to undo your write without asking the chat. I have no interest in an unfreeze that cannot be read after the call ends.

Artifact: fail the runbook before the rotation starts

I do not argue about completeness in the war room. A checker fails the tree on a laptop, or in CI, before anyone accepts the pager. The script below is a proposal you can run against the YAML shapes above. It does not prove production is healthy. It proves the four lists are present, writes are not smuggled into first commands, and unknown alerts are set to freeze.

#!/usr/bin/env python3
"""runbook_check.py — unlabeled fields fail the runbook."""
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("install pyyaml before running this checker\n")
    sys.exit(2)

REQUIRED_ALERT = {
    "id",
    "severity",
    "means",
    "not_this",
    "first_command_set",
    "escalate_after_minutes",
    "write_allowed",
}
REQUIRED_ROLES = {"primary", "secondary", "service_owner", "incident_commander"}
REQUIRED_PACKET = {"alert_id", "first_command_output_uri", "freeze_state"}
WRITE_MARKERS = ("kubectl delete", "kubectl apply", "restart", "scale ")


def fail(msg: str) -> None:
    print(f"FAIL: {msg}")
    raise SystemExit(1)


def main(root: Path) -> None:
    alerts = yaml.safe_load((root / "runbook/alerts.yaml").read_text())
    esc = yaml.safe_load((root / "runbook/escalation.yaml").read_text())
    rows = alerts.get("alerts") or fail("alerts.yaml missing alerts list")
    seen: set[str] = set()
    for row in rows:
        missing = REQUIRED_ALERT - set(row)
        if missing:
            fail(f"alert {row.get('id')!r} missing {sorted(missing)}")
        if row["id"] in seen:
            fail(f"duplicate alert id {row['id']}")
        seen.add(row["id"])
        if row.get("write_allowed") is True:
            fail(f"{row['id']} must not default write_allowed to true")
        cmd = root / "first-commands" / f"{row['first_command_set']}.sh"
        if not cmd.is_file():
            fail(f"missing first-command script {cmd}")
        text = cmd.read_text()
        for marker in WRITE_MARKERS:
            if marker in text:
                fail(f"{cmd} looks like a write; first commands must be read-only")
        minutes = row["escalate_after_minutes"]
        if not isinstance(minutes, int) or minutes <= 0:
            fail(f"{row['id']} escalate_after_minutes must be a positive int")
    roles = esc.get("roles") or fail("escalation.yaml missing roles")
    missing_roles = REQUIRED_ROLES - set(roles)
    if missing_roles:
        fail(f"missing roles {sorted(missing_roles)}")
    if esc.get("freeze_on_unknown_alert") is not True:
        fail("freeze_on_unknown_alert must be true")
    packet = set(esc.get("require_packet") or [])
    if packet != REQUIRED_PACKET:
        fail(f"require_packet must be {sorted(REQUIRED_PACKET)}")
    print(f"OK: {len(rows)} alerts, freeze-on-unknown enabled")


if __name__ == "__main__":
    main(Path(sys.argv[1]) if len(sys.argv) > 1 else Path("."))

Run it like this on a toy tree before you point it at the real rotation files:

python3 runbook_check.py .
# expected on a complete tree:
# OK: N alerts, freeze-on-unknown enabled

If the checker fails, the rotation does not start. That sounds harsh until you remember the alternative: a page whose first command does not exist, or a freeze rule that lives only in someone's memory. I would rather fail CI on Tuesday than invent YAML while the pager is already open.

How a page should actually move

When the page fires, I want a boring sequence, not a clever one. The primary copies the alert ID, greps the catalog, and runs only the named first-command script. Output goes into the incident doc before any theory gets typed. Classification either happens inside the minute bound or it does not; there is no third state called "we are pretty sure."

If the ID is missing, freeze is already on, and the packet goes to the incident commander with freeze=on and no writes. If classification fails the clock, freeze stays on and secondary plus owner get the output path. Only after those gates does anyone draft a write, and that write still needs an unfreeze block with a revert line. Would you skip a gate because the room feels aligned? The gates exist for the room that feels aligned.

Where a draft assistant fits, and where it does not

I will let a model propose means and not_this text from an exported list of alert names. I will not let it invent IDs, minute bounds, role URIs, or write commands. The draft is a comment. The YAML is the contract. That split is how you use an assistant without pretending the assistant took the page.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Comment drafts for those catalog sentences can sit in MonkeyCode, which has free model access and a free server option, and only fields that survive review get copied into alerts.yaml. The checker does not care where the English came from. The free server is a draft bench, not an incident cluster, and generated kubectl that never sat in first-commands/ stays unread.

Would I point any assistant at a production kubeconfig to "save time"? No. Time saved on typing is worthless if the first write has no revert line. Keep the model on comments. Keep the pager on lists.

Limitations you should not hand-wave

This workflow assumes you already have paging, a place to paste output, and someone who can own a write. The checker is syntactic; it cannot prove a curl target is actually read-only, and it cannot prove a role URI reaches a human who is awake. Minute bounds are guesses until you rehearse them with a game-day page. YAML that passes CI can still describe the wrong service, or a "not this" note that was true last quarter and is false tonight.

Assistants hallucinate confident not_this notes. If you paste those notes without a human who has taken that page, you will freeze the wrong subsystem or unfreeze too early. I treat every generated sentence as untrusted until a person who has carried the pager edits it. The four lists reduce improvisation. They do not replace judgment, and they do not replace a rehearsal.

Who should not use this shape

Do not use this shape if you have no pager, no secondary, and no incident doc. Do not use it to justify running model output against production because the sentences sounded operational. Do not use a shared free server as the system of record for secrets, kubeconfigs, or customer data. Tiny internal tools with one operator may prefer a single shell script over four YAML files, and that is fine.

If your culture rewards the fastest unmute, this freeze rule will feel like friction. That friction is the point of the runbook. I would rather explain a delayed write than explain an untracked restart that nobody can revert. The four lists are the runbook. Everything else is conversation, and conversation does not unfreeze writes.