Your Pager Should Resolve IDs, Not Prompts

Your pager should resolve a catalog ID, not a prompt, and that is the whole runbook. If the alert class has no pinned command, you do not improvise inside a production shell. You escalate with the catalog hash, and you keep the mutating path closed until a human pins again. Drafts belong on a laptop at a desk, not in a root shell while error budgets burn.

Why does this keep coming up? Agent-style loops are fashionable, and they look helpful until they invent kubectl flags from a guess. Are you really going to let a completion model choose selectors while customers are paging you? I am not, and I do not want the next on-call to do it either.

The failure I want to make expensive

A typical 3 a.m. path still looks like this, even in teams that own serious monitors. The alert fires, a chat window opens, and someone pastes logs into a model because the wiki is stale. The model returns a confident command, and a tired human runs it because reading diffs feels slower than hoping. Then the command is slightly wrong, and now you have two incidents sharing one pager.

Does that sound like engineering, or like gambling with a shell you already authenticated? I want improvisation to be the expensive action, not escalation to a human. Desk time is for drafting and pinning. Pager time is for executing something you already signed, then stopping.

This article proposes a compile step you can actually rehearse. It is a catalog, a tiny resolver, a freeze bit, and a small escalation packet. Treat the code as a worked example, not as a claim about any production fleet I have not named.

Compile, do not converse

Think of the runbook as a compiler with four inputs you can name out loud. You will recognize them if you have ever shipped a binary you trusted more than a scratch script sitting in /tmp.

1. Alert class — a stable name from the monitor, not a novel sentence.
2. Fingerprint — a short hash of the resource identity, so retries do not look new.
3. Catalog version — the git tag or sha of the pinned command file.
4. Freeze bit — a lock that means this catalog is untrusted until a human pins again.

The compiler output is either a command ID or an escalation. There is no third path called "just this once, I know this box." If you still need that third path, you are prompting production and calling it judgment.

What a first command is allowed to be

A first command is a pinned, named operation with a dry-run twin sitting beside it. It has a timeout, a mutate flag, and an argv list a human can read without decoding a transcript. It does not take a freeform string from a model at page time. It does not grow extra flags because the completion sounded sure.

Here is a catalog you can copy into catalog.yaml and edit at a desk. Label it a proposal until a human pins the version in git.

# proposal: example catalog — pin this in git, never in a chat log
version: "2026.09.11-1"
freeze: false
commands:
  - id: redis-conn-sat-observe
    alert_class: redis.connections.saturated
    mutate: false
    timeout_sec: 30
    argv: ["redis-cli", "-h", "${TARGET_HOST}", "INFO", "clients"]
    dry_run_argv: ["echo", "observe", "redis-cli", "INFO", "clients"]
  - id: redis-conn-sat-reload-proxy
    alert_class: redis.connections.saturated
    mutate: true
    requires_observe_id: redis-conn-sat-observe
    timeout_sec: 45
    argv: ["systemctl", "reload", "redis-proxy"]
    dry_run_argv: ["systemctl", "show", "redis-proxy", "--property=MainPID"]
  - id: disk-fill-observe
    alert_class: node.disk.fill
    mutate: false
    timeout_sec: 20
    argv: ["df", "-h", "${MOUNT}"]
    dry_run_argv: ["echo", "df", "-h", "${MOUNT}"]
escalation:
  when_no_id: page-secondary
  when_frozen: page-secondary
  when_nonzero_twice: page-secondary
  packet_fields: ["alert_class", "fingerprint", "catalog_version", "last_id", "freeze"]

Would I paste that YAML from a model straight onto the pager host? No. The model can draft it. A human pins it. Git records the pin, and the pager only sees the file that survived review.

The resolver refuses to be interesting

The pager-side tool should be boring on purpose, because clever tools invent argv under pressure. It reads the alert class, refuses unknown classes, and prints the command it would run. Mutating IDs stay blocked until observe succeeded in the same incident directory. If that feels rigid, ask who pays for a flexible reload at 3 a.m.

#!/usr/bin/env python3
"""Proposal: local catalog resolver. Not a production agent."""
from __future__ import annotations

import hashlib
import json
import os
import sys
import time
from pathlib import Path

import yaml  # PyYAML

CATALOG = Path(os.environ.get("RUNBOOK_CATALOG", "catalog.yaml"))
STATE = Path(os.environ.get("INCIDENT_DIR", "/tmp/incident")) / "state.json"


def load_catalog():
    data = yaml.safe_load(CATALOG.read_text())
    if not isinstance(data, dict) or "commands" not in data:
        raise SystemExit("catalog unreadable")
    return data


def fingerprint(alert: dict) -> str:
    raw = "|".join(
        [
            alert["alert_class"],
            alert.get("resource", "unknown"),
            alert.get("region", "unknown"),
        ]
    )
    return hashlib.sha256(raw.encode()).hexdigest()[:16]


def resolve(alert: dict) -> dict:
    cat = load_catalog()
    if cat.get("freeze"):
        return {
            "action": "escalate",
            "reason": "catalog_frozen",
            "version": cat.get("version"),
        }
    matches = [c for c in cat["commands"] if c["alert_class"] == alert["alert_class"]]
    if not matches:
        return {
            "action": "escalate",
            "reason": "no_pinned_id",
            "version": cat.get("version"),
        }
    observe = [c for c in matches if not c.get("mutate")]
    mutate = [c for c in matches if c.get("mutate")]
    state = json.loads(STATE.read_text()) if STATE.exists() else {}
    if observe and not state.get("observe_ok"):
        return {"action": "run", "command": observe[0], "version": cat.get("version")}
    if mutate and state.get("observe_ok"):
        return {"action": "run", "command": mutate[0], "version": cat.get("version")}
    return {
        "action": "escalate",
        "reason": "no_safe_next_id",
        "version": cat.get("version"),
    }


def main() -> None:
    alert = json.loads(sys.argv[1] if len(sys.argv) > 1 else sys.stdin.read())
    alert["fingerprint"] = fingerprint(alert)
    decision = resolve(alert)
    decision["fingerprint"] = alert["fingerprint"]
    decision["ts"] = int(time.time())
    print(json.dumps(decision, indent=2))
    if decision["action"] != "run":
        sys.exit(2)


if __name__ == "__main__":
    main()

Rehearse it like this when you are awake, not when you are guessing under a red Slack channel.

export RUNBOOK_CATALOG="$PWD/catalog.yaml"
export INCIDENT_DIR="$PWD/incidents/demo-1"
mkdir -p "$INCIDENT_DIR"

python3 resolve.py '{"alert_class":"redis.connections.saturated","resource":"redis-2","region":"us-east-1"}'

# expect action=run and id=redis-conn-sat-observe
# after a human marks observe ok:
echo '{"observe_ok": true}' > "$INCIDENT_DIR/state.json"
python3 resolve.py '{"alert_class":"node.disk.fill","resource":"/var","region":"us-east-1"}'

If the second call tries to mutate a class that has no mutate pin, the resolver should escalate. That refusal is the product. Do you want a clever tool, or a tool that will not get creative with systemctl?

Freeze is a catalog lock, not a chat reaction

Freeze, in this runbook, means the pinned file is untrusted. Maybe a command returned noise that no longer matches the host. Maybe a deploy landed, and the argv still names a unit that moved. Maybe someone spotted drift and does not want the next shift to replay a stale ID.

You freeze with a file write, not with a vibe in the incident channel. Unfreeze is not typing "lgtm" into Slack while the pager is still screaming.

# freeze: pager must escalate until a new pin exists
python3 - <<'PY'
from pathlib import Path
import yaml
p = Path("catalog.yaml")
data = yaml.safe_load(p.read_text())
data["freeze"] = True
p.write_text(yaml.safe_dump(data, sort_keys=False))
print("catalog frozen; pager will escalate")
PY

git add catalog.yaml
git commit -m "freeze catalog: redis-proxy reload looked stale"

Unfreeze is pinning a new version string and setting freeze: false in the same commit. If those two edits arrive in different commits, I would still treat the catalog as frozen. Why split the only two bits that decide whether a mutating ID may run?

Escalation is the default compiler output

Escalation should be cheaper to trigger than a guessed mutate, because guessed mutates create the next page. The packet is small on purpose. Secondary on-call should not reverse-engineer your chat history to learn what already ran.

ESCALATION PACKET
alert_class: redis.connections.saturated
fingerprint: a3c1f0e29b77c104
catalog_version: 2026.09.11-1
freeze: true
last_id: redis-conn-sat-observe
last_exit: 1
reason: catalog_frozen
asked_for: page-secondary

If you cannot fill those fields, you do not have a compiled incident yet. You have a mystery, and mysteries do not get systemctl reload. Would you hand a mystery to the next engineer and call that a first command?

Decision table you can print next to the pager

Alert class	Pinned first ID	Mutate ID	Freeze policy	Escalate when
redis.connections.saturated	redis-conn-sat-observe	redis-conn-sat-reload-proxy	Freeze catalog if reload exit is not 0	No ID, freeze, or observe never marked ok
node.disk.fill	disk-fill-observe	none in v1	Freeze if df shows the wrong mount	Any mutate request, because v1 has no mutate pin
unknown / empty class	none	none	Leave freeze unchanged	Always, with reason no_pinned_id
any class while freeze: true	none	none	Stay frozen until version bump	Always, with reason catalog_frozen

Print the table. Tape it near the laptop you actually take to the pager. If the table and the YAML disagree, the YAML is wrong until a human reconciles them. The pager does not get a vote, and the model does not get a vote either.

Draft the catalog where production cannot see you

The drafting environment and the production shell must not be the same window. That split is the safety story, not a branding story. I draft YAML at a desk, with the resolver tests, then I pin. I do not draft inside the SSH session that can reload a proxy.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can hold that drafting split: you iterate on catalog text and resolver tests on a box that is not production. The pager host only receives the pinned file. If a completion suggests a new mutate ID, it stays a draft until a human writes the version bump and clears the freeze bit.

Is a separate drafting box required? No. A local git repo is enough if you keep prod credentials out of the editor. The requirement is the air gap, not a vendor. Use the free server option if you want drafts off the laptop without opening production kubeconfigs. Do not let the model SSH anywhere from that box. Do not paste live secrets into a prompt to "make the YAML more realistic."

Rehearsal plan before you trust a pin

Label this as a rehearsal, because I am not citing a fleet I have not measured. Run the rows in order. If one fails, you do not pin, and you do not page-execute a guess to "just see."

1. Unknown class — feed alert_class: "totally.new" and expect exit 2 with no_pinned_id.
2. Frozen catalog — set freeze: true, replay a known class, expect catalog_frozen.
3. Observe before mutate — without state.json, a saturated Redis alert must return the observe ID only.
4. Mutate after observe — write observe_ok, replay Redis, expect the reload ID and mutate: true.
5. Disk fill has no mutate — even with observe_ok, node.disk.fill must escalate rather than invent rm.
6. Fingerprint stability — same resource twice, same 16-character fingerprint, so retries are visible.

python3 resolve.py '{"alert_class":"totally.new","resource":"x","region":"r"}'; echo exit:$?
# exit:2

If any row fails, that is a compile error at the desk. It is not an incident-skill problem for the person who just woke up. Fix the catalog, bump version, and only then copy the file onto the pager host.

Limitations, and who should not use this

This approach assumes you can name alert classes in advance. If your monitors still emit poetry, the compiler will escalate all night, and you will hate the noise. Good. That pain belongs on the monitor owners, not on the engineer holding the pager.

Do not use this if you have no human pin step. A model that writes catalog.yaml and a bot that executes it is not a compiler. It is a loop with extra files and a false sense of review. Do not use this as a substitute for IAM, sudoers, or change windows. A pinned rm is still rm, and a pinned reload still reloads.

Skip it on toy clusters where one person already types every command and reads every log line. The catalog helps when the next on-call is not the author of the argv. It also helps when you are the author, but you are tired enough to trust a fluent paragraph more than a hashed file.

I am not claiming latency numbers, token quotas, hardware, or model names. I am claiming a control: page time executes IDs you pinned at desk time. If that control feels slow, ask whether speed or correctness is the scarce resource while the alert is still firing.

Close the prompt window

So, will you keep pasting production logs into a chat box and hoping the argv is right? Or will you compile a short catalog, freeze it when it drifts, and escalate when the compiler shrugs? The second path is less cinematic. It is also the only first-command path I will trust when the operation mutates a host.

Keep drafts offline. Keep the pager dull. Pin the version, then go back to sleep if the ID did its job.