DEV Community: Emery Huang

Hold the Write Until Five Read Receipts Match the Escalation Ack

Emery Huang — Sun, 11 Oct 2026 21:24:48 +0000

I will not let a paging model choose the first command, because the alert class should already have made that choice. The only question that matters in minute one is whether five read-only receipts exist before anyone requests a write. If those receipts are missing, I keep the shell frozen and I escalate on a clock rather than on a feeling. Have you watched a confident paragraph turn a noisy page into a restart that nobody had actually approved?

The page may change attention, not authority

A page is a request for attention, and I refuse to treat it as a grant of write access on the paged host. I want that sentence in the incident channel before the bridge call collects opinions, because opinions arrive faster than evidence. The card in this article is a proposal you can adapt, not a diary of an outage I am pretending to have run last night. If your team already runs a stricter change gate, keep that gate, and steal only the receipt idea from this card.

Three constraints sit above the commands, and I repeat them whenever someone offers a shortcut around the card.

Intake may name an alert class, but intake may not mutate hosts, flags, or queues.
The first five commands are read-only, and the class picks them before a person or a model starts improvising.
Unfreeze covers one named command, and only after receipts plus an escalation ack exist in the channel.

Faster drafts did not retire the freeze

I keep seeing discussion, including this week, about models sounding smoother while the software around them stayed stubborn. I will not quote a leaderboard I have not opened, and I will not treat a headline as a primary source for your pager. The practical lesson I am willing to keep is narrower than the headline, and it is about authority rather than fluency. A cleaner paragraph does not earn a write on a sick host, even when the alert text was copied perfectly into the prompt.

If the draft is fluent and the receipts are still empty, the freeze still wins, and the escalation clock still runs. Why would smoother wording change the owner of a production shell? I want the answer written down before the bridge call turns confidence into a typed mutation.

Pick a class before you open a shell

I use three classes, because a single generic checklist hides the check that would actually explain this page. Class A means a customer-visible failure, or any credible risk that stored data was lost or corrupted. Class B means the error budget is burning, while nobody has confirmed an actual customer-facing break yet. Class C means a detector symptom that might belong to the detector itself, rather than to the service under the page.

Why would I start in a softer class when a wrong downgrade can hide a real customer break? Downgrading the page is itself a decision, and I would rather run one extra read than miss the checkout path. If I cannot defend the class in a single line, I stay on Class A until a human explicitly moves it. That bias is intentional, which is why the table below never shows an open write posture for any class.

Class	What I need to see	First goal of the reads	Write posture
A	Errors, timeouts, or data doubt	Bound impact and name a secondary	Frozen
B	Budget burn, no confirmed break	Confirm scope and the recent trend	Frozen
C	Detector-only, no user report	Separate detector fault from host fault	Frozen

The class changes the questions I ask, and it deliberately does not change the ban on mutation during intake. Would you unlock a restart for Class C merely because a draft called the alert simple? I would not, so the linter below treats every class as frozen until the receipts actually land.

Lint the first five commands on a laptop

I keep the allowlist in a script, so a tired paste cannot smuggle a write verb into the first minute of the page. The listing is unexecuted proposal code, and I would run it against a fixture on a laptop rather than on a production shell. It classifies one alert line, prints the matching card, and returns non-zero when a suggestion looks like a mutation. You should replace the paths before you trust it, because I have not executed this file on your hosts.

#!/usr/bin/env python3
"""Proposal: require read receipts before any unfreeze. Not a prod agent."""
import re, sys

WRITE = re.compile(
    r"\b(kill|delete|drop|apply|restart|scale|patch|exec|truncate|systemctl)\b",
    re.I,
)
CARDS = {
    "A": [
        "date -u",
        "hostname",
        "uptime",
        "curl -fsS --max-time 3 http://127.0.0.1/health || true",
        "tail -n 40 /var/log/app/app.log",
    ],
    "B": [
        "date -u",
        "uptime",
        "df -h",
        "ps -eo pid,pcpu,pmem,comm --sort=-pcpu | head",
        "tail -n 30 /var/log/app/app.log",
    ],
    "C": [
        "date -u",
        "hostname",
        "id",
        "journalctl -u app -n 40 --no-pager || true",
        "tail -n 20 /var/log/app/app.log",
    ],
}

def classify(text: str) -> str:
    t = text.lower()
    if any(k in t for k in ("data loss", "checkout", "5xx", "timeout")):
        return "A"
    if any(k in t for k in ("slo", "burn", "latency")):
        return "B"
    return "C"

def main() -> int:
    alert = sys.stdin.read().strip()
    kind = classify(alert)
    print(f"class={kind} posture=frozen receipts=0/5")
    for index, cmd in enumerate(CARDS[kind], start=1):
        if WRITE.search(cmd):
            print(f"reject card[{index}] {cmd}")
            return 2
        print(f"read[{index}] {cmd}")
    suggested = " ".join(sys.argv[1:])
    if suggested and WRITE.search(suggested):
        print("reject suggestion: write seen before receipts")
        return 3
    print("hold unfreeze until five receipts and an escalation ack")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

A fixture committed beside the script should stay short enough to reread on a phone during the page.

alert: checkout 5xx ratio above page threshold for 3 minutes

The local check is supposed to feel boring, which is what I want when the bridge call is already loud.

python3 receipt_card.py systemctl restart app < fixture.txt
echo "exit=$?"

An exit status of 3 means the suggestion tried to skip the receipt gate, so I discard that suggestion immediately. An exit status of 0 means I earned a reading list, and I still have not earned permission to write. After each read, I append a receipt with a tiny helper instead of trusting memory in the middle of the night.

#!/bin/sh
# Proposal: record one read receipt. Usage: ./note_receipt.sh A "date -u" 0
printf 'RECEIPT class=%s cmd="%s" at=%s exit=%s\n' \
  "$1" "$2" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$3"

RECEIPT class=A cmd="date -u" at=2026-10-12T03:12:01Z exit=0
RECEIPT class=A cmd="hostname" at=2026-10-12T03:12:04Z exit=0

Escalation runs on a clock, not a mood

I start the escalation clock when the class line is posted, because a vague status is not a policy anyone can audit the next morning. In this proposal, Class A pages the secondary at ten minutes, Class B at twenty, and Class C at thirty unless customer chatter appears. Those durations are planning defaults for a small rotation, not measurements taken from your pager data. Edit the numbers on the card before you adopt them, and do not cite this article as if it were a study.

The order of the names matters more than the speed of any generated summary you might paste.

Primary posts the class, the freeze, and the five-command card in the incident channel.
Primary names the secondary in that same message, even when the escalation timer has not fired yet.
If the timer fires without a secondary ack, primary pages them and still refuses writes.
If impact stays unclear at the next interval, primary pulls the service owner into the channel.

Could a model draft that acknowledgement faster than I can type the same four fields myself? Yes, and the speed helps only when class, freeze, secondary, and deadline all remain intact. If the draft omits the secondary, I do not send it, even when the surrounding prose sounds calm and complete. A missing name is an escalation failure, not a style problem I should polish after the page is already late.

Unfreeze is one human-typed command

Freeze means no write-class command and no temporary scale, including a one-liner that a chatbot offers as the obvious fix. Unfreeze is not a change of mood, and a red graph does not imply permission the channel never granted. I allow one command, typed by the human who holds the page, only after every gate below is already true.

Five read receipts are in the channel, and each receipt names the same class that intake recorded.
The secondary, or the service owner, has acknowledged that same class in a written channel message.
The command is absent from the read-only card, and a human typed it rather than pasting a model line untouched.
The release names an end time, and the posture returns to frozen when that time passes without a fresh line.

I keep the release on one grep-friendly line, so a later review can see exactly what the page allowed.

UNFREEZE class=A receipts=5/5 ack=lee cmd="service app reload" until=2026-10-12T03:40:00Z by=sam

If any field is missing, I stay frozen, even when the chart looks worse than it did during intake. A worse chart is a reason to escalate the page, not a reason to skip the receipts you already required. Would you sign a blank unfreeze because a draft said the reload looked safe from the alert text alone? I want that answer recorded as no, in the same channel that received the original page.

Draft the card away from production credentials

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I would draft and lint this card on a separate machine, and the operator says MonkeyCode offers free model access plus a free server option for that drafting work. I am not stating a token quota, a hardware shape, or an end date, because no primary source for those figures was attached here. If that option is closed when you read this, run the same linter on any laptop that cannot see production credentials.

What I would send is the alert sentence and the allowlist, with secrets left out of the prompt on purpose. What I would accept back is a class label or a rejection, never a live session on the host that was paged. If the draft inserts a write verb, the script's non-zero exit is the decision, and the drafting host does not get another vote. That split is the point of keeping generation away from the shell this runbook is trying to protect.

I am not using this section to rank tools, and I am not claiming the free option will still exist next quarter. Availability language here is operator-supplied, not a measurement I reran against a dated product page on the day of this draft.

Who should skip this card

This approach fits poorly when you lack an incident channel, a secondary, or a read-only check that stays safe on a sick host. It also fits poorly on safety-critical control gear, where my sample commands could be the wrong shape entirely. I am not claiming a shorter recovery, a better model score, or a replacement for the vendor runbook you already trust.

I keep the limits beside the suggestion, so a tired reader cannot skip them on the way to the code block.

The classifier is only a keyword stub, and novel alert names will miss until you extend those lists yourself.
Read-only is not automatically harmless, so the tail and curl examples stay capped, local, and easy to delete.
A free drafting host may be down, rate-limited, or unfit for secrets, so never paste credentials into the prompt.
The clocks above are proposals, not results from a published on-call study that you should quote as evidence.

If a command in the card does not exist on your hosts, delete it before the next page instead of discovering the gap live. I would rather you keep a shorter card than run an example that only matched an imaginary layout. The receipt rule still works after you swap in commands your platform team already trusts for diagnosis.

If the paper card has never been linted, run it once against a fixture before the next rotation, on a separate machine you control. A free drafting server is optional for that pass, and the freeze rule does not depend on which editor helped you type the card.

Block Write-Class Commands Until the Page Clock Names Owner and Expiry

Emery Huang — Sat, 10 Oct 2026 17:50:19 +0000

I will not let an assistant draft become a shell command until a page clock names the owner, the severity, and a hard expiry. The opening stretch of a page should stay in a read class, because an early write is only a guess with privilege. Have you ever changed a unit file while the alert title was still a blur on the second monitor? I keep that impulse behind a command class gate, and I open the gate only when a human can point at the clock.

What the page clock is allowed to decide

A page clock is a small file that records who is on the hook, how severe the page looks, and when the read window ends. It does not diagnose the outage, and it does not bless a clever patch that a model just printed into chat. I use it as a gate in front of write-class commands, so a draft can sit on scratch storage away from the paging host. Would you trust a suggestion that arrived before anyone was willing to name the severity out loud?

I would not, which is why the clock has to exist before that suggestion is eligible for a human review. The file lives on a scratch path, not beside the unit that woke you up in the first place. I treat the production shell as frozen until a human replaces a closed write lock with an explicit unfreeze. If the clock file is missing, I keep reading and I escalate, rather than improvising a fix from a chat pane.

The clock file I want beside the runbook

The sample below is a proposal for the runbook repository, and I have not executed it against a live fleet. Copy it only into a scratch directory, then replace every placeholder with values taken from the page you received. Leave production hostnames out of this file on purpose, because a clock should never become a quiet map of targets. I would rather see an empty field than a hostname copied from a suggestion that nobody reviewed.

# page-clock.yaml: proposed artifact, not a live incident record
page_id: "replace-with-alert-id"
owner: "replace-with-human-name"
severity: "unknown"
opened_at: "replace-with-utc-timestamp"
read_window_minutes: 15
write_lock: "closed"
escalation_at: "replace-with-utc-timestamp"
draft_state: "held"
rollback_intent: "unset"

First commands that stay in the read class

I start with commands that only observe, and I refuse anything that mutates packages, units, routes, or stored data. These examples assume a Linux host you already administer, and they stay proposals until your access policy allows the reads. I also keep a tiny wrapper that classifies a command string before I consider running that string for real. Have you noticed how a draft mixes a harmless status check with a restart on the next line?

Confirm the page identity in the clock file before you open a process list, because a mismatched alert wastes the window.
Reject the snapshot when owner or severity is still a placeholder, and spend that minute naming the fields instead.
Collect service state into the scratch directory, and do not pipe that output into a restart or a package tool.
Hold any command the classifier marks unknown, because an unknown verb is closer to a write than to a safe read.

# classify.sh: proposed read/write split for a scratch shell you control.
# Unknown commands fail closed. This example has not been run on a fleet.
set -eu
classify() {
  local cmd="$1"
  case "$cmd" in
    "systemctl status "*|"systemctl is-active "*|"journalctl "*|"ss "*|"ps "*|"cat "*|"ls "*)
      printf 'READ %s\n' "$cmd"
      ;;
    "systemctl restart "*|"systemctl stop "*|"apt "*|"dnf "*|"rm "*|"kubectl apply "*|"kubectl delete "*)
      printf 'WRITE blocked until the page clock opens: %s\n' "$cmd" >&2
      return 2
      ;;
    *)
      printf 'UNKNOWN hold for a human: %s\n' "$cmd" >&2
      return 3
      ;;
  esac
}
classify "journalctl -u demo.service --since 20 min ago"
classify "systemctl restart demo.service" || true

I do not smuggle a restart into this window, even when a draft sounds sure about a cause it invented. A restart can hide the evidence the next human needed, and a package change can widen the failure without rollback intent. If the service is already down, the snapshot still comes first, and the write class stays blocked. Have you restarted first and then lost the only log line that explained why the page fired?

How the gate decides an unfreeze

When the escalation time passes and the owner has not updated the clock, I stop polishing drafts and page the secondary. The secondary should inherit a closed lock and a held draft, not a half-applied change that nobody can describe. I would rather spend the read window twice than ship an unnamed edit into the same hour as the alert. If severity is still unknown at that moment, I widen the bridge instead of letting a model guess a rank.

The unfreeze sentence

The unfreeze rule is short enough to read aloud without scrolling through a long bridge novel. A human may open the write lock only after owner, severity, and rollback intent are real values you trust. An assistant may suggest wording for the rollback line, but it may not flip the lock field by itself. If your severe-page policy demands a second person, the gate stays shut until that second name is written down.

# gate.py: proposed local check, not a production agent.
# Exit 2 while the lock is closed or required fields are placeholders.
import sys

PLACEHOLDERS = {
    '',
    'unset',
    'unknown',
    'replace-with-alert-id',
    'replace-with-human-name',
}

def load(path):
    fields = {}
    for raw in open(path, encoding='utf-8'):
        line = raw.strip()
        if not line or line.startswith('#') or ':' not in line:
            continue
        key, val = line.split(':', 1)
        fields[key.strip()] = val.split('#', 1)[0].strip().strip('"')
    return fields

def main():
    fields = load('page-clock.yaml')
    problems = []
    for key in ('page_id', 'owner', 'rollback_intent'):
        value = fields.get(key, '')
        if value in PLACEHOLDERS or value.startswith('replace-'):
            problems.append(key)
    if fields.get('severity') not in ('sev1', 'sev2', 'sev3'):
        problems.append('severity')
    if fields.get('write_lock') != 'open':
        problems.append('write_lock')
    if problems:
        print('write lock stays closed:', ', '.join(problems))
        return 2
    if fields.get('draft_state') != 'reviewed':
        print('draft still held; human review required')
        return 3
    print('human gate passed; use your normal change process')
    return 0

if __name__ == '__main__':
    sys.exit(main())

Where a free draft bench fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I bring it up here because free model access and a free server can hold this scratch clock away from the production shell. I have no measured quota or hardware sheet, so I will not invent either figure or a permanence claim. The part I actually want is a clean split between a draft bench and the host that is paging.

Would I paste production credentials onto that free server just to save a copy step during the page? I would not, and the runbook should forbid that shortcut in the same paragraph as the product note. I would use the free model only after the read commands have already produced a snapshot I can stand behind on the bridge. It may tidy four short lines, and it may not invent the owner, the severity, or the rollback intent.

If you already have a MonkeyCode login, render this clock on the free server and carry the reviewed file back by hand. That is the only invitation I will make, because a runbook should stay useful when the product name is deleted. I still want the classifier and the gate even if you never open that draft bench at all. Does your bridge note still make sense after you remove every product word from the page?

A decision table for the first twenty minutes

I read this table before I accept help from anyone, including myself at minute ten when the alert still feels urgent. The rows are states of the clock, not moods, and the last column is the only next action I allow. If a row says the lock stays closed, I do not negotiate with a confident paragraph from a model. Have you ever talked yourself out of a table you wrote while you were still calm?

Clock state	Draft state	Next action I allow
File missing	Any text	Create the clock on scratch, run read-class commands only, and escalate if no owner can be named
Lock closed and severity unknown	Held	Keep observing, widen the bridge at escalation time, and do not apply a patch
Lock closed and severity named	Held	Let a human review the note, and keep production writes blocked
Lock open and draft reviewed	Reviewed	Follow the existing change process, with rollback intent still visible
Lock open and draft still held	Any text	Fail closed, because model text is not approval to write

A local test I can run without a fleet

I would test the gate on a laptop before anyone pastes it into a shared runbook, because an untested lock is only a comment. Create a temporary directory, copy the sample clock and gate script into it, and confirm a closed lock exits with status 2. Then write real-looking values, open the lock, mark the draft reviewed, and confirm the pass line prints on your machine. Set severity back to unknown afterward, and confirm the script refuses again, since that state is not a real unfreeze.

Exit codes worth trusting

I am not claiming this rehearsal caught a production incident, and I am not publishing a timing number from a borrowed machine. The only result I want is a predictable exit code in a directory you can delete when the check is done. If the closed lock exits zero, the gate is wrong, and it does not belong in the on-call document yet. Would you hand a pager a script that passes while the lock field still says closed?

# Proposed local test. Temporary directory only.
# Expect status 2, then status 0, then status 2 again.
tmpdir=$(mktemp -d)
cp page-clock.yaml gate.py "$tmpdir/"
cd "$tmpdir"
python3 gate.py || echo "closed lock failed closed, exit=$?"
python3 - <<'PY'
from pathlib import Path
path = Path("page-clock.yaml")
text = path.read_text(encoding="utf-8")
text = text.replace("replace-with-alert-id", "alert-1001")
text = text.replace("replace-with-human-name", "bridge-owner")
text = text.replace('severity: "unknown"', 'severity: "sev2"')
text = text.replace('write_lock: "closed"', 'write_lock: "open"')
text = text.replace('draft_state: "held"', 'draft_state: "reviewed"')
text = text.replace('rollback_intent: "unset"', 'rollback_intent: "revert the last config commit"')
path.write_text(text, encoding="utf-8")
PY
python3 gate.py
python3 - <<'PY'
from pathlib import Path
path = Path("page-clock.yaml")
text = path.read_text(encoding="utf-8")
text = text.replace('severity: "sev2"', 'severity: "unknown"')
path.write_text(text, encoding="utf-8")
PY
python3 gate.py || echo "unknown severity failed closed, exit=$?"

Who should skip this pattern

This pattern is a poor fit when your charter already authorizes a break-glass write, such as a documented failover that does not need a fresh clock. It is also a poor fit when you are alone on a severe page and the existing runbook already names the immediate action to take. Do not place secrets, customer payloads, or production kubeconfigs on a free server because the draft step feels convenient today. Do not use the gate to delay a human page, and do not treat model confidence as a fill-in for the owner.

Free model access can change, rate-limit, or vanish, so the runbook still has to make sense with the assistant powered off completely. If your organization forbids third-party tools on an incident bridge, skip the product path and keep the clock file plus the classifier. I would rather keep a boring read window than defend a fast draft that nobody on the bridge can explain. Are you adopting a gate you cannot test on a laptop this week, or are you only collecting another note?

What I leave in the bridge note

I end the read window with four lines, and I do not let the note grow into a novel while the lock is still closed. Those lines are the page id, the owner, the severity, and the next human action after the escalation time you already named. An assistant may tidy the grammar once those lines exist, but it may not invent them from a vague alert title alone. If you cannot fill those four lines, are you really ready to run a write-class command on the host?

A Three-Line Scratch Journal Is the Only Ticket Past the Suggestion Fence

Emery Huang — Fri, 09 Oct 2026 14:20:32 +0000

I keep every mitigation draft behind a fence until a three-line scratch journal exists on disk. That rule is the conclusion, and the rest of this note is only the machinery that makes the rule hard to skip. Would you rather read a smooth fix than the three boring lines that prove you touched the right host? I would not, because a smooth paragraph is cheap and a wrong restart on the wrong host is expensive.

I am labeling this whole workflow as an unexecuted proposal, not as a pager story from a shift I actually worked. Please copy the checker if it helps, and please do not treat my wording as evidence that a drill already passed. If your team already freezes production with a stronger control, keep that control and borrow only the journal habit.

Why I refuse the draft first

A draft that arrives before the journal is not a head start, because it is a contaminated input I did not ask for yet. I delete that file and I start the journal again, even when the sentences look calm and specific. Would you keep an early guess just because it happened to use the right service name in the first line? I would not, since the service name is often the only true fragment hiding inside an early draft.

I want the assistant fenced into a suggestion block that a human can throw away without touching a live host. Clever wording is not the goal, because clever wording is how a quiet page becomes a loud one. The fence is local, boring, and intentionally easy to audit with the two shell scripts below. If the scripts feel too small for your estate, that smallness is a feature I am willing to defend.

Three lanes before any shell

I choose a lane before I open a shell, and I write the lane name at the top of the journal. Observe means I collect read-only lines and I do not rehearse a change on any host. Rehearse means I may use a scratch host that is not production, and I still may not unfreeze anything. Escalate means I stop drafting immediately and I call the person already named in the runbook.

Lane names I refuse to blur

The checker cannot promote a lane, because promotion is a human decision I do not want hidden inside a script. If the lane is missing, I treat the page as escalate rather than inventing a policy while I am tired. How often do you discover the real severity only after you have already typed a fix into the shell? I still want the first choice written down, so the later correction has something honest to replace.

A table I can read without scrolling

I keep five rows in the table, because a table I cannot memorize will not survive a bad night. If a signal is not in the table, the lane is escalate and the freeze stays on. Would you rather debate a sixth row while the mitigation draft is already half written on disk? I would rather lose a minute to a short table than lose an hour to a guessed command.

Signal I can verify	Lane	First action	Freeze rule
Alert text only, no host I own	Escalate	Call the named secondary	Stay frozen
Synthetic fixture on a scratch host	Rehearse	Write the three-line journal	Stay frozen
Known gap, no customer impact I can show	Observe	Record the journal, do not edit	Stay frozen
Confirmed impact on a service I own	Escalate	Call, then journal if asked	Frozen until a human writes unfreeze
Draft file exists before any journal	Refuse	Delete the draft and start over	Stay frozen

The last row is the only row I actually enforce with a script rather than with memory. The other rows remain human judgment, and that judgment stays with the person who accepted the page. I do not want a model to pick the row, even when the alert text sounds confident and complete.

The only first commands I allow in the drill

I allow only three read-only commands, and I append them to journal.txt before I create suggestion.md. I do not point this drill at production, because production is not a classroom and a classroom is not a change window. The fixture can live on a scratch host, or on my laptop if I do not have a scratch host yet. Can you recite those three lines from memory without opening a wiki or a chat transcript?

If you cannot recite them, you are not ready to accept a mitigation draft from anyone else. Swap the commands if your estate uses different read-only checks, but do not raise the count and do not add a write. A fourth line is how a drill quietly becomes a change, and I do not want that slide. Keep the journal boring enough that a secondary can read it without asking what you meant.

#!/usr/bin/env bash
# Proposal only. Not executed against production in this article.
set -euo pipefail
umask 077
journal="${1:-journal.txt}"
: > "$journal"
{
  printf 'utc=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  printf 'host=%s\n' "$(hostname -s)"
  printf 'who=%s\n' "$(id -un)"
} >> "$journal"
echo "journal_lines=$(wc -l < "$journal")"

Those three lines prove time, place, and identity, and they prove nothing about the cause of the page. That emptiness is deliberate, because I want a witness that I was on the scratch host at all. I do not want a pretend root-cause essay generated before I have looked at anything myself. If your real first commands differ, swap them, but keep them read-only and keep the count at three.

The checker that holds the freeze

I run a second script against the journal file and against the suggestion file in the same directory. It fails when the journal has fewer than three non-empty lines, and it fails when the suggestion contains a mutating verb from a short deny-list. It fails closed, so a missing file is a refusal rather than a quiet pass you might miss. Would a short deny-list stop a determined person from pasting a dangerous command somewhere else entirely?

No, and I am not selling this little script as a security boundary against hostile users on your team. It is a tripwire for a tired first draft, not a sandbox and not a privilege boundary. Run it on files you already trust enough to open in an editor. If you need isolation, use the isolation your security team already approved.

#!/usr/bin/env bash
# Proposal only. Unexecuted example for a local drill.
set -euo pipefail
journal="${1:-journal.txt}"
suggestion="${2:-suggestion.md}"
test -f "$journal" || { echo "refuse: missing journal"; exit 2; }
test -f "$suggestion" || { echo "refuse: missing suggestion"; exit 2; }
lines=$(grep -c . "$journal" || true)
if [ "$lines" -lt 3 ]; then
  echo "refuse: journal has ${lines} lines"
  exit 2
fi
if grep -Eiq '(^|[^[:alpha:]])(rm|reboot|shutdown|systemctl|kubectl|terraform|git push)([^[:alpha:]]|$)' "$suggestion"; then
  echo "refuse: mutating verb in suggestion"
  exit 3
fi
echo "hold: journal cleared, freeze still on"

A deny-list is not a boundary

The sample deny-list also blocks read-only kubectl text, so treat it as a starting point rather than a law. Replace those words with the mutating verbs your own estate actually fears in a first draft. I would rather over-block a suggestion file than under-block a restart hidden in polite prose.

What hold is allowed to mean

Please read the success string out loud before you trust the script during a tired night shift. It says hold, and it does not say unfrozen, because the checker must not sound like permission. A human unfreeze is a separate line in the handoff, written by a named person after they have read the journal. The assistant may comment on that unfreeze line, and the assistant may not author it alone.

Escalation stays a phone call, not a paragraph

I name one secondary and one time box in the runbook, and I do not let the assistant choose either of them. If the lane is escalate, I call the secondary before I spend time polishing a suggestion block. If the lane is rehearse, I still call when the scratch journal disagrees with the alert text in a way I cannot explain. How long should you stare at a polished draft before you admit that you are actually stuck?

I use the time box already written down, and a blank time box means I escalate now. Freeze means no production change, no config push, and no restart that someone might later call temporary. Unfreeze means a named human writes the unfreeze line after reading the journal, not after reading a confident suggestion. I keep those verbs apart so a helpful rewrite cannot smuggle a release into a quiet edit.

If your handoff packet has no place for that line, add the place before you add another assistant prompt. I want the unfreeze line to name a person, a time, and the journal file they read. A missing name means the freeze continues, even if the suggestion file looks finished and calm. Would you unfreeze a service because a paragraph sounded sure of itself? I would not, not on a night when I am already behind.

Where a free scratch server is allowed to help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I would use free model access and a free server option only as a rehearsal bench for this fence. The operator supplying this draft says both options are available, and I am treating that claim as supplied rather than tested. I am not stating a token quota, a model list, a hardware size, or any duration. This draft has no primary source for those figures, so inventing them would make the runbook dishonest.

If a current pricing or quota page is your source of truth, read that page before you plan a rotation around it. Would I send production traffic, secrets, or real alert payloads to a free rehearsal server at all? No, and you should not do that either, even when the signup flow makes the server feel disposable.

MonkeyCode shows up here only as a place where I can open a scratch host and ask for wording inside the suggestion file. It does not choose the lane, it does not clear the checker, and it does not write the unfreeze line. If the free option is gone tomorrow, the runbook still works on a laptop with the two scripts above. That fallback matters more to me than any invitation to try a product on a quiet afternoon.

If you want the same fence on a scratch host, start from the operator's current access note and stop where that note stops. I am leaving the product mention there, because a second pitch would not make the checker any stricter. Would another slogan make the hold string any safer when you are tired and the page is still frozen? I do not think so, and I would rather you spend that minute rereading the journal than rereading an ad.

Who should skip this fence

Do not use this fence when the page requires immediate physical safety action, such as cutting power or isolating a live hazard. A journal gate is the wrong first move when people can get hurt while you create a file. Do not use a local bash script as an audit artifact for a control framework you have not scoped with your security team. Do not use an external model when your incident policy forbids third-party tools, even on fixtures that might echo real names.

I also skip the model when the alert already contains secrets, customer payloads, or hostnames I am not allowed to paste into a draft box. I strip the fixture until it is boring, or I skip the model and keep the checker alone. The checker is the artifact I am willing to defend, and the model is optional help for wording. If you need a vendor to be on the hook for uptime, a free rehearsal bench is the wrong dependency for your real rotation.

A six-step drill before the next handoff

I would run this on a laptop, with fake alert text, before I ever mention it in a real channel. Nothing below has been executed for this article, so treat every exit code as something you still need to see yourself. If a step surprises you, stop and fix the script before you talk about it in a handoff. A drill you have not watched fail on purpose is not a drill yet.

Write a one-line fake alert into alert.txt, and keep real hostnames and customer data out of that file.
Create journal.txt with the three-line script, and confirm the printed line count is exactly three.
Create suggestion.md only after the journal exists, and keep every mutating verb out of that file.
Run the checker on both files, and confirm a clean suggestion prints hold rather than a release word.
Paste one denied verb on purpose, rerun the checker, and confirm the process exits with code 3.
Delete the drill files when you finish, so the next page cannot reuse a stale journal by accident.

If step 5 does not fail, the fence is theater and I would not carry it into a rotation. If step 4 prints anything that sounds like permission to change production, rewrite the success string before you trust the script. I am leaving both scripts unexecuted here so you can read them as proposals before you run them anywhere.

The conclusion I am willing to keep

A page gets safer when the first artifact is a journal, not a fix written in a hurry. The assistant can help me phrase a suggestion after that journal exists, and it cannot grant an unfreeze by itself. Free model access and a free scratch server can host the rehearsal inside the limits I already stated. If your runbook already names a human unfreeze and a read-only first move, you can ignore this note, and that is a fine outcome.

Echo the Alert Fingerprint Before an Assistant Draft Leaves Quarantine

Emery Huang — Thu, 08 Oct 2026 07:40:05 +0000

I keep every assistant draft in quarantine until a rehearsal host echoes the same alert fingerprint I copied from the page. A fluent explanation can still describe the wrong incident, the wrong cluster, or a page that already closed an hour ago. Would you really execute a mutating command that cannot name the exact page it claims to repair? I would not take that shortcut, and this runbook is the gate I use before any production change.

How I treat the page when it first lands

I read the alert as a payload first and as a story second, even when the paging app shows a friendly title. The friendly title helps a human get oriented, but it is not the identity of this page. The fingerprint is the identity I will later demand from the rehearsal host, character for character. If the paging app hides that field behind a collapsed panel, I expand it before I copy anything else.

I write this gate because a tidy summary can belong to a different page than the one sitting in my hand. The service name matches, the verbs sound familiar, and the timestamp quietly belongs to a page that already ended. Have you ever accepted a paragraph because the cluster label looked right and the rest felt too boring to check? I want the echo to come from a probe I can hash, not from another sentence that merely sounds careful.

A model can rearrange an old runbook into confident steps without ever seeing the payload that opened this page. It can also blend two alerts that share a service name and differ only in a label a tired reader skips. Would a second generated paragraph catch that mistake if it only saw the same pasted note? I do not treat a second paragraph as evidence, so the rehearsal host must repeat the fields before I trust the note.

Where the draft and the echo are allowed to live

I let the explanation live in a scratch note, and I let the echo live on a host that holds no production credentials. I use MonkeyCode's free model access for the scratch note and its free server option for the echo, and neither one is an approver. The model may rewrite that note, but it may not open a shell on the rehearsal host at all. The rehearsal host may print the fingerprint, but it may not hold a write token for the real cluster.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I need from those two options is a place to draft and a place to echo, not a promise about capacity. Would I paste a production kubeconfig into a chat window just to save a few minutes of typing? I would not, even when the server costs nothing and the chat feels private enough for a quick paste. A free seat does not convert a secret into something safe to share with a drafting tool.

If the rehearsal host can still reach production, I treat that host as production and I stop the drill. Isolation is a property I have to prove, not a feature I get to assume from a price of zero. Have you checked which credentials are already sitting in that host's environment before you call it safe? I stop the echo until that check is written down, because a memory of isolation is not isolation.

What I copy before I ask for an explanation

I copy four fields into the scratch note, and I refuse to add color before those fields are complete. If any field is missing, I escalate with the raw page instead of asking a model to guess the gap. Guessing a timestamp feels helpful in the moment and becomes a lie the moment someone audits the note. Would you want your name beside a startsAt value that a chat invented from the word recently?

Four fields, nothing else

alertname, so the echo cannot quietly drift toward a sibling alert that shares a similar title.
fingerprint, so two similar series cannot impersonate each other inside an otherwise convincing note.
startsAt, so a closed page cannot reuse a summary that was written for yesterday's incident.
generatorURL, so I can see which rule produced the page before I trust any explanation of it.

I leave annotations, suggested fixes, and dashboard screenshots out of that first copy on purpose. Those extras help later, but they tempt the draft to sound finished before any echo exists on disk. Have you noticed how a bright screenshot can make an unverified guess feel like a finished observation? I keep the screenshot in the ticket until the fingerprint match is already written under oncall-state.

First commands, and only these

The first commands stay on my laptop until a minimal page file exists and a hash is written beside it. I am labeling the snippets as a proposal you should adapt, not as a transcript from a cluster I measured. Paths, probes, and field names will differ on your side, and that difference is your job to resolve. What should you do if your alert payload uses different names for the same four ideas?

mkdir -p ./oncall-state
jq '{alertname,fingerprint,startsAt,generatorURL}' page.json | tee ./oncall-state/page-min.json
jq -e '.alertname and .fingerprint and .startsAt and .generatorURL' ./oncall-state/page-min.json
sha256sum ./oncall-state/page-min.json | tee ./oncall-state/page-min.sha256

On macOS I swap sha256sum for shasum with the 256 flag, and I keep the same output file name. If the field check fails, I do not open a model chat and I do not start a rehearsal session. I escalate with the raw page attached and the words fingerprint incomplete in the very first line. Would you ask a model to invent a startsAt value from a vague just now typed in chat?

The rehearsal echo is a read-only probe that prints the same four fields and then exits cleanly. I run that probe only after the local minimal file has a hash I can compare later by eye. A free server helps only when that environment cannot reach production credentials or the paging API. If I cannot prove that isolation, I skip the server and escalate with the local file alone.

I save the probe output as echo.json with a redirect, and I never let the model write that file for me. The redirect is boring, which is exactly what I want from the first commands on a live page. A boring command is easier to read aloud to a secondary who is only half awake. Can you explain a clever one-liner to that person without accidentally opening a second incident?

# Proposal: run on the rehearsal host, not on production.
# Replace the probe with your own read-only status command.
# Save that probe output yourself as ./echo.json first.
jq '{alertname,fingerprint,startsAt,generatorURL}' ./echo.json | tee ./oncall-state/echo-min.json

The comparison I refuse to skip

I keep the comparison in a script so a tired shift cannot decide that two strings mostly match. A partial match is how the wrong series survives a glance and later becomes a restart. This script is a proposal, and I am not claiming I ran it against your pager or scored its misses. Read it, change the paths, and throw it away if your payload cannot supply stable fields.

#!/usr/bin/env bash
# Proposal only. Do not point this at production credentials.
set -euo pipefail
PAGE_MIN="${1:?page-min.json}"
ECHO_MIN="${2:?echo-min.json}"
STATE_DIR="${3:-./oncall-state}"
mkdir -p "$STATE_DIR"

page_fp="$(jq -r '[.alertname,.fingerprint,.startsAt,.generatorURL] | join("|")' "$PAGE_MIN")"
echo_fp="$(jq -r '[.alertname,.fingerprint,.startsAt,.generatorURL] | join("|")' "$ECHO_MIN")"

if [[ "$page_fp" == *"null"* || "$echo_fp" == *"null"* || -z "$page_fp" ]]; then
  echo 'escalate: incomplete fingerprint' | tee "$STATE_DIR/decision.txt"
  echo 'frozen' > "$STATE_DIR/prod.state"
  exit 2
fi

if [[ "$page_fp" != "$echo_fp" ]]; then
  echo 'escalate: echo mismatch' | tee "$STATE_DIR/decision.txt"
  echo 'frozen' > "$STATE_DIR/prod.state"
  exit 3
fi

echo "match: $page_fp" | tee "$STATE_DIR/decision.txt"
echo 'echo-matched-still-frozen' > "$STATE_DIR/prod.state"

Read the state aloud

After the script exits, I read the state file aloud before I touch any other tool on the laptop. A nonzero exit means I stop drafting and I send the decision line to the secondary on call. A zero exit still leaves production frozen, which surprises people who treat a match as approval. Why would a matching string be enough to restart a service you have not otherwise checked?

bash ./fingerprint-gate.sh ./oncall-state/page-min.json ./oncall-state/echo-min.json
cat ./oncall-state/prod.state
cat ./oncall-state/decision.txt

A drill that should fail closed

This fixture should exit on a mismatch and leave the state file frozen, which is the result I want from a drill. If your copy prints a match, you swapped the fingerprints and the drill has not taught you anything yet. Run it locally before you point any probe at a rehearsal host, so a path bug does not look like an incident. Did the frozen state actually stop you from pasting the next command, or did you override it from habit?

mkdir -p ./oncall-state
jq -n --arg alertname LatencyHigh --arg fingerprint abc123 --arg startsAt 2026-10-08T09:00:00Z --arg generatorURL https://example.invalid/rule '{alertname:$alertname,fingerprint:$fingerprint,startsAt:$startsAt,generatorURL:$generatorURL}' > ./oncall-state/page-min.json
jq -n --arg alertname LatencyHigh --arg fingerprint def456 --arg startsAt 2026-10-08T09:00:00Z --arg generatorURL https://example.invalid/rule '{alertname:$alertname,fingerprint:$fingerprint,startsAt:$startsAt,generatorURL:$generatorURL}' > ./oncall-state/echo-min.json
bash ./fingerprint-gate.sh ./oncall-state/page-min.json ./oncall-state/echo-min.json || echo 'drill failed closed, as intended'
cat ./oncall-state/prod.state

Escalation when the echo will not match

I escalate on three results, and I do not ask the model to repair any of them in the chat. A mismatch means I am holding two payloads, not a wording problem the draft can smooth over. An incomplete page means the rule or the receiver dropped a field I refuse to invent tonight. A rehearsal host that offers a write-capable shell is the wrong host, even when the text looks perfect.

Send the secondary the two minimal JSON files and the decision line, not a generated essay about the page.
Say whether the failure is incomplete, mismatched, or a write-capable shell you should abandon immediately.
Stop drafting commands until a human owner names the next read-only probe you are allowed to run.
Keep the original page open so nobody summarizes away the fingerprint while the call is still live.

If the secondary is dark, I use the owner already printed in the service catalog and I wait for that person. I stay on the read-only probe until that person answers, even if the draft already shows a tempting command. I am not inventing a new rotation policy in this note, and I am not replacing your existing roster. I am only refusing to let a generated paragraph become the path that wakes the next person.

Freeze, then a separate unfreeze line

I keep production frozen while the state file is missing, reads frozen, or still reads echo-matched-still-frozen. A fingerprint match is not permission to change anything a user can already feel in production. A match only shows that the rehearsal host repeated the page I think I am holding right now. Would a matching string justify a restart by itself if you still have not checked capacity or scope?

Unfreeze is a separate line I append by hand after that human check, and the model does not write it. The rehearsal host does not write it either, because a host that echoes data should not grant change rights. I want a name, a UTC time, and the fingerprint in that line so a later review can see who moved. If those three pieces are missing, the freeze continues and the draft remains a note, not a plan.

# Human only. Replace YOUR_NAME. Do not pipe model output into this file.
echo "unfreeze YOUR_NAME 2026-10-08T09:30:00Z $(jq -r .fingerprint ./oncall-state/page-min.json)" >> ./oncall-state/unfreeze.log

A decision table I can scan at 3 a.m.

Echo result	State file	What I do next
Incomplete fields	frozen	Escalate with the raw page and stop drafting
Fingerprints differ	frozen	Escalate with both minimal files and stop drafting
Fields match	echo-matched-still-frozen	Human reviews scope, then may write the unfreeze line
Shell can write	frozen	Abandon that host and pick one without production credentials

I can scan that table without asking a model to reinterpret my own rule back to me. If the table and the script disagree, I trust the more restrictive result and I fix the script later. A 3 a.m. argument about which artifact is canonical is how freezes quietly evaporate on a tired team. Would you rather debate the table in the channel, or keep production still until a named human decides?

Limitations, and who should skip this

This gate catches identity mix-ups, and it does not tell you whether the underlying alert is a true page. A bad exporter can stamp the same fingerprint on two different failures, and the script will happily match them. I have no measured false-match rate to offer, so please do not quote this note as a control study. Would you accept a teaching script as proof that your paging pipeline is safe enough for Friday?

Free model access does not make the scratch note true, current, or allowed to approve a production change. A free server option does not prove isolation, network distance, or the absence of copied secrets on disk. You still have to check credentials yourself before the first probe, or the echo is just production with extra steps. If your organization forbids external models from seeing alert text, do not paste the page into one.

Skip this approach if the alert payload has no stable fingerprint, start time, and generating rule URL. Skip it if you cannot place the echo on a host that is sealed off from production credentials entirely. Skip it during a life-safety incident where the existing human runbook already names the immediate action to take. Skip it if you wanted an assistant to act as approver, because this note never grants that role to anyone.

What I still do with the draft

I still read the draft, because a clear sentence can help a tired person notice a label they missed. I just refuse to let that sentence travel until the echo matches and a human writes the unfreeze line. If you already have a free model seat and a free server you can truly isolate, try the gate on a drill. Watch whether a bad echo stops your hands before you ever trust the same habit on a live page.

Keep the Assistant on a Scratch Host Until a Human Diff Unfreezes Prod

Emery Huang — Thu, 24 Sep 2026 20:19:22 +0000

I will not feed a live page to an assistant until a scratch host is named, isolated, and written into the freeze note. Production stays frozen while that host tries to reproduce the symptom with copied fixtures, not live credentials. A generated patch remains a suggestion until a human pastes a diff and signs the unfreeze line. If this feels fussy, ask yourself who you want holding the deploy key at 2 a.m.

Why I separate the model from the page

The failure mode I am writing against is not a dumb suggestion, but a useful suggestion aimed at the wrong machine. Someone pastes the alert, the tool offers a patch, and the same shell session still has production environment variables loaded. Would you trust a completion that can also see your current kube context and your deploy role? I would not, because a single wrong directory is enough to turn a drill into an outage.

I want the assistant where it can be wrong in public, on a host that cannot reach the production network path. That host can be small, temporary, and boring, as long as its name is in the note before the first prompt. The page still needs a human owner, because a model does not carry the pager and does not join the bridge. When the scratch host and the production host share a name, I stop and rename one of them before I continue.

Alerts I will actually accept

I accept a page only when these four lines are filled, and I bounce it back when any line is blank. Each line is a fact I can check, not a vibe about how severe the chart looks. A blank line is a bounce, not a debate, because the bridge should not invent missing facts while the chart is red. Would you rather argue about severity, or spend those minutes filling the four lines I listed below?

The alert name, the firing time, and the service identifier are copied from the page, not retyped from memory.
A symptom query is named, and that query is read-only, so the first action cannot mutate state.
A scratch host name is reserved, and that name is different from every production host in the rotation sheet.
Two clocks are written down: one for scratch reproduction, and one for a human diff review.

First commands I run before any prompt

I start on the production host with commands that only read, and I paste their output into the note before I open a chat. The goal is a symptom packet, not a fix, because the fix conversation can wait until the packet is boring and complete. If a command would change a file, restart a process, or roll a deployment, it does not belong in this first pass. I would rather look slow on the bridge than explain a write command I cannot undo after the fact.

What the probe is allowed to do

# Unexecuted example. Run only on a host you already own.
set -eu
hostname
date -u +%Y-%m-%dT%H:%M:%SZ
git -C /srv/app status --short || true
git -C /srv/app rev-parse --short HEAD || true
# Read-only symptom probe. Replace with your service's real query.
curl -fsS --max-time 5 http://127.0.0.1:8080/healthz || echo "healthz_failed"

I copy that output into a local file called symptom.txt, and I do not upload secrets that sometimes hide in environment dumps. A health check failure is enough context for a first prompt, and a stack trace can be trimmed before it leaves the host. Would you paste a database URL into a chat window just to save one tired minute? I hope your answer is no, because that saved minute is cheaper than rotating a leaked credential later.

What I refuse to paste into the prompt

I keep the prompt smaller than the note, because the note can hold more context than a chat window should see. I paste the alert name, the trimmed health output, and the question I want answered, and I leave credentials out. A model that asks for a token, a kubeconfig, or a production hostname is asking me to break the freeze. I close that chat and continue with the local editor, because convenience is not an exception to the note.

Prompt questions that stay useful

Can you rewrite this gate so a missing freeze note exits before any append, and show the diff only?
Can you list which lines in this health output are symptoms, and which lines are just timestamps?
Can you draft a freeze-note template with scratch host, reviewer, and both clocks, without inventing a service name?

Escalation clocks that are not one vague minute

I do not use one vague escalate-if-needed line, because that line is how pages stall while everyone waits for someone else. I set a reproduction clock at fifteen minutes, and I set a human-diff clock at thirty minutes from the page time. If the scratch host cannot replay the symptom by the first clock, I call the service owner instead of prompting harder. If the human diff is still missing at the second clock, I call a second engineer and keep production frozen.

Those numbers are a starting proposal for a small team, not a benchmark I measured across companies. Change them in your runbook if your bridge already has a tighter rule, and write the chosen numbers into the note. The point is that both clocks exist before the assistant is opened, so the model cannot become the escalation path. Ask yourself who you call when the suggestion looks confident and the graph still looks wrong.

Freeze and unfreeze, with the assistant in the rule

Freeze means no deploy, no config write, and no apply of generated code on any host whose name appears in the production inventory. Unfreeze means a named human has reviewed a diff that was produced on the scratch host and copied back as text. I do not unfreeze on a model claim that tests passed, because I did not watch those tests myself. The freeze note must name the scratch host, the symptom file, and the person who may sign the unfreeze line.

What the scratch gate checks

Here is the gate I want in the scratch session before anyone applies a patch, even on the scratch host. It is a proposal, not a tool I have certified, and you should read every line before you trust it. The script exits if it sees a production marker, because a scratch workflow that runs on prod is just an outage with extra steps. I keep it short so a tired reviewer can actually read it on a bridge call.

#!/usr/bin/env bash
# Proposal only. Not executed in this article. No network calls.
set -euo pipefail

: "${SCRATCH_HOST:?set SCRATCH_HOST}"
: "${FREEZE_NOTE:?set FREEZE_NOTE to a local note path}"
: "${HUMAN_REVIEWER:?set HUMAN_REVIEWER}"

if [[ "${PRODUCTION:-0}" == "1" || "${DEPLOY_ENV:-}" == "prod" ]]; then
  echo "refusing: production marker is set" >&2
  exit 2
fi

this_host="$(hostname -s)"
if [[ "${this_host}" != "${SCRATCH_HOST}" ]]; then
  echo "refusing: hostname ${this_host} is not ${SCRATCH_HOST}" >&2
  exit 3
fi

if [[ ! -f "${FREEZE_NOTE}" ]]; then
  echo "refusing: freeze note missing" >&2
  exit 4
fi

grep -q "scratch_host=${SCRATCH_HOST}" "${FREEZE_NOTE}" || {
  echo "refusing: freeze note does not name this scratch host" >&2
  exit 5
}

printf '%s host=%s reviewer=%s action=scratch_ok\n' \
  "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${this_host}" "${HUMAN_REVIEWER}" \
  >> "${FREEZE_NOTE}"
echo "scratch gate passed; still frozen until a human diff is signed"

A three-step drill before the rotation

I treat the script as untrusted until this drill passes on a host that has no production route. The drill is a proposal you can run in a shell, and I have not published timing numbers from it. If a step fails, fix the script or the note, and do not skip ahead to a model prompt. Would you trust a gate you have never watched fail on purpose during a quiet drill?

Export PRODUCTION=1 on the scratch host and confirm the gate script exits with status 2 before you unset the variable.
Point SCRATCH_HOST at a wrong name and confirm the script exits before it appends a line to the freeze note.
Run the script with matching names and confirm it appends one scratch_ok line and still prints that production remains frozen.

A small decision table for the bridge

I keep this table in the runbook so the bridge does not invent a new policy while the graph is red. Read the row, do the action, and write the time next to the row you used. If two rows seem to match, I take the stricter row and I stay frozen until the note is updated. Would you rather argue about a row for ten minutes, or follow the stricter one and argue after the page is quiet?

Situation	What I do	Freeze state
Symptom file is missing	Run the read-only probe, save output, do not prompt	Stay frozen
Scratch host name matches a prod host	Rename the scratch host and update the note	Stay frozen
Reproduction fails before the first clock	Call the service owner with the symptom file	Stay frozen
Patch exists but no human diff	Wait, or call the second engineer at the second clock	Stay frozen
Human diff is signed and tests were watched	Unfreeze only the named service, then watch the same probe	Unfreeze that service

Where a free assistant fits, and where it does not

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I want a second editor only for the scratch-host script, the freeze note, and the trimmed symptom summary. That is the only job I will give an assistant during a page, and I will not hand it the deploy role. MonkeyCode is relevant here as an open-source assistant only when free model access and a free server are still offered. I am not naming a token quota, a model, or hardware, because this draft does not cite a current pricing page.

Check the project docs yourself before you plan a shift around any free tier you might have heard about. Treat a missing or unclear limit as a reason to stay on tools you already run locally. Would you bet a page on a free tier whose current cap you have not opened and read? I would not, so the runbook names the local editor as the fallback when the docs are silent.

The free server option matters only as a candidate scratch host, and only after you confirm it cannot see production credentials or production networks. I would use it to rewrite the gate script, to draft the freeze note, or to replay a fixture I already copied by hand. I would not point it at a live cluster, and I would not paste deploy keys into the prompt to save a step. If the free server and the production VPC share a route, it is not a scratch host, no matter how convenient the signup felt.

Remove the product name and the workflow still stands: isolate the editor, freeze production, demand a human diff, and escalate on two clocks. The assistant is a text worker on a disposable machine, not an on-call engineer and not a change-management system. If the free access disappears tomorrow, I still want the same gate script and the same table in the runbook. That is the test I use before I mention a tool in a page procedure at all.

Limits, and who should not copy this

This approach assumes you can spare a separate host, a local note file, and a second human before the second clock expires. It is a poor fit for a solo operator who has no one to sign the diff, because the unfreeze rule then becomes theater. It is also a poor fit for a data-loss page that needs a vendor bridge in the first minutes. A scratch replay can waste the only useful minutes when the vendor already holds the timeline.

I have not measured mean time to recovery with this gate, and I will not pretend a blog draft is an incident report. Do not use this script as a security boundary against a hostile insider, because it only checks a few environment markers and a hostname. Do not store customer data in the symptom file, and do not treat a model transcript as an audit log. If your regulator wants a signed change record, keep using that system and let this note point at it.

What I would do before the next page

I would copy the gate script into a private drill repo and point it at a host that is already empty. I would then fail the script on purpose with a production marker, and I would keep the failure in the drill note. After that drill passes, I would add the decision table and pick both clocks with the people who carry the pager. The page gets safer when the freeze note names a host a human can still refuse, not because a tool was free.

Name a Rollback File in the Handoff Packet Before You Touch the Alert

Emery Huang — Wed, 23 Sep 2026 16:47:25 +0000

I will not accept a production page until the handoff packet names a rollback file. That sounds stubborn on a noisy night, and it should, because unowned mitigations become the next incident. Alerts without a five-minute diagnostic budget turn into guesswork, and guesswork is how freeze rules get skipped. So the packet comes first, freeze comes second, and any assistant-generated command stays off production.

Speed from agents is not permission to skip the packet

People keep asking whether an agent should call production APIs without ever leaving the editor window. Have you watched a tool-calling demo and felt your pager twitch at the same time? I have, and that uneasy feeling is exactly why this handoff packet exists before diagnosis. Fast patches are useful on a scratch box, but they are reckless while the customer path stays unfrozen.

The public conversation around agents and tool calling is loud right now, and on-call is the quiet counter-argument. An assistant that can draft a command is not an assistant that may run that command. I want command generation isolated from the cluster that is actually paging me right now.

Fill the handoff packet before you type a diagnostic

I keep a YAML packet next to the service, and I refuse to start diagnosis until every required field has a value. The packet is not a novel, and it is not a postmortem; it is a small transfer object. If I get hit by a second alert, the backup on-call should continue from this file alone.

# proposed example: oncall-handoff.packet.yaml
apiVersion: oncall.packet/v1
kind: HandoffPacket
metadata:
  alert_id: example-ALERT-0000
  service: checkout-api
  opened_at: 2026-09-23T02:14:00Z
spec:
  diagnostic_budget_seconds: 300
  rollback_file: deploy/rollback/checkout-api.last-good.txt
  rollback_file_sha256: pending
  customer_visible: true
  freeze_scope: checkout-api-canary
  unfreeze_owner: primary-oncall
  escalation_target: payments-secondary
  first_commands:
    - kubectl get deploy checkout-api -o wide
    - kubectl logs deploy/checkout-api --tail=200 --since=10m
    - curl -sS https://status.internal.example/checkout-api/healthz
  notes: proposed packet; do not treat hostnames as real inventory

Does that YAML look heavy for a single page, or does it look lighter than a wrong restart? Every field exists because I have watched pages drift when the rollback path lived only in someone's head. I treat missing fields as a failed start, not as optional documentation I can fill after mitigation.

Fields I fill in a fixed order

I copy alert_id from the page so the packet and the pager cannot silently diverge later.
I point rollback_file at a real artifact so mitigation is a file path, not a remembered command.
I set diagnostic_budget_seconds to three hundred so the first five minutes cannot quietly become an hour.
I name freeze_scope as a concrete surface, because freezing the whole company is not a strategy.
I name unfreeze_owner as one person, so thawing production is never an implied group decision at night.
I name escalation_target as a different person, so a spent budget already has a next human.

Those six fields are the contract I accept with the rotation, and I will not bargain them down during the incident. If a field is unknown, I page the service owner instead of inventing a placeholder that looks complete. A pretty packet with fake owners is worse than a refused page, because it launders uncertainty into action.

The rollback file itself is boring on purpose, and boring is what I want under a paging alert. I keep last-good image, replica count, and config identity in one text file beside the deploy path. If that file is missing, the validator below refuses the page before I touch a shell.

# proposed example: deploy/rollback/checkout-api.last-good.txt
image: checkout-api:sha-example
replicas: 3
configmap: checkout-api-2026-09-01

First commands stay read-only even when a restart looks obvious

I map each alert to commands that cannot change cluster state, and I paste them from the packet. If a command needs apply, delete, restart, or scale, it does not belong in the first_commands list. Ask yourself this: would I still run that command if the unfreeze owner were offline tonight?

# proposed read-only first commands for a latency page
kubectl get deploy checkout-api -o jsonpath='{.spec.replicas}{"\n"}'
kubectl describe pod -l app=checkout-api | sed -n '1,80p'
kubectl get events --field-selector involvedObject.name=checkout-api --sort-by=.lastTimestamp
curl -sS -o /tmp/healthz.json -w "%{http_code}\n" https://status.internal.example/checkout-api/healthz
sha256sum deploy/rollback/checkout-api.last-good.txt

I record the output hashes in the packet notes before I argue about causes with anyone. Why bother hashing terminal output during a live page, when the logs are already scrolling away? Because the next person should see the same evidence I saw, not a vanished buffer.

A freeze annotation is not a first command, even when the dashboard is screaming and the room wants motion. I will sketch it with --dry-run=client after the budget expires, and I still will not apply it from an assistant session. Human ownership stays on the freeze, and the scratch box stays a scratch box.

# not a first command; only after budget expiry and named freeze_scope
# proposed freeze annotation, human-owned, never assistant-owned
kubectl annotate deploy checkout-api oncall.packet/frozen=true --dry-run=client -o yaml

Freeze only what the packet names, then keep unfreeze as a second signature

The question is not whether freezing feels dramatic; the question is which named surface the packet is allowed to freeze. I freeze only the freeze_scope listed in the file, and I do not freeze adjacent services because the dependency graph looks scary. Unfreeze requires the named unfreeze_owner, a current packet, and a rollback file whose checksum still matches.

If any of those three checks fail, production stays frozen, and I escalate instead of improvising a thaw. Is that slower than a hopeful restart on a red dashboard at two in the morning? Yes, and that slower path is the trade I want when the customer path is still burning.

Packet state	Customer visible?	Action I take	Who may unfreeze
rollback_file missing	yes	refuse the page, page the service owner	nobody yet
budget still open	yes	run first_commands only	unfreeze_owner after budget
budget expired, cause unknown	yes	freeze freeze_scope, escalate	escalation_target after review
cause known, rollback_file present	no	optional freeze, then planned change	unfreeze_owner
assistant drafted a patch	either	keep patch on scratch server	never the assistant

That table is the whole policy I will defend on the incident bridge without extra rows. If a situation is not a row in the table, I do not invent a fifth path while the alert is still firing. Improvisation belongs in the postmortem, and it does not belong in the unfreeze signature.

The five-minute clock is a hard stop, not a suggestion

I start a timer when the packet validates, and I do not extend it because the logs look interesting. Interesting logs after minute five belong in the escalation note, not in a longer local debug session. If the first commands did not name a cause, I freeze freeze_scope and I stop typing.

I also stop asking an assistant for one more hypothesis once the budget is gone. Extra hypotheses are cheap; extra mutations during freeze are how you write the next page. The clock exists to protect the freeze rule, not to make me look decisive on the bridge.

Where an isolated assistant belongs in this loop

I still want help drafting the packet, especially when a new alert type shows up without a template. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode as a coding assistant with free model access and a free server option, kept off production. The product is in the loop only as a scratch drafting aid, never as the unfreeze owner.

The useful loop is narrow, and I will not pretend it replaces the on-call human holding the pager. I paste a redacted alert description into the isolated session and ask for candidate read-only commands, then I reject anything that mutates state. The free server is a scratch place to run the validator below, not a jump host into the cluster that is paging.

# proposed prompt I keep next to the packet (unexecuted example)
Redact customer data. Propose only read-only diagnostic commands.
Each command must name the binary, the resource, and the output I should save.
Do not propose kubectl apply, delete, rollout restart, or scale.
Do not propose unfreeze steps. Return a YAML fragment for first_commands only.

If you want a scratch box for that drafting loop, MonkeyCode's free server option is one isolated place I will try. I still copy the YAML back into the repository by hand, because the packet is an owned artifact, not a chat transcript. No generated fragment lands in git until I have read every command against the mutating-word list.

A validator I run before I accept the rotation

This is a proposed Python check, not a production metric, and it fails closed when required fields are empty. Run it on the scratch server or on your laptop; do not run it as a cluster controller. I want a refuse-the-page exit code, not a dashboard that argues with the freeze owner.

#!/usr/bin/env python3
"""Proposed handoff packet validator. Example only; not production incident data."""
from __future__ import annotations

import hashlib
import re
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    print('install pyyaml in the scratch environment, not on the frozen host')
    sys.exit(2)

MUTATING = re.compile(
    r'\b(apply|delete|restart|scale|cordon|drain|replace|patch)\b',
    re.I,
)

REQUIRED = (
    'alert_id',
    'rollback_file',
    'diagnostic_budget_seconds',
    'freeze_scope',
    'unfreeze_owner',
    'escalation_target',
)


def load_packet(path: Path) -> dict:
    data = yaml.safe_load(path.read_text())
    if not isinstance(data, dict):
        raise ValueError('packet must be a mapping')
    spec = data.get('spec') or {}
    meta = data.get('metadata') or {}
    return {**meta, **spec}


def main() -> int:
    packet_path = Path(sys.argv[1] if len(sys.argv) > 1 else 'oncall-handoff.packet.yaml')
    pkt = load_packet(packet_path)
    errors: list[str] = []

    for key in REQUIRED:
        if not pkt.get(key):
            errors.append(f'missing {key}')

    budget = pkt.get('diagnostic_budget_seconds')
    if not isinstance(budget, int) or budget <= 0 or budget > 900:
        errors.append('diagnostic_budget_seconds must be an int between 1 and 900')

    rollback = Path(str(pkt.get('rollback_file', '')))
    if not rollback.is_file():
        errors.append(f'rollback_file does not exist: {rollback}')
    else:
        digest = hashlib.sha256(rollback.read_bytes()).hexdigest()
        print(f'rollback_file sha256={digest}')

    commands = pkt.get('first_commands') or []
    if not commands:
        errors.append('first_commands must contain at least one read-only command')
    for cmd in commands:
        if MUTATING.search(str(cmd)):
            errors.append(f'mutating first command: {cmd}')

    if pkt.get('unfreeze_owner') == pkt.get('escalation_target'):
        errors.append('unfreeze_owner and escalation_target must be different people')

    if errors:
        print('REFUSE THE PAGE')
        for item in errors:
            print(f'- {item}')
        return 1

    print('packet complete; freeze_scope may be applied by the named owner')
    return 0


if __name__ == '__main__':
    raise SystemExit(main())

python3 validate_handoff_packet.py oncall-handoff.packet.yaml
# expected for a complete packet: prints sha256 and a single success line
# expected for a broken packet: prints REFUSE THE PAGE and exits 1

If the validator prints REFUSE THE PAGE, I do not start first commands, and I do not ask an assistant to skip the check. The backup on-call can run the same script on their laptop and should get the same refusal. That repeatability is the only reason I bother checking the handoff packet in code at all.

Escalation is a packet transfer, not a feeling

When the budget clock hits zero, I stop proposing new hypotheses and I freeze the named scope. I paste the packet, the command output hashes, and the validator result into the escalation note. Then I page escalation_target and I stay on the call as a reader, not as a second unfreeze owner.

What do I refuse to send during that handoff, even if the channel is already busy? I refuse secret-mixed chat logs, assistant-only patches, and freezes of services the packet never named. Escalation moves the packet, and it does not dump my anxiety onto the next engineer.

Limitations, and who should not copy this

This workflow assumes you can name a rollback file before the page, which baby services and weekend prototypes often cannot. If you have no deploy artifact, no service owner, and no secondary, the validator will refuse every page, and that refusal is correct. Do not use this contract as a reason to skip paging humans who actually know the service.

I also will not run assistant-drafted commands against production because a free server made the script look tidy. The isolated environment has no authority over freeze or unfreeze, and it should stay that way. Teams with a formal incident commander already owning freeze policy should not bolt this packet on as a second source of truth.

If your alerts are purely informational, a five-minute budget is theater, and you should fix routing instead of filling YAML. This article is a proposed on-call contract, not a claim about latency numbers, token quotas, or model rankings I did not measure. Use it where ownership is real, and ignore it where the service cannot even name a rollback file.

Keep Production Frozen Until the Runbook Names Blast Radius

Emery Huang — Tue, 22 Sep 2026 15:03:57 +0000

I will not unfreeze production until the runbook names blast radius and a hard command budget. A page without those two fields is just a loud ticket, and I treat it that way on every rotation. Can you point at the services that freeze with this alert, or are we still guessing in Slack? Guessing is how a careful read-only check becomes a write that nobody on the rotation can roll back.

Why this runbook shape exists

Most on-call docs I inherit list symptoms, then dump a wall of kubectl and curl without saying what they can break. They never say how many commands I may run before I must escalate to a human who owns the blast radius. Is that a runbook, or is it a wiki page wearing a pager costume at three in the morning? I want a contract that the alert, the first command, and the freeze rule can all satisfy without improvisation.

The industry keeps handing us faster assistants and louder suggested patches, and tired engineers will paste those patches into prod. I still want drafting help, but I want it on a box that cannot touch production traffic or secrets. Does your current runbook even say where generated output is allowed to live during an incident? Mine says the draft lives offline, and the freeze stays up until a human signs the unfreeze.

The four fields I require on every paging alert

Every paging alert in the runbook gets four required fields before I will accept the next rotation. If a field is missing, I refuse the page and I keep the named services frozen. Why would I accept a rotation that cannot tell me what breaks when checkout-api pages at 3 a.m.? I would not, and I have started saying that out loud during the handoff, not after the page.

Here is what I require, written as a list so reviewers cannot pretend they missed a field:

Alert name and severity — page, ticket, or ignore, with no silent severity drift between docs and the pager.
Blast radius — the services, queues, and data stores that freeze together with this alert.
First command — one read-only check with a timeout and a known-good output shape.
Command budget — how many commands I may run before escalation is mandatory and writes stay blocked.

A YAML contract I will actually merge

The YAML I keep in git looks like this, and I reject pull requests that leave these keys empty.

# runbook/alerts/checkout-latency.yaml
# Proposed template — review before you attach it to a pager.
alert: CheckoutP99High
severity: page
customer_facing: true
blast_radius:
  services: [checkout-api, payments-worker]
  datastores: [checkout-redis]
  queues: [checkout-jobs]
first_command:
  argv: ["kubectl", "-n", "checkout", "get", "deploy,po,hpa"]
  timeout_seconds: 20
  mutates: false
  known_good: "all checkout-api pods Ready"
command_budget:
  max_commands: 3
  after_budget: escalate
escalation:
  primary: checkout-oncall
  backup: payments-oncall
  channel: "#inc-checkout"
freeze:
  default: frozen
  unfreeze_requires: [blast_radius_named, budget_not_exceeded, human_signature]
  ai_may_unfreeze: false

Does that look heavier than a markdown heading and a prayer at three in the morning? Good, because pager weight should hurt a little, or people will keep paging on pure noise. I would rather fight a noisy review than discover a missing blast radius while the pager is screaming. The file is the contract, and Slack is not a contract no matter how many emoji ACK the thread.

First commands that cannot mutate

The first command is not a fix, and I will keep repeating that until the runbook files agree. The first command is a photograph of the blast radius, and it must finish inside the timeout. I want kubectl get, curl on a health path, or a read-only ping, never apply, never delete, never migrate. If your so-called first command needs a write flag, it is not first; it is an unfreeze request wearing a costume.

Would I trust a generated snippet that restarts a deployment because p99 looks sad on a graph? Not before I have the photograph, and not before the freeze is already locked on the blast radius. The runbook must name the jumphost, and it must say the command is read-only without exception. If either line is missing, I keep production frozen and I ping the backup on-call instead of inventing a shell history.

# Proposed read-only photograph. Do not point this article at production.
kubectl -n checkout get deploy,po,hpa -o wide
kubectl -n checkout get deploy checkout-api
kubectl -n checkout get po -l app=checkout-api
curl -fsS --max-time 5 https://checkout.internal/healthz
redis-cli -h checkout-redis PING

Known-good output is part of the command

I also want a known-good string in the YAML so a half-ready replica set cannot pass as healthy. The phrase all checkout-api pods Ready is boring, and boring is exactly what I want at 3 a.m. Fancy dashboards can wait until after the photograph matches, which is a rule I will not bargain. If the known-good line is missing, the first command is incomplete, and incomplete commands do not unlock an unfreeze.

Escalation after the command budget

Three commands is my default budget, and I do not negotiate that number during the incident. After the photograph, one extra read, and one documented diagnostic, I escalate even if I almost see the bug. Have you noticed how almost is when people start typing writes from memory instead of paging backup? The budget exists to interrupt that reflex before it ships a worse outage on the same blast radius.

Role, backup, channel

Escalation in this runbook is a role, a backup role, and a channel, not a feeling in the incident thread. I do not need a theatrical countdown; I need a named human who owns the blast radius when my budget is gone. If checkout-oncall does not answer, payments-oncall is already written down, and I do not invent a third person from the directory. The channel is #inc-checkout, and transcripts of the three commands go there before I page anyone.

After command 3:
  1. Post the photograph and the three command transcripts in #inc-checkout.
  2. Page checkout-oncall with blast_radius in the first line of the message.
  3. If no ACK, page payments-oncall with the same blast_radius line.
  4. Do not type a write. Freeze stays up.

Freeze and unfreeze are different signatures

Freeze is the default when a customer-facing alert fires against a named blast radius in git. Unfreeze is a signature, and it is never automatic just because a dashboard turned green for thirty seconds. Can a model lift that freeze because it wrote a plausible patch in confident prose during the page? Not in this runbook, because ai_may_unfreeze false is a field the validator rejects if you delete it.

What the signer must restate

I treat generated output as a draft comment on the incident thread, never as an unfreeze token for production. The human who signs must restate blast radius, confirm the command budget was not exceeded, and name the write. If they cannot restate those three things in one message, the freeze stays, and I am comfortable looking stubborn.

Drafting the runbook offline

I still want help drafting these YAML files and the read-only first commands after a long shift. Writing them by hand when I am tired is exactly how blast radius and budget fields go missing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode's free model access and free server option as an isolated drafting bench, not as a production jumphost.

It is also not something that can unfreeze a service, and I will not give it that job. The drafting loop looks like this, and every step stays off the pager host on purpose. I want the model to propose fields I can reject, not a patch I can apply. That distinction is the whole reason the freeze exists in the YAML.

The offline loop

Paste the alert name and the suspected services into the free server session.
Ask the free model access to propose a blast_radius list and a read-only first_command.
Reject any proposal that includes apply, delete, migrate, restart, or scale.
Copy the surviving YAML into a local git branch and run the validator below.
Only then open a runbook pull request for humans who own the blast radius.

Is this slower than letting a copilot patch live traffic during the page, while freeze is still down? Yes, and that slowness is the entire point of keeping the freeze in place on the blast radius. The free server is useful because it is not production, and I will not pretend it observes your metrics. If you draft runbooks the same way, that isolated bench is the only product role I am describing here.

I will not claim token ceilings, model names, hardware, or how long a free plan lasts in this article. I have not verified those details from this desk, and outdated numbers help nobody on a rotation. Free model access and a free server are enough for this drafting loop as I described it. Anything beyond that should be checked on the project itself before you depend on it.

A validator you can run before you accept the rotation

This is the artifact I actually want in CI, because reviewers miss empty keys at 5 p.m. on Friday. It does not page anyone, and it should never run against a production cluster or jumphost. It fails the build when the runbook is still a wiki page in disguise, which is the usual case. Can you merge a runbook that allows a model to unfreeze checkout-api through a missing field?

#!/usr/bin/env bash
# validate-runbook.sh — proposed local check, not a production controller.
set -euo pipefail

FILE="${1:?usage: validate-runbook.sh alerts/foo.yaml}"

need() {
  local key="$1"
  grep -Eq "^${key}:" "$FILE" || { echo "missing ${key} in ${FILE}" >&2; exit 1; }
}

need "alert"
need "severity"
need "blast_radius"
need "first_command"
need "command_budget"
need "escalation"
need "freeze"

grep -q "mutates: false" "$FILE" || { echo "first_command must set mutates: false" >&2; exit 1; }
grep -q "ai_may_unfreeze: false" "$FILE" || { echo "AI must not be allowed to unfreeze" >&2; exit 1; }

if grep -Eiq "kubectl apply|kubectl delete|migrate|DROP TABLE" "$FILE"; then
  echo "runbook contains a mutating command; refuse the rotation" >&2
  exit 1
fi

echo "ok: ${FILE} names blast radius, a read-only first command, and a freeze AI cannot lift"

Run it on every file under runbook/alerts before you say yes to the next week of nights. The find pipeline below is the whole pre-rotation gate I want, and it belongs in CI rather than in a sticky note. If the script exits nonzero, I refuse the rotation, and I send the failure to the service owner. That refusal is cheaper than an unfreeze that nobody can explain later in the incident timeline.

find runbook/alerts -name '*.yaml' -print0 | xargs -0 -n1 ./validate-runbook.sh

Decision table I keep next to the pager

I keep this table next to the pager, and I treat a skipped row as a failed runbook, not a shortcut. Can a green graph override a missing blast_radius line in git during the first minute? It cannot, because the table says freeze and refuse, not eyeball the dashboard. I want the next on-call to make the same call I would make without a private Slack aside.

Situation	Freeze?	Next action	Unfreeze allowed?
Alert missing `blast_radius`	Yes	Refuse the page; ping runbook owner	No
First command would mutate	Yes	Rewrite as read-only or escalate	No
Command budget spent	Yes	Page primary, then backup	No
Model proposed a patch	Yes	File the draft offline only	No
Human restates blast radius and the write	Stays until signature	Second on-call ACKs	Yes, after ACK

Would I skip a row because the dashboard looks fine for a minute during a noisy page? That is how silent data loss starts, so I do not skip rows even when graphs look friendly. The table is part of the runbook in git, not a slide I remember from last quarter's retro. If a situation is not in the table, I escalate rather than inventing a fifth kind of unfreeze.

What this does not solve

This workflow assumes you can freeze a blast radius without taking down the whole company, which is not always true. It also assumes git-hosted runbooks and a CI hook, which a two-person team on a shared cluster may not have. The validator is a grep net, not a policy engine, and a determined paste can still smuggle a write inside quotes. A drafting bench does not watch production metrics, and it does not replace an incident commander who can say no.

I have not claimed latency numbers or model catalogs here because I cannot verify them from this desk today. If your compliance rules forbid sending alert names to any external drafting tool, skip the bench entirely. Keep the YAML rules anyway, because blast radius and command budget do not depend on a drafting product. The freeze still belongs to humans, and that part of the runbook should survive any tool change.

Who should not use this

Do not use this if you are already in an active incident and you still lack a freeze path. Do not use it if legal will not allow service lists in an external editor, even a free server. Do not use it as an excuse to page fewer people while you experiment with generated YAML on the jumphost. Those three cases need a human process first, not another file that pretends to be a runbook.

If you cannot name blast radius, you do not get an unfreeze, and I will not pretend otherwise. If you cannot name a command budget, you do not get an unfreeze either, even when the graphs look calm. Everything else, including a helpful draft from an offline bench, waits until those two lines exist in git. That is the whole method, and it is intentionally boring on purpose for the people who get the page.

Refuse the Page Until the Runbook Names an Escalation Minute

Emery Huang — Mon, 21 Sep 2026 13:16:09 +0000

I refuse every new page until the runbook names an escalation minute, a first command, and a freeze owner. AI drafts can fill comments on a throwaway box, but they cannot sign those three fields for me. A document without a clock is a blog post, and I will not carry it overnight. Would you really take a rotation whose escalation path still says "use your judgment"?

Why the escalation minute comes first

Alerts without a minute mark turn into private investigations that never escalate and never freeze writes. Night threads often show six people typing kubectl, and nobody can say when the next role should be paged. The runbook has to name that minute before I accept the rotation, not after the customer already felt the outage. If that escalation clock is missing from git, what exactly are you pretending to be on call for?

I do not start from a chatbot essay about incident best practices, because essays do not page humans. I start from the alert name, then I write the first read-only command, then I write the minute, then I write who freezes. That strict order keeps generated text inside comments, where it cannot open a production shell tonight. Why would I hand a model the pager when it still cannot name the unfreeze owner?

The signed runbook shape I actually keep

I keep one YAML file per service, and I treat unsigned keys as pages I will not accept. The file has to list alerts, first commands, escalation, and a freeze contract, because those are the only night-safe objects. Generated prose can live under a drafts key, and the validator must fail if that key is still unsigned. Does your repo still store the real procedure in a wiki that deploys independently of git?

Here is a template, labeled as a proposal, not as a dump from some secret production cluster.

# proposal: service runbook, unsigned until owners.sign == true
service: payments-api
owners:
  sign: false
  freeze: "sre-payments"
  unfreeze: "sre-payments-lead"
drafts:
  source: "model-comment-only"
  signed: false
  notes: "AI may propose text here; it must never become a command."
alerts:
  - id: PaymentsP99Latency
    severity: page
    silence_minutes: 15
    first_commands:
      - "kubectl --context=readonly get deploy payments-api -o wide"
      - "kubectl --context=readonly logs deploy/payments-api --tail=80"
    escalation:
      minute: 10
      to: "sre-payments-lead"
      page_back_when: "p99 < 300ms for 10m AND error_rate < 1%"
    freeze:
      on_clock_start: true
      writes: ["deploy", "configmap", "rollout"]
      unfreeze_evidence: "page_back_when cleared in this file"

Fields I will not negotiate

alerts[].id must match the pager string, including punctuation, because fuzzy titles create the wrong first command.
first_commands must be read-only invocations, and the wrapper below rejects anything that is not in this list.
escalation.minute is an integer clock that starts at ack, not a feeling about how bad the graph looks.
page_back_when is a measurable condition, because "looks better" is how silent failures return at dawn.
freeze.on_clock_start means writes stop when escalation starts, not after someone remembers the wiki.
drafts.signed must stay false until a human copies accepted text into the signed keys.

First commands are read-only or they do not ship

I want the first five minutes to be evidence, not mutation, because mutation without a freeze owner is just another outage. Every first command in the file above is a get or a log read, and the allowlist wrapper will refuse a rollout. If a model suggests kubectl delete, that line stays in drafts.notes until a human rewrites it after unfreeze. Are you still pasting generated kubectl into production before the runbook even names the write blast?

I keep a boring wrapper so tired humans cannot "just this once" run a stray apply.

#!/usr/bin/env bash
# proposal: allowlist runner for first commands only
set -euo pipefail
RUNBOOK="${1:?runbook yaml}"
ALERT="${2:?alert id}"
shift 2
CMD="$*"

python3 - "$RUNBOOK" "$ALERT" "$CMD" <<'PY'
import sys, yaml
runbook, alert_id, cmd = sys.argv[1], sys.argv[2], sys.argv[3]
doc = yaml.safe_load(open(runbook))
if not doc.get("owners", {}).get("sign"):
    raise SystemExit("unsigned runbook: refuse to exec")
if doc.get("drafts", {}).get("signed") is True:
    raise SystemExit("drafts cannot be signed; copy text into alerts first")
alert = next(a for a in doc["alerts"] if a["id"] == alert_id)
if cmd not in alert["first_commands"]:
    raise SystemExit("command not in first_commands allowlist")
print("allow")
PY

# reached only after the allowlist printed allow
# first_commands already pin --context=readonly
exec $CMD

What this wrapper is allowed to do

Read the signed YAML from git, not from a paste in Slack, before any shell starts.
Refuse the whole night if owners.sign is still false on the branch that can page.
Compare the exact command string, because a nearby destructive cousin is still a miss.
Leave writes to the freeze contract, because first-command territory ends when mutation begins.

That wrapper is deliberately rude, and I actually like it that way during a noisy page. It will not run an unsigned file, and it will not run a command the alert did not name. If you need a write, you are no longer in first-command territory, and the freeze contract has to speak. Should a tired operator be able to skip the allowlist because a chatbot sounded confident?

Escalation is a clock with a page-back condition

I write escalation.minute: 10 when ten minutes of read-only evidence is enough to stop being a lone hero. At that minute the named role is paged, freeze stays on, and I stop collecting extra screenshots for a thread nobody will read. The page-back condition has to be something a graph can answer, not a vibe about error budget remaining. Who is supposed to page you back if p99 only recovered because customer traffic already died?

Escalation ladder I paste into the service file

Minute 0: ack the page, run only first_commands, and start the freeze clock if the alert is a page.
Minute 10: page escalation.to with the command output attached, not with a novel about possible causes.
Minute 20: if page_back_when is still false, page the unfreeze owner, who is the only person allowed to lift writes.
Any earlier recovery: leave freeze on until page_back_when is recorded as cleared in git, not in chat.

I want those minutes in YAML because chat messages vanish, and memories lie after adrenaline fades. The validator below fails the pull request when minute is missing or when page_back_when is an empty string. That failure is the whole point, because empty escalation is how teams pretend they have a runbook. Can you show me the minute in git, or only in a slide from last quarter?

Freeze starts when the clock starts

I do not wait for a dramatic graph before I freeze writes, because drama arrives after the bad config already rolled. When the escalation clock starts, deploys, configmaps, and rollouts for that service are blocked until the unfreeze owner records evidence. Unfreeze is a different signature from freeze, and the YAML names two different people on purpose. If one person can freeze and unfreeze alone, what stops a panicked apply at minute nine?

This is not a platform design essay; it is a git-enforced habit for a single service file. You can block writes with an admission policy, a deploy queue, or a locked pipeline without buying a new vendor. The runbook only has to name the rule so humans do not argue about it while the pager is screaming. Is your production freeze still just a Slack emoji from whoever happens to be awake?

Draft on a throwaway box, never on the bastion

I will let a coding model propose YAML comments, but I will not let it propose a live shell on the production bastion. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode only as a drafting place, because free model access and a free server keep the skeleton off the pager host. The model fills drafts.notes, I copy surviving lines into signed keys, and then I set owners.sign on my laptop. If the draft still contains writes, why would I promote those lines into first_commands tonight?

The useful artifact remains the signed file in git, not the editor that proposed the comments. Remove the drafting box and the validator still protects you, which is how this workflow should work. Generated confidence is not an escalation path, and a free server is not a production replica. Do not paste a model rollout into the allowlist and call it a runbook.

The validator I run in CI

I run this as a proposal script in CI, and I have not baked it into a particular platform. It fails closed: unsigned owners, signed drafts, missing minutes, empty page-back, or a first command that looks like a write.

#!/usr/bin/env python3
"""Proposal: fail CI unless the on-call runbook is night-safe."""
from __future__ import annotations

import re
import sys
from pathlib import Path

import yaml

WRITE_HINTS = re.compile(
    r"\b(apply|delete|replace|patch|rollout\s+undo|scale|edit)\b",
    re.I,
)


def fail(msg: str) -> None:
    raise SystemExit(f"runbook invalid: {msg}")


def main(path: str) -> None:
    doc = yaml.safe_load(Path(path).read_text())
    owners = doc.get("owners") or {}
    if owners.get("sign") is not True:
        fail("owners.sign must be true before the file can page humans")
    if not owners.get("freeze") or not owners.get("unfreeze"):
        fail("freeze and unfreeze owners must be named")
    if owners.get("freeze") == owners.get("unfreeze"):
        fail("freeze and unfreeze must be different humans or roles")

    drafts = doc.get("drafts") or {}
    if drafts.get("signed") is True:
        fail("drafts.signed must stay false; copy text into alerts first")

    alerts = doc.get("alerts") or []
    if not alerts:
        fail("at least one alert is required")

    for alert in alerts:
        aid = alert.get("id") or "<missing>"
        cmds = alert.get("first_commands") or []
        if not cmds:
            fail(f"{aid} has no first_commands")
        for cmd in cmds:
            if WRITE_HINTS.search(cmd):
                fail(f"{aid} first command looks like a write: {cmd}")
            if "readonly" not in cmd:
                fail(f"{aid} first command must pin a readonly context: {cmd}")
        esc = alert.get("escalation") or {}
        if not isinstance(esc.get("minute"), int) or esc["minute"] <= 0:
            fail(f"{aid} escalation.minute must be a positive integer")
        if not esc.get("to"):
            fail(f"{aid} escalation.to is empty")
        if not esc.get("page_back_when"):
            fail(f"{aid} page_back_when is empty")
        freeze = alert.get("freeze") or {}
        if freeze.get("on_clock_start") is not True:
            fail(f"{aid} must freeze writes when the escalation clock starts")
        if not freeze.get("unfreeze_evidence"):
            fail(f"{aid} unfreeze_evidence is empty")
    print("runbook ok")


if __name__ == "__main__":
    main(sys.argv[1])

Run it like this, and keep the unsigned template out of the branch that can page.

pip install pyyaml
python3 validate_runbook.py payments-api.runbook.yaml
# unsigned files must exit nonzero

I want that nonzero exit more than I want a prettier wiki, because pretty docs do not block a rollout. If CI is green while owners.sign is false, you built a linter that flatters the author. Should a pull request that cannot name a page-back condition be allowed to merge?

Limitations I will say out loud

This file does not detect a bad metric, and it does not replace a real observability stack during an incident. The write hints are regular expressions, so a creative destructive command can still sneak through if you work at it. The YAML cannot freeze a cluster by itself, so you still need an admission policy or a locked pipeline. If your pager string does not match alerts[].id, the allowlist will refuse, and that refusal is correct.

It also does not make generated text trustworthy, which is the whole lesson hiding under the trend around AI-authored work. A model can outline comments quickly on a free server, and that speed is useful only because the validator remains hostile. I will not claim latency numbers, token quotas, or hardware details I cannot verify from here. If you need those numbers, measure them yourself on your own box before you change a night rotation.

Who should skip this approach

Skip it if you are a solo hobby project with no pager, because freeze owners and escalation minutes are ceremony you do not need. Skip it if your compliance team requires a vendor-signed runbook format that this YAML cannot satisfy. Skip it if you expected the drafting box to execute production commands, because that is the opposite of the allowlist. And skip it if you want a model to unfreeze writes, because I will not give any model that signature.

I still want humans on the rotation, and I still want git to be ruder than Slack. Name the escalation minute, keep first commands read-only, and freeze writes when the clock starts. If you remember only one rule tonight, remember that unsigned drafts still do not page people. Would you actually merge this runbook file tonight, or are the escalation minutes still sitting blank?

Split Freeze From Unfreeze, and Map Each Alert to a Read-Only Command

Emery Huang — Sun, 20 Sep 2026 11:23:04 +0000

A paging channel without a freeze rule is just a group chat that happens to be loud. I will not treat an on-call runbook as live until every alert maps to a first command, an escalation tree names a human, and freeze is a different checklist from unfreeze. Can a model write fluent incident prose at two in the morning? Yes, and that is exactly why I keep it away from those fields.

Quiet services still page someone, and that someone still needs a command that will not write. If you skip the human sections, the next page becomes a brainstorm with extra confidence. Have you watched a channel fill with suggested kubectl lines while nobody names a freeze owner? I have, and I will not pretend a generated paragraph is a substitute for that owner.

Why pretty runbooks still fail the first page

Most half-finished runbooks fail in the same places, and none of those places are grammar. The alert does not name a service owner who can answer the phone. The first commands are write-heavy, so the first keystroke mutates production. The escalation path is a Slack handle that nobody pages after midnight. Freeze and unfreeze share one vague sentence, so nobody knows who may lift the gate.

AI-assisted drafting makes those gaps prettier, not smaller. Why would a model know your after-hours phone tree? Why would it know which replica is allowed to restart without a freeze? It will invent confident steps unless you refuse to let it own those fields. The useful work is still naming owners, commands, and gates in daylight, then leaving the model a narrow drafting job.

I still use a drafting box for symptom text, because writing that prose by hand is slow and easy to postpone. Disclosure: This article was prepared as part of MonkeyCode's product outreach. When I need a quiet place to draft non-authoritative sections, MonkeyCode's free model access and free server option are enough to generate candidate paragraphs that I then strip of any write action.

The human sections the pager actually needs

Treat the runbook as a contract the pager can parse, not as a blog post about the service. I keep these sections in a checked-in file, and I reject drafts that leave any of them as prose-only comments.

Alert catalog — each paging alert has an id, a service, a severity, and a human owner.
First-command map — each alert id points at read-only commands, with an explicit deny list for writes.
Escalation tree — primary, secondary, and manager, with a timeout that is a number, not a vibe.
Freeze checklist — who may freeze writes, what evidence they record, and what stays allowed.
Unfreeze checklist — a different owner path, a different evidence set, and a named blast check.

Notice freeze and unfreeze are not one section with a reversible verb. If they share a heading, people will unfreeze with the same confidence they used to freeze. Do you want that person to be a model that never saw your change window? I do not.

Artifact: a runbook file the pager can fail closed

This is a proposed schema, not a war story with fake metrics. Drop it in runbooks/payments-api.yaml and refuse to page off a document that cannot load.

# Proposed template — label every field human-owned or draft-only.
apiVersion: oncall.example/v1
kind: Runbook
metadata:
  service: payments-api
  owners: ["primary-oncall", "payments-tl"]
  timezone: America/Los_Angeles
spec:
  alerts:
    - id: ALRT-PAY-429
      name: checkout_latency_p99
      severity: page
      owner: "payments-primary"
      first_commands:
        - "kubectl --context=prod-ro -n payments get deploy payments-api -o wide"
        - "kubectl --context=prod-ro -n payments get pods -l app=payments-api"
        - "curl -sS https://status.internal/payments/health | jq ."
      deny_commands:
        - "kubectl delete"
        - "kubectl scale"
        - "kubectl apply"
        - "helm upgrade"
      escalate_after_minutes: 10
  escalation:
    primary: { role: "payments-primary", timeout_minutes: 10 }
    secondary: { role: "payments-secondary", timeout_minutes: 15 }
    manager: { role: "payments-tl", timeout_minutes: 20 }
    never: ["@here", "random model chat"]
  freeze:
    required_before_writes: true
    owner_role: "incident-commander"
    evidence:
      - "alert_id"
      - "first_command_output_hash"
      - "blast_radius_service_list"
    allowed_during_freeze: ["read", "page", "capture"]
  unfreeze:
    owner_role: "incident-commander"
    second_signer_role: "service-owner"
    evidence:
      - "alert_id_cleared_or_mitigated"
      - "write_plan_printed"
      - "rollback_command"
    forbidden_if_missing: ["second_signer", "rollback_command"]

Would I let a model fill deny_commands from memory of some other cluster? No, because that list is how you keep the first five minutes read-only. The model may draft the health-check narrative that sits under the YAML. It does not get to invent the owner role or the second signer.

A validator that fails before the rotation starts

I want the runbook to fail in CI, not in the paging channel. The script below is a proposed check you can run locally; it does not talk to production and it does not claim a benchmark.

#!/usr/bin/env python3
"""validate_runbook.py — proposed structural check, not an incident replay."""
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("pip install pyyaml\n")
    sys.exit(2)

WRITE_TOKENS = ("delete", "scale", "apply", "upgrade", "restart", "exec --")


def fail(msg: str) -> None:
    sys.stderr.write(msg + "\n")
    sys.exit(1)


def main(path: str) -> None:
    data = yaml.safe_load(Path(path).read_text())
    spec = data.get("spec") or {}
    alerts = spec.get("alerts") or []
    if not alerts:
        fail("no alerts: a runbook that cannot match a page is a wiki page")

    freeze = spec.get("freeze") or {}
    unfreeze = spec.get("unfreeze") or {}
    if not freeze or not unfreeze:
        fail("freeze and unfreeze must both exist as mappings")
    if freeze.get("owner_role") and freeze.get("owner_role") == unfreeze.get("owner_role"):
        # Same person may hold both hats, but the checklists cannot be identical.
        if freeze.get("evidence") == unfreeze.get("evidence"):
            fail("freeze evidence must not equal unfreeze evidence")
    if not unfreeze.get("second_signer_role"):
        fail("unfreeze needs a second signer role")

    esc = spec.get("escalation") or {}
    for key in ("primary", "secondary", "manager"):
        node = esc.get(key) or {}
        if not node.get("role") or not node.get("timeout_minutes"):
            fail(f"escalation.{key} needs role and timeout_minutes")

    for alert in alerts:
        aid = alert.get("id") or "<missing-id>"
        if not alert.get("owner"):
            fail(f"{aid}: alert has no human owner")
        cmds = alert.get("first_commands") or []
        if not cmds:
            fail(f"{aid}: no first commands")
        for cmd in cmds:
            low = cmd.lower()
            if any(tok in low for tok in WRITE_TOKENS):
                fail(f"{aid}: first command looks like a write: {cmd}")
        if not alert.get("deny_commands"):
            fail(f"{aid}: deny_commands missing")

    print(f"ok: {path} ({len(alerts)} alerts)")


if __name__ == "__main__":
    if len(sys.argv) != 2:
        fail("usage: python3 validate_runbook.py runbooks/payments-api.yaml")
    main(sys.argv[1])

Run it like this, on a laptop, before you accept a week of pages:

python3 validate_runbook.py runbooks/payments-api.yaml
# expected: ok: runbooks/payments-api.yaml (1 alerts)

If the validator prints ok while unfreeze.evidence still equals freeze.evidence, I missed a check and I want you to fail the file by hand. Should a green script be enough to unfreeze production? No. It is only enough to prove the document is not empty.

First commands stay read-only until freeze is named

I keep a tiny wrapper so the tired person at the keyboard cannot “just scale it” from muscle memory. This is a proposed shell gate, not a cluster agent.

#!/usr/bin/env bash
# firstcmd.sh — proposed read-only wrapper. Do not point this at a write kubecontext.
set -euo pipefail
ALERT_ID="${1:?alert id}"
RUNBOOK="${2:-runbooks/payments-api.yaml}"
CONTEXT="${KUBE_RO_CONTEXT:?set KUBE_RO_CONTEXT to a read-only context}"

if [[ "$CONTEXT" == *prod-w* || "$CONTEXT" == *writable* ]]; then
  echo "refusing: context looks writable: $CONTEXT" >&2
  exit 1
fi

echo "alert=$ALERT_ID runbook=$RUNBOOK context=$CONTEXT"
echo "freeze_named=${FREEZE_NAMED:-no}"
if [[ "${FREEZE_NAMED:-no}" != "yes" ]]; then
  echo "writes remain blocked until FREEZE_NAMED=yes and the freeze checklist is signed"
fi

What belongs in first_commands? Gets, describes, logs with a tail limit, and health URLs that cannot mutate. What does not belong? Anything that restarts, scales, applies, deletes, or opens an interactive shell on a prod pod. If the model drafts a “quick restart” as step one, that is a defect in the draft, not a shortcut.

Escalation is a tree with minutes, not a mention

I want numbers on the tree because “ping secondary if needed” is how pages rot in a thread. Primary has ten minutes. Secondary has fifteen. Manager has twenty. After that, you are not brainstorming; you are late.

Do not escalate to @here.
Do not escalate to a model chat that cannot take the phone.
Do not list a person who is already the freeze owner unless a second signer still exists for unfreeze.
Do record the time you moved to the next role, even if the next role is you with a different hat.

If your team is two people, the tree can still exist. It just becomes honest about how thin the bench is. Is a thin bench a reason to skip the tree? It is a reason to write the tree larger than the bench, then staff it later.

Freeze and unfreeze as two checklists

Use a decision table during the incident, on paper or in the channel topic. The table is the artifact that keeps freeze from becoming a mood.

Question	Freeze	Unfreeze
Who signs?	Incident commander	Commander plus service owner
What evidence?	Alert id, first-command output, blast list	Alert mitigated, printed write plan, rollback
Writes allowed?	No, except documented break-glass	Only the printed plan
Model output allowed as proof?	No	No
Can the same human hold both hats?	Yes, if the evidence sets differ	Only with a second signer

I will freeze as soon as a write is tempting and the blast list is still unnamed. I will not unfreeze because a generated summary says the error rate “looks better.” Looks better than what baseline, captured by which command, against which alert id? If you cannot answer that, the freeze stays.

Where a drafting model is allowed, and where it is not

Let the model propose symptom text, dashboard links you already host, and questions to ask the primary. Do not let it propose owners, phone numbers, deny lists, or unfreeze signers. Those fields are boring on purpose, because boring fields are the ones people skip when a page is loud.

If you paste a generated runbook straight into the wiki, you have a document that reads finished and behaves empty. That is the failure mode I am trying to make expensive. The validator is cheap. The second signer on unfreeze is cheaper than a write that nobody can roll back.

Limitations, and who should not use this

This approach assumes you already have paging, a read-only kube context or equivalent, and a place to store YAML that humans review. It does not measure latency, does not replay traffic, and does not prove the first command is the right command for a novel failure. The schema will happily accept a wrong-but-read-only curl if you typed it.

Do not use this if you are a solo hobby project with no pager and no production writes worth freezing. Do not use it as a reason to skip a real staging reproduction when you have time. Do not use a drafting server as the system of record for phone trees or credentials. Do not treat free model access as an on-call teammate; it cannot take the escalation slot, and it cannot sign unfreeze.

If your incident process already requires two-person review for every production write, you may only need the alert-to-command map and the validator. That is fine. The point is not to collect headings. The point is to make the first five minutes of a page boring enough that nobody invents a write.

Fill Four On-Call Lists Before You Accept the Rotation

Emery Huang — Fri, 18 Sep 2026 07:48:12 +0000

On-call is not a conversation with a model while the pager is still firing at you. A rotation is ready only when four lists exist: the alert catalog, the first commands, the escalation packet, and the freeze rule. I will not accept a week of pages that depend on a chat transcript, no matter how fluent that transcript looks. Would you really open a blank prompt at three in the morning and call that a procedure?

Narrative runbooks feel complete in daylight, then collapse when two alerts land together. People scroll, argue about meaning, and invent write commands that nobody reviewed against a named surface. I want files I can grep, scripts that only print state, a clock that forces escalation, and a freeze line that does not depend on mood. Does a five-page story help you when the ID on the page is not even in the catalog?

The useful question this week is not whether assistants can draft English. The question is whether you will call that draft an operating procedure before it survives a checker. I let a model fill comments. I reject the file until the four lists validate. That is the method, and chat is none of the lists.

The four lists, named on purpose

I keep the on-call file as four lists because each list answers a different failure. The catalog answers "what is this page." First commands answer "what do we look at before we speak." Escalation answers "who gets the evidence, and when." Freeze answers "which writes are illegal until a human writes a sentence." If any list is missing, you are improvising, and improvisation is not a runbook.

Here is the order I actually enforce when a rotation starts:

Every page ID in the last month has a catalog row, including a "not this" note.
Every catalog row points at a read-only first-command script that exists on disk.
Escalation names roles, minute bounds, and the three fields in the packet.
Unknown IDs freeze writes immediately; unfreeze is a filled sentence, not a vibe.

Would a generated paragraph satisfy those four gates? Only the parts that survive the YAML and the checker below. Everything else stays a comment in a draft branch.

List 1: the alert catalog is a table, not a blog

Every page needs a stable ID, a meaning, a severity, and a "not this" note. If the ID is missing from the catalog, the incident is already in freeze, and nobody invents a new class on the call. New classes are born in daylight, with a script on disk and a minute bound that a human typed. Why would you let a model mint an ID while customers are already waiting?

Catalog shape I keep in git

# runbook/alerts.yaml — labeled template, not a live service file
alerts:
  - id: "checkout.p95.latency"
    severity: sev2
    means: "p95 checkout latency exceeded the burn threshold for 5 minutes"
    not_this:
      - "batch jobs on the reporting warehouse"
      - "synthetic probes failing from one region"
    first_command_set: "latency-read"
    escalate_after_minutes: 15
    write_allowed: false
  - id: "payments.auth.5xx"
    severity: sev1
    means: "payment auth 5xx rate crossed the error budget page"
    not_this:
      - "client timeouts from a single ISP"
    first_command_set: "payments-read"
    escalate_after_minutes: 8
    write_allowed: false

Notice write_allowed starts false on every row. A catalog row that defaults to writes is a loaded gun with the safety off. Unknown IDs do not get a row invented during the page; they get freeze, then escalation, then a daylight edit. I want the on-call person to grep, not to negotiate meaning in a thread.

List 2: first commands print state and then stop

First commands print state. They do not restart, scale, mute, deploy, or patch. I want the output pasted into the incident doc before anyone discusses a mutation, because the paste is the only evidence the next human can trust. If the command is not in the set named by the alert, it does not run, even when someone swears it is "just a quick look." Would you skip the paste step because the output looked obvious?

Read-only command sets

# first-commands/latency-read.sh
# labeled example: swap in your own read endpoints after review
set -euo pipefail
echo "UTC $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "alert_id=${ALERT_ID:-unset}"
curl -fsS "$STATUS_URL/checkout/p95" | head -c 4096
echo
kubectl get deploy checkout -o wide
kubectl get hpa checkout -o yaml | sed -n '1,80p'

# first-commands/payments-read.sh
set -euo pipefail
echo "UTC $(date -u +%Y-%m-%dT%H:%M:%SZ)"
curl -fsS "$STATUS_URL/payments/auth/5xx" | head -c 4096
echo
kubectl logs deploy/payments-auth --tail=80 --since=10m \
  | grep -E '5[0-9]{2}|timeout' | tail -n 40

I keep these scripts tiny on purpose. A first-command file that contains apply, delete, restart, or scale is not a first command; it is a change that snuck in through naming. The checker later treats those strings as failures, because I have watched "just restart it" show up in a reconnaissance folder.

List 3: escalation is a clock plus a packet

Escalation is not "ping someone if you feel stuck." It is a minute bound, a named role, and a packet of evidence that another human can read without joining your head. The packet is the alert ID, the first-command output path, and the freeze state. If any of those three is missing, you are not escalating; you are venting into a channel. What happens when the named person is asleep and the runbook only lists a nickname?

Escalation decision table

Clock	Condition	Who you page	Packet you must send
T+0	Page fires	primary on-call	alert ID, catalog hit or miss
Catalog minute bound	First commands cannot classify	secondary and service owner	output path, freeze=on
Any time	Alert ID missing from catalog	incident commander	freeze=on, no writes
After classify	A write is proposed	owner of the write surface	command text and revert line

I keep roles as functions, not heroic names. Put the rotation source next to the role so the packet can move when the human changes. A Slack handle with no rotation pointer is a hope, and hope is not an escalation path.

# runbook/escalation.yaml — labeled template
roles:
  primary: "pagerduty://checkout-primary"
  secondary: "pagerduty://checkout-secondary"
  service_owner: "pagerduty://checkout-owner"
  incident_commander: "pagerduty://ic-weekday"
freeze_on_unknown_alert: true
require_packet:
  - alert_id
  - first_command_output_uri
  - freeze_state

List 4: freeze unknown pages; unfreeze with a sentence

Here is the freeze rule I will defend in a postmortem. If the alert ID is absent from the catalog, freeze writes immediately and escalate. If the first-command set cannot classify the symptom inside the catalog minute bound, freeze writes and escalate. Unfreeze requires the catalog ID, a classified symptom, a named write owner, and an explicit unfreeze block in the incident doc. Is that slower than a heroic restart? Yes, and that delay is the control.

Unfreeze is a sentence, not a feeling in the room. I write it like this, and I refuse anything shorter:

UNFREEZE checkout writes
alert_id=checkout.p95.latency
classified_as=db_pool_saturation
write_owner=checkout-owner
command=increase checkout pool from 20 to 40
revert=set pool back to 20
expires=20m

If you cannot fill those fields, you stay frozen. The revert line is not decoration; it is the only way the next on-call person knows how to undo your write without asking the chat. I have no interest in an unfreeze that cannot be read after the call ends.

Artifact: fail the runbook before the rotation starts

I do not argue about completeness in the war room. A checker fails the tree on a laptop, or in CI, before anyone accepts the pager. The script below is a proposal you can run against the YAML shapes above. It does not prove production is healthy. It proves the four lists are present, writes are not smuggled into first commands, and unknown alerts are set to freeze.

#!/usr/bin/env python3
"""runbook_check.py — unlabeled fields fail the runbook."""
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("install pyyaml before running this checker\n")
    sys.exit(2)

REQUIRED_ALERT = {
    "id",
    "severity",
    "means",
    "not_this",
    "first_command_set",
    "escalate_after_minutes",
    "write_allowed",
}
REQUIRED_ROLES = {"primary", "secondary", "service_owner", "incident_commander"}
REQUIRED_PACKET = {"alert_id", "first_command_output_uri", "freeze_state"}
WRITE_MARKERS = ("kubectl delete", "kubectl apply", "restart", "scale ")


def fail(msg: str) -> None:
    print(f"FAIL: {msg}")
    raise SystemExit(1)


def main(root: Path) -> None:
    alerts = yaml.safe_load((root / "runbook/alerts.yaml").read_text())
    esc = yaml.safe_load((root / "runbook/escalation.yaml").read_text())
    rows = alerts.get("alerts") or fail("alerts.yaml missing alerts list")
    seen: set[str] = set()
    for row in rows:
        missing = REQUIRED_ALERT - set(row)
        if missing:
            fail(f"alert {row.get('id')!r} missing {sorted(missing)}")
        if row["id"] in seen:
            fail(f"duplicate alert id {row['id']}")
        seen.add(row["id"])
        if row.get("write_allowed") is True:
            fail(f"{row['id']} must not default write_allowed to true")
        cmd = root / "first-commands" / f"{row['first_command_set']}.sh"
        if not cmd.is_file():
            fail(f"missing first-command script {cmd}")
        text = cmd.read_text()
        for marker in WRITE_MARKERS:
            if marker in text:
                fail(f"{cmd} looks like a write; first commands must be read-only")
        minutes = row["escalate_after_minutes"]
        if not isinstance(minutes, int) or minutes <= 0:
            fail(f"{row['id']} escalate_after_minutes must be a positive int")
    roles = esc.get("roles") or fail("escalation.yaml missing roles")
    missing_roles = REQUIRED_ROLES - set(roles)
    if missing_roles:
        fail(f"missing roles {sorted(missing_roles)}")
    if esc.get("freeze_on_unknown_alert") is not True:
        fail("freeze_on_unknown_alert must be true")
    packet = set(esc.get("require_packet") or [])
    if packet != REQUIRED_PACKET:
        fail(f"require_packet must be {sorted(REQUIRED_PACKET)}")
    print(f"OK: {len(rows)} alerts, freeze-on-unknown enabled")


if __name__ == "__main__":
    main(Path(sys.argv[1]) if len(sys.argv) > 1 else Path("."))

Run it like this on a toy tree before you point it at the real rotation files:

python3 runbook_check.py .
# expected on a complete tree:
# OK: N alerts, freeze-on-unknown enabled

If the checker fails, the rotation does not start. That sounds harsh until you remember the alternative: a page whose first command does not exist, or a freeze rule that lives only in someone's memory. I would rather fail CI on Tuesday than invent YAML while the pager is already open.

How a page should actually move

When the page fires, I want a boring sequence, not a clever one. The primary copies the alert ID, greps the catalog, and runs only the named first-command script. Output goes into the incident doc before any theory gets typed. Classification either happens inside the minute bound or it does not; there is no third state called "we are pretty sure."

If the ID is missing, freeze is already on, and the packet goes to the incident commander with freeze=on and no writes. If classification fails the clock, freeze stays on and secondary plus owner get the output path. Only after those gates does anyone draft a write, and that write still needs an unfreeze block with a revert line. Would you skip a gate because the room feels aligned? The gates exist for the room that feels aligned.

Where a draft assistant fits, and where it does not

I will let a model propose means and not_this text from an exported list of alert names. I will not let it invent IDs, minute bounds, role URIs, or write commands. The draft is a comment. The YAML is the contract. That split is how you use an assistant without pretending the assistant took the page.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Comment drafts for those catalog sentences can sit in MonkeyCode, which has free model access and a free server option, and only fields that survive review get copied into alerts.yaml. The checker does not care where the English came from. The free server is a draft bench, not an incident cluster, and generated kubectl that never sat in first-commands/ stays unread.

Would I point any assistant at a production kubeconfig to "save time"? No. Time saved on typing is worthless if the first write has no revert line. Keep the model on comments. Keep the pager on lists.

Limitations you should not hand-wave

This workflow assumes you already have paging, a place to paste output, and someone who can own a write. The checker is syntactic; it cannot prove a curl target is actually read-only, and it cannot prove a role URI reaches a human who is awake. Minute bounds are guesses until you rehearse them with a game-day page. YAML that passes CI can still describe the wrong service, or a "not this" note that was true last quarter and is false tonight.

Assistants hallucinate confident not_this notes. If you paste those notes without a human who has taken that page, you will freeze the wrong subsystem or unfreeze too early. I treat every generated sentence as untrusted until a person who has carried the pager edits it. The four lists reduce improvisation. They do not replace judgment, and they do not replace a rehearsal.

Who should not use this shape

Do not use this shape if you have no pager, no secondary, and no incident doc. Do not use it to justify running model output against production because the sentences sounded operational. Do not use a shared free server as the system of record for secrets, kubeconfigs, or customer data. Tiny internal tools with one operator may prefer a single shell script over four YAML files, and that is fine.

If your culture rewards the fastest unmute, this freeze rule will feel like friction. That friction is the point of the runbook. I would rather explain a delayed write than explain an untracked restart that nobody can revert. The four lists are the runbook. Everything else is conversation, and conversation does not unfreeze writes.

I Will Not Unfreeze Prod Until the Runbook Hash Matches the Page

Emery Huang — Thu, 17 Sep 2026 05:50:16 +0000

I will not let an assistant invent the incident path while the pager is still screaming. A usable on-call runbook names the alert class, the first command, the escalation owner, and the freeze hash before anyone types. Chat logs are not runbooks, and generated shells are not change tickets, no matter how confident they sound. If those four fields are missing, I keep writes frozen and I keep the assistant on a scratch box.

Why am I this stubborn about a document that looks like YAML? Because the last seven days of AI-coding talk keep confusing fluency with ownership, and on-call work punishes that mix. A model can draft a plausible restart in twenty seconds, then miss the replica that actually holds the lock. Would you rather argue with a chat transcript at 03:00, or execute a hashed runbook you already signed?

The contract I actually page against

I treat every page as a four-field contract, not as a brainstorming session with extra urgency. The contract is boring on purpose, and boring is what I want when the graph is red. If a field cannot be filled from the alert payload and the repo, I do not improvise a fifth field in Slack.

Here is the schema I keep in runbooks/<service>.yml. Copy it, then refuse to run writes until freeze.hash matches the file on disk.

# runbooks/payments-api.yml
apiVersion: oncall.example.com/v1
service: payments-api
owner_oncall: payments-primary
escalation:
  t_plus_15m: payments-secondary
  t_plus_30m: payments-manager
  t_plus_45m: incident-commander
alert_classes:
  - id: PAYMENTS_P99_LATENCY
    severity: page
    observe_commands:
      - id: oc1
        argv: ["kubectl", "-n", "payments", "get", "deploy", "payments-api", "-o", "wide"]
      - id: oc2
        argv: ["kubectl", "-n", "payments", "top", "pod", "-l", "app=payments-api"]
    write_commands: []   # stays empty until unfreeze
freeze:
  state: frozen
  hash_of: runbooks/payments-api.yml
  require:
    - named_blast_radius
    - matching_git_sha
    - human_signature

Notice what is missing on purpose: there is no prompt, no vibe, and no "try whatever the model suggested." Observe commands are argv arrays, not English. Write commands start empty. The freeze block is data, not a pep talk.

Field 1: alert class, not a paragraph

I bind the pager to a stable alert_class id, because humans rename dashboards and models paraphrase titles. PAYMENTS_P99_LATENCY is allowed to wake me; "the site feels slow" is not. If the alert cannot map onto one id in the YAML, I treat it as observe-only noise until a human classifies it.

Page-worthy classes get severity: page and a short observe list.
Ticket-worthy classes get severity: ticket and never unfreeze writes.
Unknown classes inherit freeze and a required escalation note.

Can your current alert rule print that id into the notification body? If it cannot, the runbook is already lying before you open a terminal.

# example Alertmanager annotation, not a prompt
description: class=PAYMENTS_P99_LATENCY ns=payments deploy=payments-api

Field 2: first commands stay read-only

My first commands are inventory, not remediation. I want the replica count, the ready condition, the recent events, and the error budget burn, in that order. Anything that mutates pods, feature flags, or DNS waits behind the freeze hash. Have you ever watched a generated one-liner delete the healthy deployment because the label selector was almost right?

I keep a tiny allowlist checker next to the YAML so the laptop refuses clever extra flags.

# tools/check_first_commands.py
import sys, yaml

ALLOWED = {
    "kubectl": {"get", "describe", "logs", "top", "api-resources"},
    "dig": {"+short"},
    "curl": {"-sS", "-o", "/dev/null", "-w"},
}

def main(path):
    doc = yaml.safe_load(open(path))
    errors = []
    for cls in doc["alert_classes"]:
        for cmd in cls["observe_commands"]:
            argv = cmd["argv"]
            bin_ = argv[0]
            if bin_ not in ALLOWED:
                errors.append(f"{cls['id']}: binary {bin_} not in observe allowlist")
                continue
            if any(tok in argv for tok in ("delete", "apply", "patch", "scale", "drain")):
                errors.append(f"{cls['id']}: mutating token in observe argv {argv}")
        if cls.get("write_commands"):
            errors.append(f"{cls['id']}: write_commands must be empty while frozen")
    if doc.get("freeze", {}).get("state") != "frozen":
        errors.append("freeze.state must start as frozen")
    if errors:
        print("\n".join(errors)); sys.exit(1)
    print(f"ok: {path} observe-only and frozen")

if __name__ == "__main__":
    main(sys.argv[1])

Run it before the page, not during it.

python3 tools/check_first_commands.py runbooks/payments-api.yml
sha256sum runbooks/payments-api.yml | tee runbooks/payments-api.yml.sha256

Field 3: escalation is a clock, not a group chat

I write escalation as timestamps against the first page, because "ask in Slack" is how incidents grow extra owners and zero decisions. Fifteen minutes of observe with no named blast radius promotes to secondary. Thirty minutes promotes to the manager. Forty-five minutes asks for an incident commander who is not also typing kubectl. Who is allowed to unfreeze if the primary is the person who wrote the bad deploy?

T+0 to T+15: primary runs observe commands only, pastes command ids into the incident doc.
T+15: secondary joins, repeats the same observe ids, and challenges any extra argv.
T+30: manager confirms customer impact and names the blast radius in one sentence.
T+45: commander owns communication; primary still cannot unfreeze without the hash check.

I do not escalate because I feel nervous. I escalate because the clock in the runbook elapsed and a field is still blank.

Field 4: freeze hash, then a signed unfreeze

This is the rule that keeps assistants useful instead of dangerous. The freeze hash is sha256 of the runbook file at the git sha I am paging from. Unfreeze is a separate artifact, not a vibes-based "looks good." If the assistant rewrites the YAML during the call, the hash breaks, and writes stay frozen. Is that annoying? Yes. Is it worse than a silent kubectl apply from a regenerated plan? Not even close.

# tools/unfreeze.sh — still a proposal until a human runs it on a signed laptop
set -euo pipefail
RB="${1:?runbook yml}"
INCIDENT="${2:?incident id}"
BLAST="${3:?blast radius sentence}"
SIG="${4:?path to signature file}"

expected=$(cut -d' ' -f1 "${RB}.sha256")
actual=$(sha256sum "$RB" | cut -d' ' -f1)
test "$expected" = "$actual" || { echo "hash mismatch: freeze holds"; exit 2; }
test -s "$SIG" || { echo "no human signature"; exit 3; }
test -n "$BLAST" || { echo "blast radius unnamed"; exit 4; }

printf 'unfrozen\nincident=%s\nblast=%s\n' "$INCIDENT" "$BLAST" > "${RB}.unfreeze"
echo "writes allowed only for commands listed after this file exists"

I still require write commands to be appended as argv arrays after unfreeze, never as free text. The assistant may propose a patch on a scratch clone. It does not get to flip freeze.state.

Where a free coding assistant actually helps

I will use a coding assistant to draft the YAML, generate the allowlist tests, and argue with my own escalation clock, but only on a throwaway box. MonkeyCode is relevant here because free model access and a free server option give me that scratch loop without borrowing a production jumphost. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I paste the schema, the checker, and a redacted alert sample, then I ask for missing observe commands that still fit the allowlist. I do not paste kubeconfigs, customer payloads, or unfreeze signatures. If the model invents a rollout restart, the checker fails, and that failure is the lesson. The free server is for running check_first_commands.py, not for talking to prod.

# labeled as a proposal I would run on a scratch server, not on prod
python3 tools/check_first_commands.py runbooks/payments-api.yml
python3 -c "import yaml,sys; yaml.safe_load(open(sys.argv[1]))" runbooks/payments-api.yml

Would I let the same session open a tunnel to the cluster? No. The runbook is the contract; the assistant is a typist with a linter.

A dry-run I actually practice

I rehearse with a fake page so the first real page is not also the first parse of the YAML. The drill is short, scripted, and mean about extra commands.

export INCIDENT=inc-drill-2026-09-17
export ALERT_CLASS=PAYMENTS_P99_LATENCY
python3 tools/check_first_commands.py runbooks/payments-api.yml
# observe only — replace with your read-only kube context
kubectl --context scratch -n payments get deploy payments-api -o wide
kubectl --context scratch -n payments top pod -l app=payments-api
# stop here unless tools/unfreeze.sh succeeded on this same hash

If the drill needs a command that is not in observe_commands, I update the YAML in git and rehash. I do not "just this once" extend the path in the terminal. That exception is how freeze gates die.

Limitations, and who should skip this

This workflow assumes you can map alerts to ids, keep argv allowlists, and block writes in the default path. It will feel heavy if you are a solo hobby project with no pager, or if your platform cannot distinguish read kubectl from write kubectl. It also will not save you if the runbook hash is computed after the assistant edits the file. I am not claiming latency numbers, model rankings, or token budgets here, because those claims go stale and they are not the point.

Do not use this approach to launder a generated production change behind a decorative freeze file. Do not store signatures, kubeconfigs, or customer data on a shared demo server. Do not skip escalation because the model sounded calm. If you cannot name the blast radius in one sentence, the hash check should keep failing.

The core conclusion does not change when the models get more fluent. Encode the page as alert class, first command, escalation clock, and freeze hash, then make unfreeze a signed break of that hash. Everything else is optional commentary. If you want a scratch box to lint the YAML and the allowlist, MonkeyCode's free server option is enough to practice the checks; it is not an on-call seat.

Copilot Patches Stay Read-Only Until You Sign the Unfreeze

Emery Huang — Wed, 16 Sep 2026 04:04:15 +0000

An AI copilot can draft a remediation in seconds, but it cannot own production blast radius. I still freeze writes until the alert class, first-read commands, and an escalation owner are named. If those three are missing, the patch stays in a scratch shell, not on the cluster. Would you really unfreeze kube-system just because a chat window sounded confident at 03:12?

The wrong debate for a page

Feeds keep asking whether models already outcode most working engineers on ordinary tickets. That question is interesting at lunch, and it is honestly dangerous on a live pager. On-call is not a coding contest; it is a control problem with a clock and a blast radius. A fluent patch that restarts the wrong Deployment is still an incident amplifier, not a win.

I treat every model suggestion as an untrusted diff sitting against a frozen write path. The runbook below is a proposal I keep in the repo, not a story about a specific outage. If your team already pastes chat output into kubectl, this is the gate I wish sat in front of that habit.

What I page on, and what I refuse to page on

I only auto-page when the alert already carries a stable fingerprint, a service owner, and a read-only first command. Everything else can wait for a ticket, because a vague prompt is not an incident. Ask yourself: if the alert cannot name the workload, how can a model name a safe write?

Payload contract

Minimum fields I want on the page:

alertname plus a fingerprint hash, not a prose summary
namespace, workload_kind, and workload_name when Kubernetes is involved
owner as a team alias that actually answers
first_command that is strictly read-only
write_allowed defaulting to false until unfreeze

# proposal: pager payload contract (not executed against a live cluster)
alertname: KubeDeploymentReplicasUnavailable
fingerprint: "a3f1c9e2"
namespace: payments
workload_kind: Deployment
workload_name: checkout-api
owner: payments-oncall
severity: page
write_allowed: false
first_command: >
  kubectl -n payments get deploy checkout-api -o jsonpath='{.status.conditions}'

If any of those keys are empty, I do not run writes, and I do not ask a model for a restart. I escalate with the incomplete payload, because guessing owners at 3 a.m. is how you page the wrong people. Can a copilot invent an owner alias that paging actually honors? It cannot, and I will not pretend otherwise.

First commands are reads, and they are boring on purpose

My first five minutes are observe-only, even when the copilot is already waving a Helm rollback. I want evidence that matches the fingerprint, not a soothing narrative that matches my anxiety. The commands below are the ones I type before anyone on the call talks about unfreeze.

The boring bundle

# proposal: observe-only bundle, labeled unexecuted
export NS=payments APP=checkout-api KIND=deploy NAME=checkout-api
date -u
echo "fingerprint=${FINGERPRINT:-missing}"

kubectl -n "$NS" get deploy,sts,po,ep -l app="$APP" -o wide
kubectl -n "$NS" describe "$KIND" "$NAME" | sed -n '1,80p'
kubectl -n "$NS" logs "deploy/$NAME" --tail=100 --timestamps | tail -n 40

# still read-only: compare live object to last applied
kubectl -n "$NS" get "$KIND" "$NAME" -o yaml > "/tmp/${NAME}.live.yaml"

Notice there is no rollout restart, no scale, and no force delete in that first bundle. If the model prints those, I copy them into a scratch file named candidate.sh, and I leave production frozen. Why would a first command need a write if we still have not proved the alert is real?

Escalation is a lane, not a vibe

I escalate when the fingerprint is real and the owner is not me, or when the read-only evidence disagrees with the model's story. Restarting just to see is more expensive than a two-minute handoff with a complete note. The three questions I send in the escalate note stay small on purpose, and they travel well.

The note I actually send

Which fingerprint and workload did we actually observe?
Which read-only commands already ran, with paste of the output hashes?
What write is proposed, and what blast radius did we name?

# proposal: escalate note
fingerprint: a3f1c9e2
observed: Deployment/checkout-api ns=payments replicas 1/3
reads_run: get,describe,logs (see incident/2026-09-16/reads.log)
proposed_write: kubectl -n payments rollout undo deploy/checkout-api
blast_radius: checkout-api only; payments-worker not in selector
owner_needed: payments-oncall
unfreeze: blocked

If the next engineer cannot answer those, we stay frozen through the next scheduled check-in. A model that cannot fill this note should not get a shell on any cluster. Does that slow you down on a scary page, and should it, when a fast write can still take the site down?

The freeze and unfreeze rule for copilot patches

Here is the rule I want on a sticky note above the keyboard during every page. Production writes stay frozen until four signatures exist together in one unfreeze file on disk. Missing any signature means the copilot patch remains a candidate, even if the diff looks brilliant.

Four signatures

Unfreeze requires all of the following:

Alert fingerprint matches the live object you just read
Owner alias is a human who accepted the page
Blast radius is a named workload list, not "the cluster"
Candidate commands were dry-run in a scratch environment, not on prod

# proposal: unfreeze gate
export INCIDENT_ID=20260916-checkout
export UNFREEZE_FILE="./incident/${INCIDENT_ID}/unfreeze.json"

python3 - <<'PY'
import json, os, sys, pathlib
p = pathlib.Path(os.environ["UNFREEZE_FILE"])
data = json.loads(p.read_text())
required = ["fingerprint", "owner", "blast_radius", "dry_run_ok"]
missing = [k for k in required if not data.get(k)]
if missing or data.get("write_allowed") is not True:
    print("FROZEN: missing", missing or ["write_allowed"])
    sys.exit(2)
print("UNFREEZE ok for", data["blast_radius"])
PY

Example unfreeze.json I will not apply until the checker exits zero:

{
  "fingerprint": "a3f1c9e2",
  "owner": "payments-oncall",
  "blast_radius": ["deploy/checkout-api"],
  "dry_run_ok": true,
  "write_allowed": false
}

I do not source the candidate script until that checker exits zero on the unfreeze file. If you skip the file, you are not using a runbook; you are gambling with autocomplete. Would I make an exception for a Sev-1? Only with a named owner and a named blast radius.

Dry-run the model where it cannot page you twice

I want the copilot to argue with a replica of the failing object, not with production kube-apiserver. A scratch box, a kind cluster, or a disposable VM is enough for most Deployment pages. The goal is simple: does the proposed command even parse, and does it target the named blast radius?

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode, an open source project, only as a scratch place with free model access and a free server option. That keeps the dry run off production credentials, which matters more than chat convenience during a page. I am not claiming a particular model name, a token ceiling, or a hardware spec, because those change and I will not invent them. If you need a disposable box for that dry run, the free server option is enough to keep the experiment off prod.

Static allowlist

# proposal: keep the model off prod kubeconfig
export KUBECONFIG="$HOME/scratch/kind-incident.kubeconfig"
export INCIDENT_ID="20260916-checkout"

# reject any line that is not in a tiny allowlist
grep -E '^(kubectl |helm |echo |#)' candidate.sh >/dev/null

python3 ./tools/assert_blast_radius.py \
  --script candidate.sh \
  --allow namespace=payments \
  --allow name=checkout-api \
  --forbid kube-system \
  --forbid default

# only after static checks: server-side dry-run on the scratch cluster
kubectl -n payments apply --dry-run=server --validate=true -f candidate.yaml
kubectl -n payments diff -f candidate.yaml

# tools/assert_blast_radius.py — proposal checker, unexecuted against prod
import argparse, sys

def tokens(line: str):
    return line.split()

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--script", required=True)
    p.add_argument("--allow", action="append", default=[])
    p.add_argument("--forbid", action="append", default=[])
    args = p.parse_args()
    allow = dict(a.split("=", 1) for a in args.allow)
    text = open(args.script, encoding="utf-8").read().splitlines()
    for i, line in enumerate(text, 1):
        s = line.strip()
        if not s or s.startswith("#"):
            continue
        parts = tokens(s)
        if s.startswith("kubectl") and "-n" in parts:
            ns = parts[parts.index("-n") + 1]
            if ns in args.forbid or ns != allow.get("namespace"):
                print(f"line {i}: namespace {ns} not in blast radius")
                sys.exit(1)
        if any(bad in s for bad in ("--force", "delete ns", "drain ")):
            print(f"line {i}: forbidden verb")
            sys.exit(1)
    print("dry-run static checks passed")

if __name__ == "__main__":
    main()

If that checker fails, I do not unfreeze, and I do not negotiate with the model. I either rewrite the candidate by hand or I escalate with the failing line number. A free scratch server is useful here because I can burn the VM after the page, secrets and all.

Decision table I keep next to the runbook

The matrix below is what I actually glance at when the copilot starts sounding sure. It is deliberately blunt, because a clever exception is how frozen writes become thawed writes. Please read it once before you open a chat window on an active production page.

Signal	First action	Copilot allowed?	Unfreeze?
Fingerprint missing	Reject page / convert to ticket	No	No
Fingerprint matches, reads clean	Capture logs, wait	Comments only	No
Reads confirm bad rollout	Draft undo in scratch	Yes, files only	After signatures
Owner unreachable	Escalate, stay frozen	No writes	No
Candidate touches extra NS	Delete candidate.sh	No	No
Checker exit 0 + owner ack	Run named write once	No further prompts	Yes, time-boxed

The table is the article, if I am honest with the rest of the on-call rotation. Models are optional during a page, and they stay optional after the unfreeze file exists. The freeze file is not optional, and I will argue that in the incident review.

Limitations, and who should not copy this

When I throw the runbook away

This workflow assumes you can freeze writes without making the outage worse, which is not true for every class of alert. Data corruption, certificate expiry in the next few minutes, and disk-full nodes sometimes need a human-owned write with no model in the loop. I also assume you will not paste secrets, kubeconfigs, or customer payloads into any chat, free or otherwise.

Do not use this approach if you lack RBAC that can actually deny your own user. Skip it as well if your so-called scratch cluster still shares any credentials with production. Do not use it as an excuse to skip the owner, either, when the page is loud. A free server does not become a production control plane just because the model answered quickly.

I am not publishing latency numbers, token quotas, or win rates, because I did not measure them for this piece. The only artifact I am defending here is the unfreeze gate, not a vendor benchmark. If your incident commander wants a different freeze policy, follow that policy and keep the checker as a pre-commit hook on runbook PRs.

When the page is quiet again, I file the unfreeze file next to the read logs. That packet is what I want in the review, not a screenshot of a confident chat. Keep the model in the scratch lane, and keep production behind a signature you can explain.