Keep Production Frozen Until the Runbook Names Blast Radius

I will not unfreeze production until the runbook names blast radius and a hard command budget. A page without those two fields is just a loud ticket, and I treat it that way on every rotation. Can you point at the services that freeze with this alert, or are we still guessing in Slack? Guessing is how a careful read-only check becomes a write that nobody on the rotation can roll back.

Why this runbook shape exists

Most on-call docs I inherit list symptoms, then dump a wall of kubectl and curl without saying what they can break. They never say how many commands I may run before I must escalate to a human who owns the blast radius. Is that a runbook, or is it a wiki page wearing a pager costume at three in the morning? I want a contract that the alert, the first command, and the freeze rule can all satisfy without improvisation.

The industry keeps handing us faster assistants and louder suggested patches, and tired engineers will paste those patches into prod. I still want drafting help, but I want it on a box that cannot touch production traffic or secrets. Does your current runbook even say where generated output is allowed to live during an incident? Mine says the draft lives offline, and the freeze stays up until a human signs the unfreeze.

The four fields I require on every paging alert

Every paging alert in the runbook gets four required fields before I will accept the next rotation. If a field is missing, I refuse the page and I keep the named services frozen. Why would I accept a rotation that cannot tell me what breaks when checkout-api pages at 3 a.m.? I would not, and I have started saying that out loud during the handoff, not after the page.

Here is what I require, written as a list so reviewers cannot pretend they missed a field:

• Alert name and severity — page, ticket, or ignore, with no silent severity drift between docs and the pager.
• Blast radius — the services, queues, and data stores that freeze together with this alert.
• First command — one read-only check with a timeout and a known-good output shape.
• Command budget — how many commands I may run before escalation is mandatory and writes stay blocked.

A YAML contract I will actually merge

The YAML I keep in git looks like this, and I reject pull requests that leave these keys empty.

# runbook/alerts/checkout-latency.yaml
# Proposed template — review before you attach it to a pager.
alert: CheckoutP99High
severity: page
customer_facing: true
blast_radius:
  services: [checkout-api, payments-worker]
  datastores: [checkout-redis]
  queues: [checkout-jobs]
first_command:
  argv: ["kubectl", "-n", "checkout", "get", "deploy,po,hpa"]
  timeout_seconds: 20
  mutates: false
  known_good: "all checkout-api pods Ready"
command_budget:
  max_commands: 3
  after_budget: escalate
escalation:
  primary: checkout-oncall
  backup: payments-oncall
  channel: "#inc-checkout"
freeze:
  default: frozen
  unfreeze_requires: [blast_radius_named, budget_not_exceeded, human_signature]
  ai_may_unfreeze: false

Does that look heavier than a markdown heading and a prayer at three in the morning? Good, because pager weight should hurt a little, or people will keep paging on pure noise. I would rather fight a noisy review than discover a missing blast radius while the pager is screaming. The file is the contract, and Slack is not a contract no matter how many emoji ACK the thread.

First commands that cannot mutate

The first command is not a fix, and I will keep repeating that until the runbook files agree. The first command is a photograph of the blast radius, and it must finish inside the timeout. I want kubectl get, curl on a health path, or a read-only ping, never apply, never delete, never migrate. If your so-called first command needs a write flag, it is not first; it is an unfreeze request wearing a costume.

Would I trust a generated snippet that restarts a deployment because p99 looks sad on a graph? Not before I have the photograph, and not before the freeze is already locked on the blast radius. The runbook must name the jumphost, and it must say the command is read-only without exception. If either line is missing, I keep production frozen and I ping the backup on-call instead of inventing a shell history.

# Proposed read-only photograph. Do not point this article at production.
kubectl -n checkout get deploy,po,hpa -o wide
kubectl -n checkout get deploy checkout-api
kubectl -n checkout get po -l app=checkout-api
curl -fsS --max-time 5 https://checkout.internal/healthz
redis-cli -h checkout-redis PING

Known-good output is part of the command

I also want a known-good string in the YAML so a half-ready replica set cannot pass as healthy. The phrase all checkout-api pods Ready is boring, and boring is exactly what I want at 3 a.m. Fancy dashboards can wait until after the photograph matches, which is a rule I will not bargain. If the known-good line is missing, the first command is incomplete, and incomplete commands do not unlock an unfreeze.

Escalation after the command budget

Three commands is my default budget, and I do not negotiate that number during the incident. After the photograph, one extra read, and one documented diagnostic, I escalate even if I almost see the bug. Have you noticed how almost is when people start typing writes from memory instead of paging backup? The budget exists to interrupt that reflex before it ships a worse outage on the same blast radius.

Role, backup, channel

Escalation in this runbook is a role, a backup role, and a channel, not a feeling in the incident thread. I do not need a theatrical countdown; I need a named human who owns the blast radius when my budget is gone. If checkout-oncall does not answer, payments-oncall is already written down, and I do not invent a third person from the directory. The channel is #inc-checkout, and transcripts of the three commands go there before I page anyone.

After command 3:
  1. Post the photograph and the three command transcripts in #inc-checkout.
  2. Page checkout-oncall with blast_radius in the first line of the message.
  3. If no ACK, page payments-oncall with the same blast_radius line.
  4. Do not type a write. Freeze stays up.

Freeze and unfreeze are different signatures

Freeze is the default when a customer-facing alert fires against a named blast radius in git. Unfreeze is a signature, and it is never automatic just because a dashboard turned green for thirty seconds. Can a model lift that freeze because it wrote a plausible patch in confident prose during the page? Not in this runbook, because ai_may_unfreeze false is a field the validator rejects if you delete it.

What the signer must restate

I treat generated output as a draft comment on the incident thread, never as an unfreeze token for production. The human who signs must restate blast radius, confirm the command budget was not exceeded, and name the write. If they cannot restate those three things in one message, the freeze stays, and I am comfortable looking stubborn.

Drafting the runbook offline

I still want help drafting these YAML files and the read-only first commands after a long shift. Writing them by hand when I am tired is exactly how blast radius and budget fields go missing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode's free model access and free server option as an isolated drafting bench, not as a production jumphost.

It is also not something that can unfreeze a service, and I will not give it that job. The drafting loop looks like this, and every step stays off the pager host on purpose. I want the model to propose fields I can reject, not a patch I can apply. That distinction is the whole reason the freeze exists in the YAML.

The offline loop

1. Paste the alert name and the suspected services into the free server session.
2. Ask the free model access to propose a blast_radius list and a read-only first_command.
3. Reject any proposal that includes apply, delete, migrate, restart, or scale.
4. Copy the surviving YAML into a local git branch and run the validator below.
5. Only then open a runbook pull request for humans who own the blast radius.

Is this slower than letting a copilot patch live traffic during the page, while freeze is still down? Yes, and that slowness is the entire point of keeping the freeze in place on the blast radius. The free server is useful because it is not production, and I will not pretend it observes your metrics. If you draft runbooks the same way, that isolated bench is the only product role I am describing here.

I will not claim token ceilings, model names, hardware, or how long a free plan lasts in this article. I have not verified those details from this desk, and outdated numbers help nobody on a rotation. Free model access and a free server are enough for this drafting loop as I described it. Anything beyond that should be checked on the project itself before you depend on it.

A validator you can run before you accept the rotation

This is the artifact I actually want in CI, because reviewers miss empty keys at 5 p.m. on Friday. It does not page anyone, and it should never run against a production cluster or jumphost. It fails the build when the runbook is still a wiki page in disguise, which is the usual case. Can you merge a runbook that allows a model to unfreeze checkout-api through a missing field?

#!/usr/bin/env bash
# validate-runbook.sh — proposed local check, not a production controller.
set -euo pipefail

FILE="${1:?usage: validate-runbook.sh alerts/foo.yaml}"

need() {
  local key="$1"
  grep -Eq "^${key}:" "$FILE" || { echo "missing ${key} in ${FILE}" >&2; exit 1; }
}

need "alert"
need "severity"
need "blast_radius"
need "first_command"
need "command_budget"
need "escalation"
need "freeze"

grep -q "mutates: false" "$FILE" || { echo "first_command must set mutates: false" >&2; exit 1; }
grep -q "ai_may_unfreeze: false" "$FILE" || { echo "AI must not be allowed to unfreeze" >&2; exit 1; }

if grep -Eiq "kubectl apply|kubectl delete|migrate|DROP TABLE" "$FILE"; then
  echo "runbook contains a mutating command; refuse the rotation" >&2
  exit 1
fi

echo "ok: ${FILE} names blast radius, a read-only first command, and a freeze AI cannot lift"

Run it on every file under runbook/alerts before you say yes to the next week of nights. The find pipeline below is the whole pre-rotation gate I want, and it belongs in CI rather than in a sticky note. If the script exits nonzero, I refuse the rotation, and I send the failure to the service owner. That refusal is cheaper than an unfreeze that nobody can explain later in the incident timeline.

find runbook/alerts -name '*.yaml' -print0 | xargs -0 -n1 ./validate-runbook.sh

Decision table I keep next to the pager

I keep this table next to the pager, and I treat a skipped row as a failed runbook, not a shortcut. Can a green graph override a missing blast_radius line in git during the first minute? It cannot, because the table says freeze and refuse, not eyeball the dashboard. I want the next on-call to make the same call I would make without a private Slack aside.

Situation	Freeze?	Next action	Unfreeze allowed?
Alert missing blast_radius	Yes	Refuse the page; ping runbook owner	No
First command would mutate	Yes	Rewrite as read-only or escalate	No
Command budget spent	Yes	Page primary, then backup	No
Model proposed a patch	Yes	File the draft offline only	No
Human restates blast radius and the write	Stays until signature	Second on-call ACKs	Yes, after ACK

Would I skip a row because the dashboard looks fine for a minute during a noisy page? That is how silent data loss starts, so I do not skip rows even when graphs look friendly. The table is part of the runbook in git, not a slide I remember from last quarter's retro. If a situation is not in the table, I escalate rather than inventing a fifth kind of unfreeze.

What this does not solve

This workflow assumes you can freeze a blast radius without taking down the whole company, which is not always true. It also assumes git-hosted runbooks and a CI hook, which a two-person team on a shared cluster may not have. The validator is a grep net, not a policy engine, and a determined paste can still smuggle a write inside quotes. A drafting bench does not watch production metrics, and it does not replace an incident commander who can say no.

I have not claimed latency numbers or model catalogs here because I cannot verify them from this desk today. If your compliance rules forbid sending alert names to any external drafting tool, skip the bench entirely. Keep the YAML rules anyway, because blast radius and command budget do not depend on a drafting product. The freeze still belongs to humans, and that part of the runbook should survive any tool change.

Who should not use this

Do not use this if you are already in an active incident and you still lack a freeze path. Do not use it if legal will not allow service lists in an external editor, even a free server. Do not use it as an excuse to page fewer people while you experiment with generated YAML on the jumphost. Those three cases need a human process first, not another file that pretends to be a runbook.

If you cannot name blast radius, you do not get an unfreeze, and I will not pretend otherwise. If you cannot name a command budget, you do not get an unfreeze either, even when the graphs look calm. Everything else, including a helpful draft from an offline bench, waits until those two lines exist in git. That is the whole method, and it is intentionally boring on purpose for the people who get the page.