Your On-Call Runbook Should Be Executable: Alerts, First Commands, and a Freeze Rule You Can Audit

Most on-call runbooks fail for a boring reason: they are readable documents, not executable procedures. When the pager fires at 03:12, nobody has the attention budget to translate prose into commands and then argue about whether a deploy counts as a "write". If your runbook cannot be run, tested, and diffed, it is a wish list with headers.

So here is the version I keep coming back to: every page has three parts — a triage block, a first-commands block, and a freeze rule that is enforced in code rather than in a wiki page nobody re-reads. The goal is not to remove human judgement; it is to make the judgement reproducible and auditable.

Alerts: collect three fields before you touch anything

An alert without context is just an anxiety generator. Before any command, I want the alert fingerprint, the affected surface, and the change window. Write those three fields into the incident channel first — not as bureaucracy, but because they decide everything that follows.

Why the fingerprint? Because "checkout is slow" and "checkout returns 5xx on one route in one region" are different incidents with different owners, and the fingerprint is what stops two responders from debugging two different problems for forty minutes.

The first five commands

Keep this list short enough to memorize. Everything else is a follow-up, not a first move.

#!/usr/bin/env bash
set -euo pipefail

# 0. Fingerprint the alert before acting.
printf 'alert=%s surface=%s region=%s\n' "$ALERT_ID" "$SURFACE" "$REGION"

# 1. What is actually serving right now?
curl -sS --max-time 5 "$STATUS_URL/healthz" | jq '{status, version, build}'

# 2. What changed in the last 60 minutes?
git -C "$REPO" log --since='60 minutes ago' --pretty='%h %an %s'

# 3. One probe per dependency, capped and timed.
for dep in "$@"; do
  printf '%-30s ' "$dep"
  curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --max-time 3 "$dep/healthz"
 done

# 4. Rate of failure, not raw count.
curl -sS "$METRICS/api/v1/query" --data-urlencode \
  'query=sum(rate(http_requests_total{code=~"5.."}[5m])) by (route)'

Notice what is missing: no restarts, no scaling, no cache flushes. Those are interventions, and interventions belong after the freeze rule has been evaluated, not before. Restarting first is how you destroy the evidence that would have told you what broke.

The freeze/unfreeze rule, written as code

This is the part most teams skip, and it is the part that prevents a small incident from becoming a long one. "Don't deploy during an incident" is not a rule; it is a sentiment. A rule names the owner, the ticket, the expiry, and the blocked action classes — and then it refuses writes that violate it.

A policy file you can diff

# freeze.yaml
freeze:
  active: true
  owner: "@emery-oncall"
  ticket: "INC-214"
  reason: "5xx rate above SLO on checkout"
  started_at: 1792000000      # epoch seconds
  expires_at: 1792005400      # 90 minutes; no silent forever-freezes
  blocked_actions:
    - deploy
    - migrate
    - config_write
    - secret_rotate
unfreeze_requires:
  - "error rate under SLO for 30 consecutive minutes"
  - "rollback file named in the handoff packet"
  - "second human acknowledgement"

The gate that enforces it

# freeze_gate.py — deny write-class actions while a freeze is active and unexpired.
import time
from pathlib import Path
import yaml

POLICY_PATH = Path("freeze.yaml")

class Denied(Exception):
    """Raised when an action is not permitted right now."""

def assert_writable(action: str, policy: dict, now: float | None = None) -> None:
    now = time.time() if now is None else now
    freeze = policy.get("freeze") or {}
    if not freeze.get("active"):
        return
    if action not in freeze.get("blocked_actions", []):
        return
    if not freeze.get("owner") or not freeze.get("ticket"):
        raise Denied("freeze has no owner or ticket; writes stay blocked")
    if float(freeze["expires_at"]) < now:
        raise Denied("freeze expired; renew explicitly or lift it, do not ignore it")
    raise Denied(f"frozen by {freeze['owner']} for {freeze['ticket']}")

def main() -> int:
    policy = yaml.safe_load(POLICY_PATH.read_text())
    try:
        assert_writable("deploy", policy)
    except Denied as exc:
        print(f"blocked: {exc}")
        return 1
    print("allowed")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Prove it with tests, not with hope

import time
import pytest
from freeze_gate import Denied, assert_writable

BASE = {
    "freeze": {
        "active": True,
        "owner": "@emery-oncall",
        "ticket": "INC-214",
        "expires_at": 1792005400,
        "blocked_actions": ["deploy", "migrate"],
    }
}

def test_blocks_deploy_during_freeze():
    with pytest.raises(Denied):
        assert_writable("deploy", BASE, now=1792000001)

def test_allows_unlisted_action():
    assert_writable("read_logs", BASE, now=1792000001) is None

def test_expired_freeze_does_not_silently_lift():
    with pytest.raises(Denied, match="renew"):
        assert_writable("deploy", BASE, now=1792009999)

def test_ownerless_freeze_is_invalid():
    policy = {"freeze": {**BASE["freeze"], "owner": ""}}
    with pytest.raises(Denied, match="owner"):
        assert_writable("deploy", policy, now=1792000001)

The expired-freeze test is the important one. A freeze that lapses on a timer and quietly re-enables deploys mid-incident is worse than no freeze at all, because now you have the illusion of a control.

Decision table: page, note, or wake someone up

Use this when you are unsure whether the incident justifies pulling a second person out of bed. It is deliberately blunt.

Signal	Action
SLO burn rate normal, one noisy alert	Note in channel, close after 15 min
5xx above SLO on one route, no customer report	Page primary, start triage block
Any data loss or corruption path	Wake secondary + data owner immediately
Write performed during an active freeze	Stop, revert, escalate to freeze owner
Unknown cause after 30 min of triage	Escalate; do not keep guessing alone

Three sentences of instruction beat a paragraph of philosophy, and a table beats both. The escalation ladder is short on purpose: primary → secondary → service owner → incident commander. Each hop has a time budget you agreed on before the page, not during it.

Where a free model and a free server actually fit

Here is the honest part. Two things in this workflow are tedious and low-judgement: rehearsing the runbook against historical alert text, and hosting the freeze state somewhere that is not a laptop that closes when the responder walks away.

That is where MonkeyCode's free model access and free server option come in for me. I use the free model access to dry-run the triage block against redacted past alerts and to flag steps in the runbook that contradict each other, and I keep the freeze gate plus the policy file on the free server so the state survives a closed lid.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes a free token allowance alongside the free model access and free server option; I have not benchmarked either, and the current allowance is whatever the project page says today, not a number you should hard-code into a design doc.

Two boundaries I hold firmly. The model never decides whether to escalate — that stays a human call with a timestamp. And I would not paste raw customer data or secrets into any rehearsal, free or paid, without checking the project's data handling first.

Limitations, and who should not use this

• You do not have an SLO yet. A freeze rule without a target to unfreeze against is just a longer outage with extra steps.
• You cannot name an owner per alert. The gate will block writes and nobody will be able to lift the freeze; fix the ownership map first.
• Regulated or contractual data. Check data handling before routing any incident text through a hosted model.
• You want the assistant to make the call. Don't. Use it for rehearsal and contradiction-hunting, not for escalation authority.

Also treat free tiers as free tiers: useful for exercises and low-stakes internal tooling, never as the only place your production paging path depends on.

A 30-minute drill that proves the runbook is real

1. Pick one real alert from the last 30 days and redact it.
2. Run the triage block out loud with a colleague and time each command.
3. Flip active: true in freeze.yaml, then attempt a deploy and confirm the gate blocks it.
4. Set expires_at to the past, confirm it still blocks, and confirm a human has to lift it explicitly.
5. Record the two steps that took longest and rewrite them as a single command.

If step five produces no changes, you either have a genuinely excellent runbook or you ran the drill in your head. If you want to try the gate end to end without provisioning anything, the two files above plus the free server option are enough to get a working rehearsal today.