Your Runbook Has No Unfreeze Test: Rehearsing On-Call Freezes With a Fake Clock

Every runbook I have inherited tests the entrance and forgets the exit. We rehearse the page, rehearse the first command, rehearse the escalation, and then we ship a freeze rule whose only validation is that a human remembered to lift it. So here is the conclusion up front: a freeze without an executable unfreeze step and a rehearsal that proves it runs is not a safety mechanism, it is a scheduled outage with a nicer name.

I have watched a read-only window outlive its incident by four hours because the ticket that created it had an owner and the ticket that closed it did not. The alert that page fired was correct, the customer impact was real, and the fix was a single annotation removal nobody was assigned to type. Once you have seen that, you stop trusting "the freeze expires on its own" as a control.

False freezes outlive incidents for three boring reasons

None of these are exotic. They are process gaps that a parser can catch in under a second, which is exactly why it is embarrassing that we catch them during a postmortem instead.

1. The freeze has an owner but the unfreeze has no assignee. The page names whoever pressed the button; nothing names whoever releases it, so release competes with sleep.
2. Expiry is treated as a command. A timestamp in a ticket is not an action, and anything that silently reverts at 09:30 during a deploy is a load test you did not schedule.
3. The exit path is undocumented, so it is improvised. The person who unfreezes at 3 a.m. is rarely the person who froze at 8 p.m., and reverse-engineering a kubectl invocation from bash history is not a runbook.

So the question I now ask of every runbook is narrow and rude: show me the test that fails when the unfreeze step is deleted. If you cannot produce it, the freeze rule is decoration.

The runbook contract I now require

I keep the document machine-readable next to the human prose, because a contract that cannot be parsed cannot be tested. Here is the shape I have been using for a checkout service; it is deliberately small, and it is the fixture my harness reads.

# runbook.yaml
service: checkout-api
alerts:
  - name: CheckoutP99LatencyHigh
    first_commands:
      - run: kubectl -n checkout top pods --sort-by=cpu | head -6
        timeout_s: 30
      - run: curl -s localhost:9090/metrics | grep checkout_queue_depth
        timeout_s: 15
    escalate_to: payments-oncall
    escalate_after_s: 600
freeze:
  id: checkout-writes
  owner: "@oncall-primary"
  expires_at: "2026-10-17T09:30:00Z"
  unfreeze:
    command: kubectl -n checkout annotate deploy/checkout-api freeze- --overwrite
    verify: kubectl -n checkout get deploy/checkout-api -o jsonpath='{.metadata.annotations.freeze}'
    expected: ""

Notice what is mandatory in that block and what is not. The owner, the expiry, an unfreeze command, and a verify-plus-expected pair are all required; a freeze with no way to observe its own removal is rejected, because "I ran the command" is a claim and an empty annotation is evidence.

The rehearsal harness: a fake clock plus an exit assertion

The harness does two jobs, and I care about the second one far more. First it lints the contract; then it replays a page against an injected clock and refuses to pass unless the unfreeze step actually executed inside the grace window.

# drill.py — stdlib + PyYAML only
import sys
from datetime import datetime, timedelta, timezone
import yaml

GRACE = timedelta(minutes=15)
MAX_ESCALATE = timedelta(minutes=15)


def parse_ts(value):
    return datetime.fromisoformat(str(value).replace("Z", "+00:00"))


def lint(doc, page_at):
    errs = []
    alerts = doc.get("alerts") or []
    if not alerts:
        errs.append("E_NO_ALERTS")
    ladder = []
    for alert in alerts:
        name = alert.get("name", "<unnamed>")
        cmds = alert.get("first_commands") or []
        if not cmds:
            errs.append(f"E_NO_FIRST_COMMAND:{name}")
        for i, cmd in enumerate(cmds):
            if not cmd.get("run"):
                errs.append(f"E_EMPTY_CMD:{name}#{i}")
            timeout = cmd.get("timeout_s")
            if not isinstance(timeout, int) or not 0 < timeout <= 120:
                errs.append(f"E_BAD_TIMEOUT:{name}#{i}")
        after = alert.get("escalate_after_s")
        if not isinstance(after, int) or not 0 < after <= MAX_ESCALATE.total_seconds():
            errs.append(f"E_ESCALATION_BUDGET:{name}")
        else:
            ladder.append(after)
    if ladder != sorted(ladder):
        errs.append("E_ESCALATION_NOT_MONOTONIC")

    freeze = doc.get("freeze")
    if freeze:
        if not freeze.get("owner"):
            errs.append("E_FREEZE_NO_OWNER")
        if not freeze.get("expires_at"):
            errs.append("E_FREEZE_NO_EXPIRY")
        unfreeze = freeze.get("unfreeze") or {}
        if not unfreeze.get("command"):
            errs.append("E_NO_UNFREEZE_COMMAND")
        if not unfreeze.get("verify") or "expected" not in unfreeze:
            errs.append("E_NO_UNFREEZE_VERIFY")
        else:
            if parse_ts(freeze["expires_at"]) <= page_at:
                errs.append("E_FREEZE_ALREADY_STALE_AT_PAGE")
    return errs


def rehearse(doc, page_at, unfreeze_at):
    errs = lint(doc, page_at)
    freeze = doc.get("freeze")
    if freeze:
        if unfreeze_at is None:
            errs.append("E_FREEZE_NEVER_LIFTED")
        else:
            deadline = parse_ts(freeze["expires_at"]) + GRACE
            if unfreeze_at > deadline:
                errs.append("E_UNFREEZE_LATE")
    return errs


if __name__ == "__main__":
    doc = yaml.safe_load(open(sys.argv[1]))
    page_at = parse_ts(sys.argv[2])
    unfreeze_at = parse_ts(sys.argv[3]) if len(sys.argv) > 3 else None
    failures = rehearse(doc, page_at, unfreeze_at)
    print("\n".join(failures) or "OK")
    sys.exit(1 if failures else 0)

Run it against a good runbook and a bad one; the difference should be exit codes, not opinions.

python drill.py runbook.yaml 2026-10-17T08:05:00Z 2026-10-17T09:12:00Z   # prints OK, exit 0
python drill.py runbook.yaml 2026-10-17T08:05:00Z                          # E_FREEZE_NEVER_LIFTED, exit 1
python drill.py stale-freeze.yaml 2026-10-17T10:00:00Z 2026-10-17T10:20:00Z # E_FREEZE_ALREADY_STALE_AT_PAGE, exit 1

In CI I do not run the third form against production state. The harness reads a checked-in fixture of the alert and freeze metadata, so the drill is deterministic and safe to run on every pull request; the only live call it makes is the read-only verify command, and I gate that behind a label.

The three fixtures that earn their keep

• missing-unfreeze.yaml — freeze block with an owner and expiry but no command. Expects E_NO_UNFREEZE_COMMAND.
• unfreeze-after-close.yaml — a realistic page where the incident closed at 08:52 but the annotation was removed at 11:40. Expects E_UNFREEZE_LATE.
• silent-timeout.yaml — a first command with no timeout_s, which in practice means someone waits on a hung kubectl until the escalation budget is gone. Expects E_BAD_TIMEOUT.

Each fixture is a two-line diff someone can review in ten seconds, and each one maps to a failure I have actually seen. That is the bar for adding a new check.

Where a free model actually helped, and where it did not

Drafting first_commands from a bare alert name is tedious, so I tried generating candidates with a hosted model instead of typing them. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project's page advertises free model access and a free server option, and I used the free tier for exactly this step: propose three read-only commands per alert, then let the harness grade them. I am not going to quote model names, quotas, or hardware here, because those details change and my job is the workflow; check the project page for current terms before you plan around them.

What made the free tier usable was treating model output as untrusted input, not as a runbook. A second pass rejects anything that mutates state before a human ever sees it.

# quarantine.py — deny-first, allow-by-exception
DENY = ("delete", "scale", "rollout undo", "apply", "patch", "rm ", "drain", "cordon", "secrets")

def quarantine(candidates):
    safe, needs_human = [], []
    for cmd in candidates:
        line = cmd.get("run", "")
        if not cmd.get("timeout_s"):
            needs_human.append((line, "missing timeout"))
        elif any(token in line for token in DENY):
            needs_human.append((line, "write-class verb"))
        else:
            safe.append(cmd)
    return safe, needs_human

The failures were instructive and always the same shape: the model is happy to suggest kubectl rollout undo as a "first command", which is an action, not a diagnosis. It also invents metric names that look plausible. Both defects are caught by the timeout rule and the denylist, but only because I refused to let it write directly into the YAML.

I also used the free server option to host the drill runner as a tiny internal service that accepts a runbook path and returns the failure list. It holds no cluster credentials, reads fixtures from disk, and is disposable; if it disappears tomorrow I lose ten minutes, not an incident. If you want to try the same setup, the project page is the place to check what is currently free before you commit a weekend to it.

Limitations, and who should skip this

This harness checks the shape of your runbook and the ordering of your exit path. It cannot tell you whether kubectl top is the right first command, and it will happily pass a runbook full of confident nonsense. It also assumes you can express freeze state as something observable, which rules out teams whose lockdown lives in a human conversation.

Skip it if you are a two-person team without a paging rotation, because you will spend more time maintaining fixtures than you save. Skip it if your freezes are already enforced by infrastructure that audits its own removals. And skip it if you cannot yet name an owner for the unfreeze, because that is not a tooling problem and no test will fix it.

If you take one thing from this, take the failing assertion rather than the parser. Write down who lifts the freeze, how you observe that it is lifted, and how long after the incident closing that must happen. Then delete the unfreeze line and confirm your pipeline turns red. What does your runbook do today when someone removes that one line?