Your Runbook Is a Graph, Not a Document: Proving Every Alert Has a Way Out

The most expensive runbook bug is not a wrong command. It is a step nobody can reach, or an alert that lands on a page with no exit. We keep reviewing runbooks as prose, though, so the structural defects stay invisible until 3am.

So here is the reframe I want to argue for: treat your runbook as a directed graph, audit it like one, and let a machine tell you which alerts are stranded. That single change catches a class of failure that grammar, lint, and even careful human review will happily miss. Does your runbook currently survive that test? Most don't, and mine didn't either on the first pass.

Three Tables Beat One Long Document

The graph model needs only three collections, and you can hold all three in one YAML file that lives next to the service it describes.

1. Alerts — the name, the source, the severity, and the entry step where a human starts.
2. Steps — the owner, the commands, the allowed next steps, and whether the step is terminal (we are done).
3. Rules — freeze windows, who can lift them, and which steps are only safe while a freeze is active.

That is it. No prose narrative, no screenshots of dashboards, no "see the wiki". If a step has no owner, it is not a step, it is a hope.

# runbook.yaml
alerts:
  - name: API5xxHigh
    source: prometheus
    severity: page
    entry: triage_api
  - name: DiskFullNode
    source: prometheus
    severity: page
    entry: triage_disk

steps:
  triage_api:
    owner: oncall-platform
    commands:
      - kubectl -n prod rollout status deploy/api --timeout=30s
      - kubectl -n prod get pods -l app=api
    next: [rollback_api, escalate_platform]
  rollback_api:
    owner: oncall-platform
    commands:
      - kubectl -n prod rollout undo deploy/api
    terminal: true
  escalate_platform:
    owner: oncall-platform
    commands:
      - page the platform secondary
    requires_freeze: true
    next: []
  triage_disk:
    owner: oncall-infra
    commands:
      - df -h /var/lib
    next: [escalate_platform]

Spot the defect yet? escalate_platform has no terminal: true and no outgoing edge, so it is a dead end — the page goes to a human who has no recorded next action. You only see it because the shape is now inspectable, and that is exactly the point.

Four Checks Worth Automating Today

Once the runbook is a graph, a short script answers questions that reviews cannot answer consistently.

• Orphan alerts: every alert's entry must exist as a step, otherwise the page renders a blank page at the worst moment.
• Dead ends: every reachable non-terminal step needs at least one outgoing edge, or the responder stalls.
• Hop distance: the nearest terminal step should be within a small number of hops, because a five-hop path is a five-decision path under stress.
• Freeze traps: any step with requires_freeze must also declare an unfreeze_trigger, or the runbook can strand a live incident inside a change freeze.

I would also flag unreachable steps, which is dead code written in English. Nobody deletes them, everybody reads them, and they quietly teach the wrong procedure.

Listing 2: The Reachability Checker

This is a scaffold, not a finished product — stdlib plus PyYAML, BFS over the step graph, non-zero exit code so CI can gate on it. I have not executed this exact copy, so run it against your file and adjust before you trust it.

#!/usr/bin/env python3
"""runbook_lint.py - reachability audit for an on-call runbook graph."""
import sys
from collections import deque
import yaml

MAX_HOPS = 3

def load(path):
    with open(path, encoding="utf-8") as fh:
        return yaml.safe_load(fh)

def walk(steps, start):
    """Return {step: hops} for everything reachable from start."""
    dist, queue = {start: 0}, deque([start])
    while queue:
        cur = queue.popleft()
        for nxt in steps.get(cur, {}).get("next") or []:
            if nxt not in dist:
                dist[nxt] = dist[cur] + 1
                queue.append(nxt)
    return dist

def audit(rb):
    steps, alerts, problems = rb["steps"], rb["alerts"], []
    reachable = set()
    for alert in alerts:
        entry = alert.get("entry")
        if entry not in steps:
            problems.append(f"{alert['name']}: entry step {entry!r} missing")
            continue
        dist = walk(steps, entry)
        reachable |= set(dist)
        terminals = [s for s in dist if steps[s].get("terminal")]
        if not terminals:
            problems.append(f"{alert['name']}: no terminal step reachable")
        elif min(dist[s] for s in terminals) > MAX_HOPS:
            problems.append(f"{alert['name']}: terminal too far (>{MAX_HOPS} hops)")
        for step in dist:
            if steps[step].get("requires_freeze") and not steps[step].get("unfreeze_trigger"):
                problems.append(f"{alert['name']}: freeze at {step} has no unfreeze trigger")
    for name, step in steps.items():
        if not step.get("owner"):
            problems.append(f"{name}: no owner")
        if name not in reachable:
            problems.append(f"{name}: unreachable from any alert")
        elif not step.get("next") and not step.get("terminal"):
            problems.append(f"{name}: dead end")
    return problems

if __name__ == "__main__":
    issues = audit(load(sys.argv[1]))
    print("\n".join(issues) if issues else "ok: runbook graph is reachable")
    sys.exit(1 if issues else 0)

Run it on the YAML above and you get three complaints in one pass: the dead end, the missing unfreeze trigger, and the disk alert whose only route runs through a freeze-gated step. That is a real design smell, and it took four seconds.

Let a Free Model Draft the Graph, Keep the Machine as Judge

Hand-writing YAML for forty steps is boring, so I treat the conversion as a drafting task rather than a design task. Give a model the existing prose runbook and a strict output contract, and ask only for structure — never for new commands.

Convert the runbook below into YAML with exactly three top-level keys:
- alerts: [{name, source, severity, entry}]
- steps: {<id>: {owner, commands[], next[], terminal?, requires_freeze?, unfreeze_trigger?}}
- rules: {freeze_windows[]}
Rules: copy commands verbatim, never invent a command, mark a step terminal
only if the text says the incident is resolved, and output YAML only.

This is where MonkeyCode fits the workflow rather than the marketing: it advertises free model access and a free server option, which makes the drafting pass cheap enough to run on every runbook PR instead of once a quarter. The free-token allowance I was given for this article was described to me as 10 million tokens, and I have not independently benchmarked it or the server, so check the project's current terms before you design a pipeline around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The important discipline is that the model's output is a draft, and the checker is the judge. Never let the drafter be the reviewer, because a model that invents next edges will produce a graph that looks beautiful and routes nowhere.

Where the Free Server Actually Earns Its Keep

Be honest with yourself: a 90-line static checker does not need a cloud box, and a laptop cron is fine. The scheduled part that benefits is the diff — running the audit weekly, committing the report, and letting the exit code fail a PR when a new alert arrives without an entry step.

# crontab -e on the free server, Monday 06:17
17 6 * * 1 /opt/runbook/venv/bin/python /opt/runbook/runbook_lint.py \
  /opt/runbook/runbook.yaml >> /var/log/runbook_lint.log 2>&1 || \
  /opt/runbook/notify.sh "runbook graph audit failed"

Two lines, one log, one notification, and the runbook stops rotting silently between incidents. That is the whole pitch — not intelligence, just regularity.

What This Does Not Prove

A reachability audit is a structural check, and it is important to say plainly what it cannot see.

• It does not verify that any command is correct, safe, or idempotent — a perfectly connected graph can still run the wrong kubectl.
• It does not know whether your owners still exist; a named team that disbanded three months ago looks fine to BFS.
• It is only as fresh as the YAML, so the YAML has to be the source of truth, not a copy of the wiki.
• MAX_HOPS is a judgment call, not a law of nature, and parallel steps flatten into a single path in this model.
• Free tiers change; if the drafting pass becomes expensive, the checker still works with a human doing the conversion.

Who Should Skip This

If you have four alerts and one service, this is overhead — a two-line README beats a graph. If your runbooks are already generated from infrastructure code, you likely get reachability from that generator and should not build a parallel one. And if nobody will run the checker in CI, the YAML will decay into a second stale document, which is worse than the prose you started with.

Your situation	Does the graph audit pay off?
10+ alerts, 3+ services, shared on-call rotation	Yes — orphans and dead ends appear fast
Compliance demands freeze windows be respected	Yes — the unfreeze-trigger check is cheap insurance
Single service, single responder, weekly pages	Probably not yet
Runbook is generated from IaC tooling	No — extend the generator instead

Start with the four checks, wire the exit code into one pipeline, and let the first failing run tell you which alert has been pointing at a blank page all along. If you want to try the drafting half without spending tokens, the project's free model access and free server option are documented on the MonkeyCode site — but run the checker against whatever it produces before you trust a single step.