Your Runbook Drifted Since the Last Incident: Audit Alert Names, Permissions, and Step Intent in CI

A runbook rarely fails because the idea behind it was wrong; it fails because something around it moved while nobody was looking. The alert got renamed during a dashboard cleanup, the rotation lost a cluster role in a permissions refactor, or the prose says "restart the pod" while the command quietly deletes the deployment. All three of those are mechanical, testable, and cheap to catch before the pager goes off — so why do we still discover them from a laptop at 3 a.m.?

Here is the workflow I keep coming back to: give every runbook step a small machine-readable block, then run three passes over it — alert-name resolution, permission proof, and a semantic check that the command actually matches the stated intent. The first two are deterministic and run in CI, and the third only needs a throwaway model endpoint.

Why runbooks rot quietly

Review culture is optimized for the day the runbook is written, not the six months after it. Docs get review comments about clarity, tone, and ordering, but almost nobody re-reads them when the alerting rules change or when the platform team tightens RBAC. That is why drift accumulates silently and surfaces only under pressure.

There are three drift classes worth separating, because they need different detectors:

• Renamed or deleted alerts. The runbook references CheckoutLatencyP99High, but the rule is now checkout_latency_p99_high or lives in a different rule file entirely.
• Revoked permissions. The step was authored by someone with cluster-admin, so it "worked" once; the on-call rotation only has namespace-scoped patch rights.
• Prose that no longer matches the command. Someone edited the command during an incident and updated the title loosely, so the step now claims one outcome and produces another.

Give every step a machine-readable block

If you keep runbook steps as free-form prose, no tool can help you without an NLP project. Pick a boring convention instead: one ### heading per step, followed by a fenced runbook block of YAML. Humans still read the prose around it; the auditor reads the block.

### Checkout latency page

The p99 latency alert fires when the checkout API is slow but still serving traffic.

```runbook
alert: CheckoutLatencyP99High
namespace: prod
command: kubectl -n prod rollout restart deploy/checkout-api
requires:
  - patch deployments
intent: restart the checkout API pods so a bad in-memory cache is rebuilt
read_only: false
```

If the restart does not clear the alert within ten minutes, escalate per the contacts section.

That is it. One block per step, four fields that matter, and a requires list written in the same verb resource shape that kubectl auth can-i already understands.

Pass 1 — every alert name must resolve

The cheapest check is also the one that catches the most embarrassing failures. Parse the runbook, collect the alert fields, then parse your Prometheus rule files and confirm each name exists.

# alerts/checkout.yml
groups:
  - name: checkout
    rules:
      - alert: CheckoutLatencyP99High
        expr: histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m]))) > 1.5
        for: 10m

Run promtool check rules alerts/checkout.yml first so you are comparing against syntactically valid rules, then compare the name sets yourself. A missing name is not a style nit; it means the person paged at 3 a.m. will search for an alert that no longer exists.

Pass 2 — prove the rotation can actually run the command

This is the check most teams skip, and it is the one that converts a runbook into something executable. kubectl auth can-i is a read-only self-check: it answers what the current identity may do, without changing anything. Run it with the rotation's credentials, not your admin kubeconfig — otherwise every probe returns yes and proves nothing.

# As the on-call identity, not as an admin:
kubectl auth can-i patch deployments -n prod
# yes  -> the step is runnable
# no   -> the runbook is fiction for anyone on rotation

Do the same for whatever the step actually touches. If a step shells into a bastion, run ssh -o BatchMode=yes bastion true and check the exit code. If it calls an internal API, hit a /healthz endpoint with the token the rotation holds. The point is to turn "we think this works" into a recorded yes or no.

Pass 3 — catch prose that lies about the command

Name resolution and permissions are binary, but intent mismatch is fuzzy, which is exactly where a model earns its keep. For each step, send the intent and the command and ask one narrow question: does this command plausibly achieve that intent? Keep the prompt boring and demand JSON, so the output is parseable rather than a paragraph of encouragement.

SEMANTIC_PROMPT = """You audit an on-call runbook step. Reply with JSON only.
{"mismatch": true|false, "reason": "<one sentence>"}
Set mismatch=true when the command does not plausibly achieve the stated intent."""

This is the AI usage I actually endorse during the current wave of AI fatigue: narrow, gated, reversible, and attached to a review step rather than a production action. It flags candidates; a human still deletes the step.

Where free model access fits — and the honest caveat

MonkeyCode is an open-source project that provides free model access and a free server option, and it advertises a free allotment in the range of ten million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not benchmarked its throughput or model quality for this workflow, and the quota figure is the project's own claim rather than something I measured.

What makes it a reasonable fit for pass 3 is the shape of the job: a handful of short requests per pull request, an endpoint behind an environment variable, and zero tolerance for a surprise bill. Point the auditor at any OpenAI-compatible base URL and the semantic pass gets cheaper to try.

export MC_BASE_URL="https://<your-endpoint>/v1"
export MC_MODEL="<your-model-id>"
export MC_API_KEY="<your-key>"
python tools/runbook_audit.py runbooks/checkout.md \
  --rules alerts/checkout.yml --semantic

The auditor, in one file

Here is the whole thing: parse the blocks, run the three passes, exit non-zero on drift.

#!/usr/bin/env python3
"""runbook_audit.py — catch runbook drift before the next page."""
import argparse, json, os, pathlib, re, shutil, subprocess, sys, urllib.request
import yaml

STEP_RE = re.compile(r"^### +(?P<title>.+)\n```

runbook\n(?P<body>.*?)\n

```", re.S | re.M)

def load_steps(runbook):
    steps = []
    for m in STEP_RE.finditer(pathlib.Path(runbook).read_text(encoding="utf-8")):
        step = yaml.safe_load(m.group("body")) or {}
        step["title"] = m.group("title").strip()
        steps.append(step)
    return steps

def alert_names(rules):
    names = set()
    for doc in yaml.safe_load_all(pathlib.Path(rules).read_text(encoding="utf-8")):
        for group in (doc or {}).get("groups", []):
            for rule in group.get("rules", []):
                if "alert" in rule:
                    names.add(rule["alert"])
    return names

def check_alerts(steps, known):
    return [f"[alert] {s['title']}: '{s.get('alert')}' not in rules"
            for s in steps if s.get("alert") not in known]

def check_grants(steps, kubectl="kubectl"):
    if not shutil.which(kubectl):
        return ["[grant] kubectl not found; pass skipped"]
    problems = []
    for s in steps:
        for spec in s.get("requires") or []:
            verb, _, resource = spec.partition(" ")
            cmd = [kubectl, "auth", "can-i", verb, resource]
            if s.get("namespace"):
                cmd += ["-n", s["namespace"]]
            out = subprocess.run(cmd, capture_output=True, text=True)
            if out.stdout.strip().lower() != "yes":
                problems.append(f"[grant] {s['title']}: cannot {verb} {resource}")
    return problems

SEMANTIC_PROMPT = """You audit an on-call runbook step. Reply with JSON only.
{"mismatch": true|false, "reason": "<one sentence>"}
Set mismatch=true when the command does not plausibly achieve the stated intent."""

def check_intent(steps, base_url, model, api_key):
    problems = []
    for s in steps:
        payload = {
            "model": model,
            "temperature": 0,
            "messages": [
                {"role": "system", "content": SEMANTIC_PROMPT},
                {"role": "user", "content": json.dumps(
                    {"intent": s.get("intent"), "command": s.get("command")})},
            ],
        }
        req = urllib.request.Request(
            f"{base_url.rstrip('/')}/chat/completions",
            data=json.dumps(payload).encode(),
            headers={"Content-Type": "application/json",
                     "Authorization": f"Bearer {api_key}"},
        )
        with urllib.request.urlopen(req, timeout=60) as resp:
            content = json.load(resp)["choices"][0]["message"]["content"]
        verdict = json.loads(re.search(r"\{.*\}", content, re.S).group(0))
        if verdict.get("mismatch"):
            problems.append(f"[intent] {s['title']}: {verdict.get('reason')}")
    return problems

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("runbook", type=pathlib.Path)
    ap.add_argument("--rules", type=pathlib.Path)
    ap.add_argument("--semantic", action="store_true")
    ap.add_argument("--base-url", default=os.environ.get("MC_BASE_URL"))
    ap.add_argument("--model", default=os.environ.get("MC_MODEL"))
    args = ap.parse_args()

    steps = load_steps(args.runbook)
    problems = check_grants(steps)
    if args.rules:
        problems += check_alerts(steps, alert_names(args.rules))
    if args.semantic:
        problems += check_intent(steps, args.base_url, args.model,
                                 os.environ.get("MC_API_KEY", ""))
    for p in problems:
        print(f"DRIFT {p}")
    print(f"{len(steps)} steps checked, {len(problems)} problems")
    return 1 if problems else 0

if __name__ == "__main__":
    sys.exit(main())

I am labeling this as a reference implementation, not a battle-tested product: I have not run it against your cluster, and pass 2 is only as trustworthy as the identity you run it with.

Wire it into the handoff, not just CI

name: runbook-audit
on:
  pull_request:
    paths: ["runbooks/**", "alerts/**"]
  schedule: [{cron: "0 6 * * 1"}]
jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install pyyaml
      - run: python tools/runbook_audit.py runbooks/checkout.md --rules alerts/checkout.yml

The scheduled run matters more than the pull-request run, because drift arrives from other teams' changes. Alerts get edited in a rules repository you don't own, and roles get tightened in a platform repo you never touch. A weekly audit catches both without asking anyone to remember.

What this audit will not catch

• It cannot tell you the step is correct, only that it is consistent. A step whose alert exists, whose permission exists, and whose prose matches the command can still be bad advice.
• It ignores judgment steps entirely. "Decide whether to fail over to the secondary region" has no command and no intent to compare, so passes 2 and 3 skip it silently.
• The semantic pass has false positives. Treat its output as a review prompt, and start it as a warning rather than a blocking gate until you have tuned it against your own runbooks.
• It says nothing about escalation contacts. Names resolve, permissions hold, and the escalation section can still point at someone who left the company.
• It does not work if you run pass 2 as an admin. You will get green checks that mean nothing, which is worse than no check at all.

Who should skip this? Teams with two or three runbooks that the same two people maintain from memory will spend more time on the YAML convention than they save. If your pages are already handled by a small group whose runbooks live in their heads, fix the documentation problem first and come back when the rotation grows.

Limitations on the model endpoint

Before you enable --semantic, decide what leaves your network. Runbook steps often embed internal hostnames, ticket prefixes, and occasionally a token that someone pasted in a hurry. Send a redacted copy, or run the model on the free server option if your data policy allows a hosted endpoint at all. Also remember that a hosted model may be revised or retired without warning, so the semantic pass should be optional and never the only thing standing between you and a broken runbook.

Pre-handoff checklist

1. Every step has a runbook block with alert, command, intent, and requires.
2. promtool check rules passes on the rule files the runbook references.
3. kubectl auth can-i was run as the rotation identity, not as an admin.
4. The semantic pass ran at least once and produced a diff someone read.
5. The escalation section names people who are still on the team this quarter.
6. The audit runs weekly on a schedule, not only when the runbook is edited.

If you want to try pass 3 without wiring a paid key, point MC_BASE_URL at MonkeyCode's free server and see how many of your steps survive the intent check — the format and the first two passes are the part that pays off regardless.