Your Runbook Steps Expire: A Freshness Ledger That Fails CI Before the Alert Does

A runbook is rarely wrong on the day you write it. It goes wrong on the day a flag gets renamed and nobody re-runs the step for four months.

I have spent the last few posts arguing that a runbook is a graph, that its steps need time budgets, and that its escalation ladder belongs in a table. This one is about the axis I kept avoiding: time. How do you make a stale step visible before it gets discovered at 3am by an engineer who has never executed it? My answer is unglamorous — stamp every step with an expiry date, check that date in CI, and use a model only to summarize what changed around the step so re-verification costs five minutes instead of fifty.

"Last reviewed" at the top of the doc is a lie you tell auditors

Almost every runbook I have read has one date in the header and forty steps underneath it. Somebody opens the file, fixes one command, bumps the header date, and now the whole document claims to be fresh. The step that still references a decommissioned bastion inherits that freshness for free.

So why does nobody notice? Because rot is invisible at render time. A broken dashboard link, a flag that no longer exists, a permission that was removed in a quarterly access review — none of them throw errors inside the document. They throw errors at the worst possible moment, in front of someone whose hands are already shaking.

The six ways a step actually rots

• A flag, label, or queue name was renamed by the service owner and the alert still fires.
• The endpoint moved, or a required header appeared in a platform migration.
• A dashboard or doc link returns 404 and nobody clicked it.
• A credential or role was revoked during an access review.
• A timeout or retry default changed, so the command now hangs instead of failing fast.
• The owning team changed, and the step's real expert is now unreachable.

Each of these is detectable by a human in about two minutes. The problem is never detection difficulty — it is the absence of a trigger that says look here, now.

Put the expiry date next to the step, not on the document

I prefer an HTML comment immediately after each step heading. It renders as nothing, survives copy-paste, stays greppable, and needs no YAML front-matter parser for a file that is still fundamentally prose.

## Step: rotate-db-primary
<!-- verified: 2026-09-14 | tier: 1 | owner: platform-data -->

1. Confirm replication lag is under 5s: `...`
2. Drain the primary with `...`
3. Verify writes land on the new primary before removing the old DNS record.

The tier field is the whole trick, because it lets you set different clocks for different risk instead of freezing the entire document at one cadence.

Tier	What lives here	Re-verify every	Who signs off
1	Destructive or credential-touching steps	30 days	Current on-call lead
2	Service-scoped restarts, drains, rollbacks	90 days	Owning team
3	Read-only investigation, log queries	180 days	Anyone on rotation

Have I actually kept a 30-day cadence on a large runbook? Not perfectly, no. But the table is honest about the ambition, and the CI check makes the gap visible instead of theoretical.

The auditor: one file, stdlib only, no dependencies

This script parses the step headings, reads the tier and date, and exits non-zero when anything is over budget. It never touches your infrastructure, so it is safe to run on every pull request.

#!/usr/bin/env python3
"""runbook_freshness.py - fail CI when a runbook step outlives its budget."""
import argparse
import datetime as dt
import re
import sys
from pathlib import Path

STEP_RE = re.compile(r"^##\s+Step:\s+(?P<id>[a-z0-9][a-z0-9-]*)\s*$")
META_RE = re.compile(
    r"<!--\s*verified:\s*(?P<date>\d{4}-\d{2}-\d{2})"
    r"\s*\|\s*tier:\s*(?P<tier>[123])"
    r"\s*\|\s*owner:\s*(?P<owner>[A-Za-z0-9_.@-]+)\s*-->"
)
BUDGET_DAYS = {1: 30, 2: 90, 3: 180}


def parse(path: Path) -> list[dict]:
    steps, current = [], None
    for line in path.read_text(encoding="utf-8").splitlines():
        m = STEP_RE.match(line)
        if m:
            current = {"id": m["id"], "verified": None, "tier": None, "owner": None}
            steps.append(current)
            continue
        if current and (meta := META_RE.search(line)):
            current["verified"] = dt.date.fromisoformat(meta["date"])
            current["tier"] = int(meta["tier"])
            current["owner"] = meta["owner"]
    return steps


def audit(steps: list[dict], today: dt.date) -> list[dict]:
    findings = []
    for s in steps:
        if s["verified"] is None:
            findings.append({**s, "state": "missing-meta", "overrun": 10**6})
            continue
        overrun = (today - s["verified"]).days - BUDGET_DAYS[s["tier"]]
        if overrun > 0:
            findings.append({**s, "state": "stale", "overrun": overrun})
    findings.sort(key=lambda f: f["overrun"], reverse=True)
    return findings


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("runbook", type=Path)
    ap.add_argument("--today", type=dt.date.fromisoformat, default=dt.date.today())
    args = ap.parse_args()

    steps = parse(args.runbook)
    if not steps:
        print(f"no '## Step: <id>' headings found in {args.runbook}", file=sys.stderr)
        return 2

    findings = audit(steps, args.today)
    for f in findings:
        print(f"{f['id']:<32} tier={f['tier']} state={f['state']} "
              f"overrun={f['overrun']}d owner={f['owner']}")
    print(f"\n{len(steps)} steps, {len(findings)} need re-verification")
    return 1 if findings else 0


if __name__ == "__main__":
    sys.exit(main())

Pinning the clock makes the test reproducible

Never let the auditor read the wall clock in CI, because then yesterday's green build becomes today's red build with no code change. Pass --today explicitly:

python runbook_freshness.py docs/runbook.md --today 2026-11-05; echo "exit=$?"

Exit code 0 means everything is in budget, 1 means at least one step is stale or missing metadata, and 2 means the parser found no steps at all. That third case matters: a silent zero-step run is how this check quietly stops protecting you.

Treat missing metadata as worse than stale

A stale date is an honest signal. A missing comment means the step was added without a tier assignment, which usually means it was copied from somewhere without thinking about blast radius. Failing on missing-meta with the largest possible overrun forces the author to make that judgment at write time, when they still have the context.

Turn the findings into a worksheet, not a guilt list

A CI failure that only prints step IDs will be ignored within two weeks. What reviewers actually need is the exact block to re-run, so paste it back rather than making them scroll.

def worksheet(runbook: Path, findings: list[dict]) -> str:
    text = runbook.read_text(encoding="utf-8")
    blocks = {}
    for chunk in re.split(r"(?m)^##\s+Step:\s+", text)[1:]:
        blocks[chunk.splitlines()[0].strip()] = chunk
    fence = "`" * 3
    out = []
    for f in findings:
        out.append(f"### [ ] {f['id']} (tier {f['tier']}, {f['overrun']}d over, {f['owner']})")
        out.append("Re-run every command below, then update the verified: date.")
        out.append(fence)
        out.append(blocks.get(f["id"], "(step body not found)"))
        out.append(fence)
    return "\n".join(out)

The ordering is deliberate: sorted by overdue margin, so the worksheet starts with whatever has been neglected longest. Reviewers on rotation are busy, so the top of the page has to be the most dangerous item, not the alphabetically first one.

Where a free model actually helps, and where it absolutely does not

Here is the part where AI is genuinely useful and where it is theater. The useful job is bulk summarization: take the last thirty days of alert titles, deploy events, and change tickets that touch a step's service, and produce a short "what moved" note so the reviewer starts warm instead of cold.

The useless job is letting the model decide whether a rename is semantically safe, or letting it rewrite the command block. I have watched that go wrong in the boring way: the model produces confident prose about a flag it has never seen.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project offers free model access and a free server option, and its operator states that the current free allowance covers the kind of high-volume, low-stakes summarization described here — I would still check the project page for today's terms, because quotas and model lineups move.

Why is summarization a good fit for a free tier specifically? Because the failure mode is cheap. A mediocre summary costs the reviewer one extra glance, whereas a bad command costs an outage. High volume, low blast radius, no state to corrupt.

# NOT RUN. This is the request shape I would start from, not a tested client.
import json, os, urllib.request

def summarize(titles: list[str]) -> str:
    body = json.dumps({
        "model": os.environ["MONKEYCODE_MODEL"],  # whatever the docs list today
        "messages": [
            {"role": "system",
             "content": "Summarize operational change. Do not suggest commands."},
            {"role": "user", "content": "\n".join(titles)},
        ],
        "max_tokens": 300,
    }).encode()
    req = urllib.request.Request(
        os.environ["MONKEYCODE_BASE_URL"].rstrip("/") + "/chat/completions",
        data=body,
        headers={"Authorization": f"Bearer {os.environ['MONKEYCODE_API_KEY']}",
                 "Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.load(r)["choices"][0]["message"]["content"]

I have not executed the block above against a live endpoint, so treat it as a starting shape rather than a working SDK wrapper. Strip or hash anything customer-identifying before it leaves your boundary, and never concatenate raw payload bodies into the prompt.

The scheduled half, and why a free server removes the excuse

The auditor plus summary is a weekly job, not an interactive tool. Previously the reason this never shipped was procurement: someone had to justify a small always-on instance for a script that runs for four seconds a week.

With a free server option, that conversation disappears. A cron entry runs the auditor, generates the worksheet, and files one issue per stale step assigned to the owner named in the comment:

0 6 * * 1  cd /srv/runbook && python runbook_freshness.py docs/runbook.md \
  --today "$(date -I)" > freshness.txt 2>&1 || true

Note the guard rails: no production credentials on that box, read-only access to the runbook repository, and output committed to an issue rather than posted to a channel where it becomes noise.

What this approach does not do

• It does not prove a step still works. Only a human re-run or a scripted dry run does that.
• It cannot flag a step that is correct but incomplete.
• It will never catch a step that nobody wrote down, which is the most common runbook failure of all.
• A determined person can bump the verified: date without executing anything, and the script will believe them.

Who should skip this entirely

Teams with fewer than ten steps should not build a ledger, because the review conversation is cheaper than the tooling. Teams that cannot commit to acting on the output should not either, since a weekly stale report nobody reads is worse than no report. And if alert titles cannot legally leave your environment, keep the auditor and drop the model half without guilt.

If you want to try the summarization half before wiring up a paid endpoint, MonkeyCode's free model access and free server option are enough to run the whole loop end to end — and if the worksheet still goes unread after a month, you have learned something useful about the process rather than the tooling.