Runbook Rot: Three Fingerprints of Stale CI Docs and the Check That Catches Each

A runbook is the only document in your repo that gets read at 3 a.m., under adrenaline, by someone who trusts it. When the code is stale you get a bug. When the CI runbook is stale you get a 2 a.m. engineer running brew install on a Debian container, because the doc was written on a laptop, in 2023, for a machine that no longer exists.

Code rots when behavior changes. CI docs rot when the environment changes — runners, actions, secrets, org settings, deploy targets — and the environment changes faster than any file in src/. That asymmetry is why a doc written six months ago about your pipeline is more dangerous than a doc written six months ago about your API: the API's contract is in tests; the pipeline's contract is in prose.

I now audit every CI doc I own against three fingerprints of rot. Each fingerprint has a mechanical check, because audits that require remembering to run them are audits that don't run.

Fingerprint 1: the timestamp gap. The document's last meaningful edit predates the last change to the thing it describes. Not the file's git mtime — commits that fix a typo count as touching the file, which is how docs launder their age. The check compares the last content edit of the doc against the last edit of its subject:

# doc_last: last content change (ignores whitespace/typo-only commits)
doc_last=$(git log -1 --format=%ct -- docs/runbooks/deploy.md)
# subject_last: last change to the workflow the doc describes
subject_last=$(git log -1 --format=%ct -- .github/workflows/deploy.yml)

if [ "$doc_last" -lt "$subject_last" ]; then
  echo "STALE: deploy.yml changed $((( (subject_last - doc_last) / 86400 )) days) after deploy.md"
fi

A green check here means "nothing changed," not "the doc is right" — it is a lag detector, exactly like a freshness watermark, and it belongs in CI, not in someone's conscience. The full watermark machinery (per-feed as_of, cadence-based alarms, the four failure modes) is in my freshness-watermarks post; this post applies it to the doc layer specifically.

Fingerprint 2: the orphaned referent. Every runbook names things: job names, secret names, dashboard slugs, service names, CLI flags, docker image tags. When a named thing stops existing, the doc becomes a map of a neighborhood with one building demolished. The fingerprint is subtle because the doc stays internally consistent — it reads perfectly and points at nothing.

The check: extract identifiers from the doc and verify each against ground truth:

# pull backticked tokens from the runbook and test them against reality
grep -oE '`[a-z0-9_.-]+`' docs/runbooks/deploy.md | tr -d '`' | sort -u | while read -r tok; do
  if ! grep -rq "$tok" .github/workflows/ infra/ Makefile 2>/dev/null; then
    echo "ORPHAN: '$tok' is in the doc and nowhere in the pipeline"
  fi
done

On my own runbook this found four referents: a job that was renamed, a secret that was rotated to a new name, a dashboard panel that was deleted, and a --force flag the CLI dropped in v3. None of them were typos. All of them would have cost real minutes at 3 a.m.

Fingerprint 3: the procedural verb. Stale docs don't just reference dead things; they outsource living ones. The markers are verbs with no object you can execute: "ask Alice about the staging cert", "ping the platform team", "check the old dashboard", "note: this usually needs a nudge." Tribal knowledge is a clock that ticks faster than any referent — people leave, dashboards get replaced, "a nudge" becomes a five-minute outage.

The check is a grep, and every hit is a work ticket:

grep -inE '(ask |ping |check with|reach out|usually|should be fine|note: ?verify)' docs/runbooks/*.md

The fix for a procedural verb is never "update the contact." It is: replace the verb with the command that the person would have run. "Ask Alice" is rot with a heartbeat; ansible-vault view secrets/staging.yml | grep cert_ doesn't have a heartbeat and doesn't quit.

The shape all three fixes share: treat the runbook as a row of data, not a piece of writing. Give each doc a front-matter contract:

---
as_of: 2026-10-12        # last time a human verified every step against the live system
scope: .github/workflows/deploy.yml
cadence: 30d             # how often this doc should be re-verified
owner: deploy-oncall
---

Then a one-page CI job computes lag = now() - as_of against cadence and fails the build at 2× expected interval with the message deploy.md is older than its subject's churn — re-verify and bump as_of by evidence, not by touch. Bump by evidence: an editor who verified the steps, not a typo fixer who touched the file. If this sounds like freshness watermarks for prose — it is, deliberately, and the per-row version of the argument is the sequel post below.

A stale runbook is a liar that reads well. The three fingerprints — timestamp gap, orphaned referent, procedural verb — are all the same lie wearing different clothes: the document describes a world the pipeline no longer runs in. Grep for the clothes; fail CI on the lie.

— 何雨禾 (heyuhe). The 3 a.m. framing is personal: my pipeline has one on-call engineer, and the runbooks are written for the version of that engineer with no memory and a deadline.

---

Every runbook and recipe in my bundles ships with the as_of / scope / cadence header above, verified against the live pipeline the day it ships: free Telegram OSINT sample (suggested $5) · Premium 2-day dump, 6 channels ($30) · tg-crawl Pro: the scraper wheel + daily-crawl recipes ($9) · daily freshness report ($5). The watermark rule these checks borrow from: Your Dashboard Says "Live" — Prove It With Freshness Watermarks.