Let a Model Draft Your First Five On-Call Commands — Then Prove They Stay Read-Only

The riskiest lines in a runbook are rarely the paragraphs; they are the first five commands, because one wrong flag turns a page into a second incident. I now treat that command block as untrusted input: a model may draft it, a linter must prove it is read-only, and a replay step must show it still matches the alert payload that fired. The whole loop is one YAML schema plus roughly 120 lines of Python, and you can wire it into CI before your next rotation.

Why the first commands carry the risk

Four failure modes show up in almost every public runbook review, and none of them are about missing knowledge.

• A read-looking command with a write side effect. Something like curl -X POST https://api.internal/health/refresh returns JSON, reads like a probe, and quietly enqueues work at 03:00.
• A leftover mitigation. Someone added a restart command mid-incident because it helped, and it never got demoted into the escalation section afterwards.
• A pipe that executes. A formatting convenience ends in | sh or xargs ... kubectl delete pod, and now the runbook mutates state.
• Stale flags. The command was correct for the previous API version, so it fails in a way that looks like a brand-new incident symptom.

So the fix is not better prose. The fix is a three-stage gate: draft, verify, replay. Anyone can do stage one; stages two and three are what make the runbook safe to hand to a tired human.

A runbook schema that separates drafting from execution

The schema matters more than the model, because the schema decides what a machine can check. Here is the version I would start from:

service: checkout-api
owner: payments-oncall

alerts:
  - name: CheckoutApiHighLatency
    signal: p99_latency_ms > 900 for 10m
    severity: page
  - name: CheckoutApiErrorBudgetBurn
    signal: burn_rate_1h > 14.4
    severity: page

first_commands: |
  date -u
  kubectl get pods -n checkout -l app=checkout-api -o wide
  kubectl logs -n checkout deploy/checkout-api --tail=80 --since=15m
  aws cloudwatch get-metric-statistics --namespace AWS/ELB \
    --metric-name TargetResponseTime --period 60 --statistics p99

escalation:
  - after: 10m
    if: error_rate still above 2%
    to: payments-oncall-secondary
  - after: 25m
    if: no mitigation applied
    to: payments-incident-commander

freeze:
  declared_by: incident-commander
  scope: [deploys, schema_migrations, feature_flags]
  max_duration: 60m
  unfreeze_requires:
    - two consecutive green 5xx SLO windows
    - follow_up_ticket_id present
    - secondary_oncall_ack within 15m

A few deliberate choices. first_commands is a single block string so it can be diffed and linted as one artifact, not scattered across prose. escalation is structured data with after, if and to, so an on-call can read it as a table instead of hunting for sentences. And freeze is data, which brings me to the part most runbooks get wrong.

The freeze rule should be a state, not a sentence

"Freeze deploys during incidents" is not a rule, it is a mood. Record a freeze as an object with a declarer, a scope, an expiry and explicit unfreeze criteria; then a five-line script can answer "are we frozen right now?" without asking a human. Every unfreeze_requires entry must be computable — a green SLO window, a ticket id, an ack timestamp. If a criterion cannot be computed, it is a wish, and wishes are what people argue about at 04:10.

The verifier: fifty lines that say no

The verifier is boring on purpose. It allowlists binaries and subcommands, denies write verbs outright, rejects pipes into shells, and fails closed on anything it does not recognise.

#!/usr/bin/env python3
"""runbook_lint.py - prove the first_commands block is read-only."""
import re
import shlex
import sys
from pathlib import Path

import yaml

ALLOWED_SUBCOMMANDS = {
    "kubectl": {"get", "describe", "logs", "top", "events", "explain", "version", "api-resources"},
    "aws": {"describe", "list", "get"},
    "gcloud": {"describe", "list", "get-iam-policy"},
}

PLAIN_READ_ONLY = {"date", "hostname", "uptime", "dig", "ss", "df", "free"}

DENY_TOKENS = (
    " delete ", " apply ", " patch ", " scale ", " rollout restart ", " rollout undo ",
    " exec ", " cordon ", " drain ", " taint ", " edit ",
)

DENY_SHELL = ("| sh", "| bash", "|sh", "|bash", "xargs ", "eval ", "> /", ">>")


def split_block(block: str):
    out = []
    for raw in block.strip().splitlines():
        line = raw.strip().lstrip("-").strip()
        if not line or line.startswith("#"):
            continue
        for part in re.split(r"&&|\|\|", line):
            if part.strip():
                out.append(part.strip())
    return out


def violations(command: str):
    hits = []
    padded = f" {command} "
    for token in DENY_TOKENS:
        if token in padded:
            hits.append(f"write verb {token.strip()!r}")
    for token in DENY_SHELL:
        if token in command:
            hits.append(f"pipe or redirect {token!r}")
    try:
        argv = shlex.split(command)
    except ValueError as exc:
        return hits + [f"unparsable command: {exc}"]
    if not argv:
        return hits
    binary = Path(argv[0]).name
    if binary in PLAIN_READ_ONLY:
        return hits
    allowed = ALLOWED_SUBCOMMANDS.get(binary)
    if allowed is None:
        return hits + [f"unknown binary {binary!r}: allowlist it or drop the command"]
    sub = argv[1] if len(argv) > 1 else ""
    if not any(sub.startswith(prefix) for prefix in allowed):
        hits.append(f"{binary} {sub!r} is not an allowlisted read-only subcommand")
    return hits


def main(path: str) -> int:
    doc = yaml.safe_load(Path(path).read_text())
    block = doc.get("first_commands", "")
    if not isinstance(block, str) or not block.strip():
        print(f"FAIL {path}: no first_commands block")
        return 1
    failures = 0
    for command in split_block(block):
        for hit in violations(command):
            failures += 1
            print(f"FAIL {command!r}: {hit}")
    print(f"{'PASS' if failures == 0 else 'FAIL'} {path}: {failures} violation(s)")
    return 0 if failures == 0 else 1


if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "runbook.yaml"))

Run it locally, then run it over every runbook in CI:

python -m pip install pyyaml
python runbook_lint.py runbooks/checkout-api.yaml
for f in runbooks/*.yaml; do python runbook_lint.py "$f" || exit 1; done

The exit code is the contract. A failing lint blocks the merge, which means nobody can quietly add a mutation to the page path in a hurry.

Replay the block against a recorded payload

A command that lints clean can still be wrong for today's cluster, so add a replay stage with fixtures. Capture a sanitised alert payload plus the outputs of each read-only command, then assert that the runbook still produces the observations it claims to produce.

1. Store fixtures/CheckoutApiHighLatency/alert.json, pods.txt and logs.txt next to the runbook.
2. Run each command against a dry-run kubeconfig or a recorded namespace, never against production.
3. Assert that the fields your runbook reasons about — pods_ready, error_rate, p99_latency_ms — actually exist in the parsed output.
4. Assign an owner for fixture refresh, because a stale fixture is the same bug as a stale runbook.

Does this sound like a lot of ceremony for five commands? It is about the same effort as one postmortem action item, and it is reusable across every page you own.

Where free model access and a free server actually help

Two parts of this loop are tedious and low-judgement enough to delegate: drafting first_commands from an alert payload plus a service README, and summarising a long dump mid-incident into three observed facts.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project whose operator states that it offers free model access — currently advertised as a 10M-token free allowance — and a free server option, which is enough to run this loop end to end without a paid account on day one. Verify the current terms before you depend on any specific number, because free-tier numbers move, and treat every capability claim as something to re-check rather than something I can promise on their behalf.

Concretely, the free tier fits three jobs in this workflow:

• Drafting. Send the alert name, the signal expression and the list of read-only tools your service exposes, and ask for a command block in the schema above.
• Nightly lint. Run the linter over runbooks/*.yaml on the free server and open an issue when a command starts failing, so drift surfaces on a Tuesday instead of at 03:00.
• Incident note summarising. Paste text you have already read; keep the process environment free of write credentials and long-lived cloud tokens.

The rule I would not bend: model output never reaches a human without passing the linter first. The value is a faster first draft, not an authority, and your on-call should still read each command before running it. If you want to rehearse the drafting half before wiring anything to production, the free model access and the free server option are enough to close the loop.

Decision table: what to delegate

Task	Give it to a model?	Gate before merge
Draft first_commands from payload + README	Yes	runbook_lint.py passes, human reads every line
Choose escalation thresholds	No	SLO owner decides, runbook records the number
Summarise a 40-line output dump mid-incident	Yes	Read-only text in, no credentials in the environment
Rewrite unfreeze_requires	No	Incident commander owns the criteria
Generate a follow-up ticket skeleton	Yes	Human signs the ticket id into the runbook

Limitations and who should not use this

A deny-list is not a sandbox. The linter catches known write verbs and unknown binaries; it cannot understand a custom CLI where status means "apply pending changes". Read-only is also not the same as safe to paste — a get secrets output is read-only and still a credential leak, so keep that class of command out of the block entirely. And if your team has no review habit, this automation mostly produces confident-looking green checkmarks.

You should skip this approach if your runbooks have no machine-checkable command block, if your environment is air-gapped and cannot use a hosted server option, or if you need a formal safety guarantee rather than a guardrail. The technique is a speed bump sized for humans who are awake at the wrong hour.

A twenty-minute adoption checklist

1. Move the first five commands of one page into first_commands as a single block.
2. Copy runbook_lint.py, install PyYAML, and fix whatever it flags.
3. Convert the freeze sentence into a freeze object with an expiry and computable unfreeze criteria.
4. Add the lint call to CI and make it a required check on runbook changes.
5. Capture one fixture set from a real, sanitised alert and add a replay step.
6. Optionally, point a drafting prompt at the free model access and diff its output against your reviewed block.

What would your runbook look like if every command had to survive a linter before it could page anyone? Probably shorter, definitely more readable, and a lot less likely to become the next incident.