The Runbook That Trusted Its Memory: Free AI, Versioned YAML, and a Freeze Rule

Most runbooks rot because nobody owns them, and an AI assistant with a long chat memory makes that rot invisible until the pager fires at 3 AM. Here is my conclusion up front: make the runbook a versioned YAML file, let the model read only that file, and encode a freeze/unfreeze rule in the same artifact. The snippets below form a reference design rather than a production system, and the test plan at the end tells you whether it is safe for yours.

Last week's DEV feed was buzzing about LLM memory that trusts everything and about reviewers that nobody ever tested, and both threads hit the same nerve. A model is confident because it has context, not because the context is true. On-call is where that confidence hurts the most: the bot happily suggests a rollback for a service that retired two incidents ago, and you are too tired to argue.

Why chat memory is the wrong database

A runbook in a wiki is already a liability because it accumulates opinions faster than corrections. A model's conversation history is worse, because it accumulates the same stale facts plus every wrong turn the previous on-caller made. Would you trust a human sysadmin who refused to look at the current deployment manifest and instead recited last quarter's memory? I didn't think so, and yet that is exactly how most of us wire our AI tools.

The rule I now follow is simple: the checked-in YAML is the only source of truth, and the model never sees the conversation history when it answers an incident question. Every lookup starts from disk, and the memory is treated as decoration rather than evidence.

The artifact: a runbook as data

The core of the setup is a single YAML file that describes every alert that matters, plus the first commands and the escalation path. Here is a trimmed example that you can paste into your own repo tonight:

version: 3
owner_team: platform
slack_channel: '#oncall-platform'

alerts:
  - name: HighErrorRate
    severity: P1
    first_commands:
      - 'kubectl get pods -n api -o wide'
      - 'kubectl logs -n api -l app=api --tail=500'
    escalation:
      - after_minutes: 15
        contact: 'primary'
      - after_minutes: 30
        contact: 'platform-eng'
    freeze_required: true

  - name: DbConnectionPoolExhausted
    severity: P2
    first_commands:
      - 'kubectl exec -it -n data primary-db-0 -- patronictl list'
      - 'kubectl get po -n data | grep -i pool'
    escalation:
      - after_minutes: 20
        contact: 'db-owner'
    freeze_required: false

freeze_policy:
  window_minutes: 30
  unfreeze_requires_ack: true

The lookup script

In this reference design, the service loads that file from disk on every request, never from memory, and the query is a one-liner in Python:

import yaml, sys

def load_runbook(path='runbook.yml'):
    with open(path) as f:
        return yaml.safe_load(f)

def lookup(name, runbook):
    for alert in runbook['alerts']:
        if alert['name'].lower() == name.lower():
            return alert
    return None

if __name__ == '__main__':
    entry = lookup(sys.argv[1], load_runbook())
    print(yaml.dump(entry) if entry else 'unknown alert: check runbook.yml')

You can test it immediately with python runbook.py lookup HighErrorRate, and the response changes the moment the file changes. That is the whole point: the artifact is boring, deterministic, and diffable in code review.

The freeze/unfreeze rule nobody writes down

Every team has an implicit freeze during a major incident, but implicit rules get violated at 4 AM. The pattern here is a small state machine that writes exactly one JSON file, and the commands are short enough to type while half asleep:

python freeze.py freeze 'INC-42: error budget exhausted'
python freeze.py status
python freeze.py unfreeze --ack emery

The decision table lives next to the code, and it reads like this:

Condition	State	Action
New P1 alert arrives while frozen	frozen	Queue it, do not auto-apply any command
Deploy requested during freeze	frozen	Reject with the incident ID
Ack received from the incident owner	frozen	Start a 15-minute quiet window
Quiet window elapses without a new alert	unfrozen	Allow deploys and rollbacks again
Runbook entry is older than 30 days	unfrozen	Bot prints a stale warning before any summary

The state file is deliberately boring, a single JSON line with a frozen flag, an incident ID, a timestamp, and an acked boolean. Boring is the point, because this is the one part of the system that a model is not allowed to interpret or override.

Where a free model and a free server fit

To make the setup usable in the heat of the moment, the design adds an AI summary step that reads the matched YAML entry and the metric payload, then produces three bullet points of context. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that, at this writing, advertises free model access with a 10M token allowance and a free server tier for small services; quotas and terms change, so check the repository README before you rely on them.

What I actually let the model do

The workflow is short: the model compresses the YAML and the current alert payload into a two-line summary, while the state machine still decides the freeze. The model never decides anything, it only summarizes, and the lookup endpoint can be hosted on MonkeyCode's free server tier so the team gets a stable URL without paying for a cloud function.

A five-minute test plan

Try this before you trust the setup in anger, and treat each step as a pass/fail gate:

1. Delete one first_commands entry from runbook.yml, commit, and run the lookup again — the response must change immediately.
2. Add a fake alert with a 40-day-old comment, then query it — the service should print the stale warning instead of a confident summary.
3. Run freeze.py freeze 'TEST' and confirm that a pretend deploy command gets rejected with the incident ID.
4. Run freeze.py unfreeze --ack <yourname> and confirm that the quiet window resets to 15 minutes.
5. Ask the model for the same alert summary twice in one conversation — the output must be identical, because it only ever sees the current file.

If any step fails, the wiring is wrong, and would you really trust it during a real incident? I would fix the wiring before the next pager.

Limitations and who should skip this

This approach is a runbook assistant, not an alerting platform, and it will not replace PagerDuty, Grafana, or a human brain. The free tier is great for a weekend prototype, but do not plan a production SLA around a free allowance that can change. The local JSON state file also assumes one team and one region, and it falls apart if two incident commanders freeze and unfreeze concurrently. If your team already has solid runbook automation with enforced change windows, this setup adds little, and if you do not have incident discipline yet, no YAML file will create it for you.

The next time the pager drags you out of bed, the worst question is 'what exactly did we decide last month?' Put the answer in a file, freeze it during the fire, and let the model summarize the file, not your memory. If that workflow sounds useful, MonkeyCode's free allowance is a cheap way to try it tonight.