The Alert Renamed a Label, the Runbook Rendered Blank: Contract-Testing Payloads Before 3am

Your runbook rarely fails because somebody wrote the wrong command. It fails because the alert that fires it quietly stopped carrying the label that command depends on, and nobody noticed until a template rendered an empty string during an incident. I want that discovery to happen in a CI log at 2pm, not in a terminal at 3am with six people watching. So the artifact below is a payload contract test: assert the labels, render the commands, resolve the owner, and refuse to emit anything that mutates while a freeze is active.

The quiet break: a template that renders to nothing

Most runbooks I read are Markdown with a YAML header, and the commands inside are string templates over alert labels. That design is good, because it keeps one runbook for every service instead of forty near-copies. It is also fragile, because a template has no idea whether the label it references still exists in the alert rule.

# runbooks/checkout-error-rate.yaml
alert: CheckoutErrorRateHigh
first_commands:
  - name: pod health
    cmd: "kubectl -n {{ $labels.namespace }} get pods -l app={{ $labels.service }}"
    class: read
  - name: recent deploys
    cmd: "kubectl -n {{ $labels.namespace }} rollout history deploy/{{ $labels.service }}"
    class: read
escalation:
  owner: "{{ $labels.owner }}"

Now let someone rename namespace to k8s_namespace in the rule file during a refactor. The rule still evaluates, the alert still pages, and the test suite around the alerting rules still passes, because no test ever asked what the runbook needs. The command renders as kubectl -n get pods -l app=checkout, and an empty namespace flag usually collapses to whatever your current kubeconfig context points at. Is that a good way to find out which cluster you are actually talking to?

Three things the runbook actually needs from the alert

I reduce the contract to three assertions, because every other check I have tried was either unenforceable or theatre.

1. Identity labels — the values that make a command target a specific workload (service, namespace, cluster).
2. Routing — an owner or team label that resolves to a real rotation, not a dead alias.
3. Action text — a non-empty first command per severity, so the page is executable rather than decorative.

Everything below is in service of those three, and each check fails the build rather than warning.

Check 1: assert the label set in a rule unit test

If you already use Prometheus-compatible rules, promtool can pin the exact label and annotation set an alert emits. The test file feeds synthetic series, evaluates the rule, and compares the emitted alert to what you declared.

# tests/checkout.alerts.test.yaml
rule_files:
  - rules/checkout.alerts.yaml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout",code="500"}'
        values: '0+10x20'
      - series: 'http_requests_total{job="checkout",code="200"}'
        values: '0+100x20'
    alert_rule_test:
      - eval_time: 12m
        alertname: CheckoutErrorRateHigh
        exp_alerts:
          - exp_labels:
              severity: page
              service: checkout
              namespace: checkout-prod
              owner: payments-oncall
              runbook: https://runbooks.example.com/checkout-error-rate
            exp_annotations:
              summary: "checkout 5xx above 5%"
              first_command: "kubectl -n checkout-prod get pods -l app=checkout"

promtool test rules tests/checkout.alerts.test.yaml

When someone drops or renames a label, this goes red in the same PR that would have broken the runbook. If your alerting stack is not Prometheus-shaped, do not port promtool; instead export one alert payload per alert name as JSON and assert the same key set with a fifteen-line script. The tool matters less than the habit of treating the payload as an interface.

Check 2: render every command against a captured payload

The unit test proves the alert can emit the labels. A renderer proves the runbook actually consumes them without producing empty strings, and it is the check I trust most because it exercises the exact string a human will paste.

#!/usr/bin/env python3
"""Render a runbook's first commands against a captured alert payload."""
import json
import re
import sys
from pathlib import Path

import yaml

LABEL = re.compile(r"\{\{\s*\$labels\.(\w+)\s*\}\}")


class MissingLabel(Exception):
    pass


def render(template: str, labels: dict) -> str:
    def sub(match: re.Match) -> str:
        key = match.group(1)
        value = labels.get(key)
        if not value:
            raise MissingLabel(f"$labels.{key} is empty in {template!r}")
        return value

    return LABEL.sub(sub, template)


def main(manifest: Path, payload: Path, freeze: Path | None) -> int:
    spec = yaml.safe_load(manifest.read_text())
    labels = json.loads(payload.read_text())["labels"]
    frozen = bool(freeze and json.loads(freeze.read_text()).get("active"))
    failures = 0

    for command in spec["first_commands"]:
        if frozen and command["class"] != "read":
            print(f"SKIP (freeze active): {command['name']}")
            continue
        try:
            rendered = render(command["cmd"], labels)
        except MissingLabel as exc:
            print(f"FAIL {command['name']}: {exc}")
            failures += 1
            continue
        print(f"OK   {command['name']}: {rendered}")

    return 1 if failures else 0


if __name__ == "__main__":
    args = sys.argv[1:]
    sys.exit(main(Path(args[0]), Path(args[1]), Path(args[2]) if len(args) > 2 else None))

Capture one fixture per alert from a real non-incident delivery, redact anything sensitive, and commit it:

{
  "labels": {
    "alertname": "CheckoutErrorRateHigh",
    "service": "checkout",
    "namespace": "checkout-prod",
    "owner": "payments-oncall"
  }
}

python tools/render_runbook.py runbooks/checkout-error-rate.yaml fixtures/checkout.json

The class field is small but load-bearing: read commands render during a freeze, anything else is skipped with an explanatory line. That is the whole freeze rule — no fake clock, no simulation framework — just a renderer that will not hand you a mutating command while the freeze file says the account is locked. You still need a human decision about when the freeze ends; the script only refuses to pretend it does not exist.

Check 3: escalation owners must resolve

A runbook that pages the wrong rotation is worse than no runbook, because it manufactures the belief that someone is coming. Keep an owners.yaml next to your runbooks and fail when a fixture references an owner that is not in it.

#!/usr/bin/env bash
set -euo pipefail
missing=0
for owner in $(jq -r '.labels.owner' fixtures/*.json | sort -u); do
  if ! grep -qx -- "- ${owner}" owners.yaml; then
    echo "unroutable owner: ${owner}" >&2
    missing=1
  fi
done
exit "${missing}"

Check	What it proves	Fails the build when	Cost
promtool rule test	The alert emits the promised label set	A label or annotation is renamed or dropped	Minutes per alert
Render pass	The runbook consumes labels safely	A template interpolation is empty	Minutes per command
Owner resolution	Escalation reaches a real rotation	An owner alias is stale or misspelled	One script
Freeze flag	No mutating command appears mid-freeze	class is missing on a command	One if-statement

Where the check runs when nobody's laptop is awake

None of this needs a laptop. The whole job is a shell script, a Python file, and a promtool binary, so it fits on almost anything that can run a scheduled container. One option I have been pointing people at is MonkeyCode's free server option, which is enough for a nightly render of a handful of runbooks, paired with its free model access for the tedious first pass of drafting fixtures and candidate first commands.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use the model for the boring part only — proposing label keys and a first read-only command per alert — then review every line by hand before it lands in a repo. Free tiers and quotas change often, so check the current limits on the project's own page instead of trusting any number quoted in a blog post, including this one. And never let a generated command with class: mutate merge without a named human reviewer attached to the PR.

Who should skip this

• Teams with fewer than five alerts. A checklist in your head is cheaper than a renderer, and honesty about scale beats cargo-culting a pipeline.
• Stacks where rules are not machine-readable. If alerts live in a GUI and cannot be exported, start there; this contract needs a file to test.
• Anyone without an owner file. Check 3 is the one that keeps people safe, and it is also the one that requires organisational agreement, not code.
• Runbooks with no commands. If yours is prose only, fix that first; a renderer over zero commands is a very well-tested no-op.

What I would do this week

1. Pick the alert that scared you most recently and save one real payload as a fixture.
2. Add class: read or class: mutate to each existing command, and delete any command you cannot classify.
3. Run the renderer locally and fix every empty interpolation it reports.
4. Wire the three checks into CI and make the nightly job fail loudly, not silently.

If a label rename can break your first five commands, how confident are you that the next refactor will be the polite kind? Start with one alert and one fixture, and let the runbook prove it still works before the pager has to ask.