An on-call page is not a conversation, and it is not a license to improvise a restart. I want every alert to compile into a frozen command plan before anyone opens a shell. The plan names the read-only first commands, the escalate clock, and the freeze latch that blocks writes. If a coding assistant cannot live inside that latch, it does not belong on the pager.
Why chat-shaped runbooks fail at 3 a.m.
Most on-call wikis still read like essays, and most assistants still emit paragraphs that look dangerously executable. Have you ever watched a tired human paste a generated kubectl line because the graph was red? I stop trusting prose as the pager interface when the only artifact is a chat transcript. The page should become a small graph of allowed argv, not another prompt waiting for courage.
A useful plan has three properties that a chat window never actually guarantees at three in the morning. First, every node is either a read, an escalate signal, or a write sitting behind a latch. Second, writes carry a blast-radius field that a human must type, and a model must never fill. Third, the argv is hashed so a small assistant edit cannot sneak into the production shell.
I am not claiming this compiler replaces your paging vendor, your chat bridge, or your existing service catalog. I am proposing a boring gate that sits between alert JSON and a terminal, then refuses anything it cannot name. If that sounds rigid, ask what “flexibility” meant the last time somebody bounced the wrong Deployment because a paragraph looked confident.
Compile the page, do not narrate it
Treat the incoming alert as source code that either type-checks or does not run. If required fields are missing, the compiler fails closed and prints only a read of the raw payload. If the alert class is known, it emits a plan file and refuses any command whose digest is not listed. That is the entire trick, and I want it boring enough that a new on-call engineer can follow it without a pep talk.
Here is the on-disk shape I want committed next to the service, not pasted into a wiki comment at incident time.
# proposed example — unexecuted, not production config
alert_class: checkout_latency
observe_seconds: 180
escalate_after_seconds: 600
read_lane:
- id: ledger
argv: ["python3", "incident_ledger.py", "append", "--alert-id", "${ALERT_ID}"]
- id: p95
argv: ["python3", "read_slo.py", "--service", "checkout", "--window", "15m"]
- id: owners
argv: ["python3", "owners.py", "--service", "checkout"]
write_lane:
- id: shed_traffic
argv: ["python3", "shed.py", "--service", "checkout", "--percent", "10"]
blast_radius: "checkout, 10 percent synthetic and canary, 5 minutes"
freeze: locked
Notice the first command is not a restart, a rollback, or a cluster-wide scale event of any kind. It is an append-only ledger write that lives on a path you already control, so the next person on the bridge can see what already ran. Why start there, instead of another dashboard screenshot that dies in scrollback? Because a ledger survives the chat window, and a screenshot does not explain argv.
Artifact: a local latch that hashes argv
The Python below is a proposed local gate, not a battle-tested control plane, and it does not call exec on purpose. It compiles the YAML, prints the read lane, and blocks writes until a human unfreezes with a blast-radius string that must match the plan. Run it on a laptop or a jump box you already trust, and keep production credentials out of its environment until a dry-run review says otherwise.
#!/usr/bin/env python3
"""proposed_oncall_latch.py — example compiler, not a production pager."""
from __future__ import annotations
import hashlib
import json
import shlex
import sys
import time
from pathlib import Path
import yaml # proposed example dependency
LEDGER = Path("/tmp/incident_ledger.jsonl")
STATE = Path("/tmp/incident_latch_state.json")
def digest(argv: list[str]) -> str:
blob = "\0".join(argv).encode("utf-8")
return hashlib.sha256(blob).hexdigest()[:16]
def load_plan(path: Path) -> dict:
plan = yaml.safe_load(path.read_text())
required = ("alert_class", "read_lane", "write_lane", "escalate_after_seconds")
missing = [key for key in required if key not in plan]
if missing:
raise SystemExit(f"plan missing fields: {missing}")
return plan
def append_ledger(event: dict) -> None:
event = {**event, "ts": time.time()}
with LEDGER.open("a", encoding="utf-8") as handle:
handle.write(json.dumps(event) + "\n")
def save_state(state: dict) -> None:
STATE.write_text(json.dumps(state, indent=2))
def load_state() -> dict:
if not STATE.exists():
return {"freeze": "locked", "started_at": time.time(), "ran": []}
return json.loads(STATE.read_text())
def print_plan(plan: dict) -> None:
print(f"class={plan['alert_class']} freeze={load_state()['freeze']}")
print("read_lane:")
for step in plan["read_lane"]:
print(f" [{digest(step['argv'])}] {shlex.join(step['argv'])}")
print("write_lane (blocked while freeze=locked):")
for step in plan["write_lane"]:
print(f" [{digest(step['argv'])}] {shlex.join(step['argv'])}")
print(f" blast_radius: {step['blast_radius']}")
def run_step(plan: dict, step_id: str, freeze_token: str | None) -> None:
state = load_state()
lanes = {step["id"]: step for step in plan["read_lane"] + plan["write_lane"]}
if step_id not in lanes:
raise SystemExit("unknown step id; I will not guess a command")
step = lanes[step_id]
is_write = step_id in {item["id"] for item in plan["write_lane"]}
if is_write and state["freeze"] != "open":
raise SystemExit("writes are frozen; unfreeze with a matching blast-radius string")
if is_write and freeze_token != step["blast_radius"]:
raise SystemExit("blast radius did not match the plan; still frozen")
append_ledger({"step": step_id, "argv": step["argv"], "digest": digest(step["argv"])})
print("LEDGER_OK", shlex.join(step["argv"]))
print("This example does not exec(). Wire subprocess only after a dry-run review.")
state["ran"].append(step_id)
save_state(state)
def unfreeze(reason: str) -> None:
if len(reason.strip()) < 12:
raise SystemExit("unfreeze reason is too short to be a blast-radius statement")
state = load_state()
state["freeze"] = "open"
state["unfreeze_reason"] = reason
save_state(state)
append_ledger({"event": "unfreeze", "reason": reason})
print("latch open; writes still must match hashed argv")
def maybe_escalate(plan: dict) -> None:
state = load_state()
elapsed = time.time() - state["started_at"]
if elapsed >= plan["escalate_after_seconds"] and "escalated" not in state:
print("ESCALATE: clock exceeded; page the service owner, do not restart yet")
append_ledger({"event": "escalate", "elapsed": elapsed})
state["escalated"] = True
save_state(state)
def main(argv: list[str]) -> None:
if len(argv) < 3:
raise SystemExit("usage: proposed_oncall_latch.py PLAN.yaml print|run|unfreeze|clock [args]")
plan = load_plan(Path(argv[1]))
cmd = argv[2]
if cmd == "print":
print_plan(plan)
elif cmd == "run":
token = argv[4] if len(argv) > 4 else None
run_step(plan, argv[3], token)
elif cmd == "unfreeze":
unfreeze(" ".join(argv[3:]))
elif cmd == "clock":
maybe_escalate(plan)
else:
raise SystemExit("unknown verb")
if __name__ == "__main__":
main(sys.argv)
Would I ship this file tomorrow as the global pager runtime for every service in a company? I would not, because a /tmp ledger is a teaching artifact and not an audit store. I would keep subprocess disabled until the YAML lives in the same repo as the service, and until a teammate can reject a plan in review the same way they reject a bad migration. The point of the artifact is the gate: print, hash, freeze, escalate, then maybe write.
A dry-run you can type without touching prod
Use the verbs in this order, and stop at the first unexpected exit. The failing write is the success case, because it proves the latch still holds.
# proposed dry-run only — do not point this at production kube contexts
python3 proposed_oncall_latch.py checkout.yaml print
python3 proposed_oncall_latch.py checkout.yaml run ledger
python3 proposed_oncall_latch.py checkout.yaml run p95
python3 proposed_oncall_latch.py checkout.yaml clock
# expected: writes are frozen
python3 proposed_oncall_latch.py checkout.yaml run shed_traffic
python3 proposed_oncall_latch.py checkout.yaml unfreeze \
"checkout, 10 percent synthetic and canary, 5 minutes"
# still requires the blast-radius string as the extra argument
python3 proposed_oncall_latch.py checkout.yaml run shed_traffic \
"checkout, 10 percent synthetic and canary, 5 minutes"
If print already looks wrong, you do not have an incident yet; you have a bad plan file. Fix the YAML in daylight, not while the error budget is burning.
First commands I want printed, in order
When the page fires, I want the human to run three reads and nothing that mutates customer traffic. Can a model suggest extra cluster soup in the sidebar while those reads run? It can talk, but that soup does not get a digest, so the latch should refuse it. The allowed first commands are the ones already compiled, and they should be dull.
- Append the alert identifier, the raw class, and the page time to the local ledger so the bridge does not depend on screenshot memory.
- Read the SLO window that the alert claims was breached, using the service name from the plan rather than a random dashboard you like.
- Print the service owner and the escalate path, even when you are sure you already know who owns checkout this week.
- Look at the clock next, and if you are still inside the observe window, leave the freeze latch locked on purpose.
After those steps, restart is still a write, so it stays behind the latch with every other mutation. If that feels slow, ask whether the last surprise bounce was actually faster once you counted the second incident it created.
Escalation is a command, not a feeling
People escalate when they feel scared, or they never escalate because a restart looks like courage on a status page. I want the compiler to print ESCALATE when the clock says so, and I want that line in the ledger where later readers can find it. The human can still call someone earlier, and that is fine. The bot cannot suppress the clock because a generated paragraph said “looks like GC, just bounce it.”
A short decision table keeps that argument out of the bridge chat, where feelings usually win.
| Clock | Freeze | Allowed next step |
|---|---|---|
| t < observe_seconds | locked |
read_lane only |
| observe_seconds ≤ t < escalate_after_seconds | locked |
read_lane, draft the unfreeze sentence, still no writes |
| t ≥ escalate_after_seconds | locked | page the owner, record ESCALATE, still no writes |
| any t | open |
write_lane whose digest and blast_radius both match |
If your team cannot fill that table for an alert class, the class is not ready for automation of any kind. Leave it as a human page, and spend the afternoon naming reads before you name writes. Have you noticed how many “AI on-call agents” skip this table and jump straight to a shell? That skip is the incident.
The freeze / unfreeze rule on a sticky note
Here is the rule I want on paper next to the keyboard, not buried in a tool’s system prompt. Writes stay frozen until a human types the blast-radius sentence that already lives in the plan. The sentence is not a vibe, and it is not lgtm, and it is not a thumbs-up emoji from the bridge. It names the service, the percentage or the blast, and the time box.
Unfreeze does not run the write. Unfreeze only opens the latch, which is a different verb on purpose. The write still has to match the hashed argv, so a rewritten flag is a different command and the gate stays closed. That double door is the difference between a runbook and a chatbot that happens to know kubectl.
- If the assistant rewrites a flag, the digest changes, and the latch must refuse the new argv.
- If the human cannot name blast radius in the plan’s own words, the latch stays closed and the clock keeps running.
- If the alert class has no
write_laneat all, there is nothing to unfreeze, and that absence is a feature you should keep. - If someone SSHs around the gate, the ledger will not save you; this compiler is a seatbelt, not a lock on the building.
Where a coding assistant is allowed to sit
I still want help drafting YAML comments and reviewing whether a new alert class forgot a ledger step before the plan is merged. I do not want help emitting production argv from a live page, and I do not want an assistant that can unfreeze itself. That split is the whole workflow: models annotate the plan in daylight, humans hold the latch at night.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
When I am editing the plan offline, MonkeyCode’s free model access and free server option are enough to iterate on the compiler and the YAML without borrowing a production box. The assistant can propose a new read step in a comment, or it can complain that blast_radius is too vague to type under stress. It cannot obtain an unfreeze token, and it cannot skip the digest check, which is exactly why I am willing to put it next to a runbook at all. If you already have a scratch machine, keep using it; the latch does not require that product, and the method still works when every MonkeyCode mention is deleted.
Limitations, and who should not use this
This compiler does not talk to your paging vendor, your cluster, or your secret store, and it will not magically complete an alert that arrived half-empty. It will not stop a human who SSHs around the gate with a personal kubecontext. It will not detect a bad YAML that names the wrong service inside blast_radius while keeping a pretty digest. Garbage in still means fail closed, and fail closed still means you read the raw payload instead of guessing a write.
Do not use this approach if you are already inside an active incident and you have no plan file checked in yet. Do not use it to grant a model production credentials, even on a “temporary” jump host. Do not use it as a compliance control, because a JSON file under /tmp is not an audit system and I will not pretend otherwise. Do not use it if your writes are not enumerable in advance; a free-form “fix it” agent cannot be hashed, and an unhashable write does not belong on this pager.
I would also skip this pattern if the team cannot agree on observe and escalate clocks during a calm review. A latch without a clock becomes another wiki page that nobody opens, and a clock without a latch becomes a restart contest. If both arguments are still open, you need a conversation, not a Python file.
Keep the runbook smaller than the chat window
Print the plan. Hash the argv. Keep writes frozen until a human names blast radius in the plan’s own sentence. Escalate on a clock, not on a generated paragraph that sounds sure of itself. That is the on-call runbook I want sitting beside the pager, and it is intentionally smaller than the assistant window that will try to eat it.
If you try the example, run print and clock long before you ever wire subprocess.run. The first successful night is the one where the ledger shows three reads, one escalate, and a phone call, not a surprise restart that nobody can replay.
Top comments (0)