I stopped asking the on-call bot for cleverness at two in the morning, and I started asking it for a named owner instead. A restart looks decisive in Slack, but it is just a loud guess when nobody owns the blast radius. My working rule is blunt on purpose: if the packet cannot name a human owner, the bot does not receive mutating tools. Can a language model invent a service owner at three in the morning? Sure it can, and that is exactly why I refuse to let it try.
This write-up is a shadow-mode runbook, not a victory lap with invented graphs or a customer count. I treat last week's pages as the only honest training set I currently trust. Production credentials stay off the classifier host, and the artifact below is a replay harness you can run while you are still awake.
What this runbook is actually for
I keep seeing agent demos jump from a red panel to kubectl rollout restart as if that jump were courage. Is it courage, or is it a missing org chart wearing an incident emoji? Most of my painful nights were not missing a clever command. They were missing a person who could say whether the blast radius was one pod or a whole billing region.
Use this when you already have alerts, a service catalog that is at least half true, and a human who still owns the pager. Do not treat it as a license to auto-remediate a cluster you barely understand. The bot proposes checklists. The owner map decides whether proposing is even allowed tonight.
The packet, minus the poetry
Every alert that enters the shadow lane has to look like a boring JSON object, not a novel. I do not let the model free-type the incident into existence. I fill the fields first, then I ask the model to classify the packet, not to invent an owner, a region, or a restart.
{
"alert_id": "page-2026-09-09-1842",
"service": "checkout-api",
"env": "prod",
"region": "us-east-1",
"deploy_sha": "a1b2c3d",
"symptom": "p99_latency",
"started_at": "2026-09-09T18:42:11Z",
"related_services": ["payments-gw", "cart-cache"],
"page_channel": "#inc-checkout"
}
Notice what is missing on purpose. There is no owner field in the packet, because the owner must come from a map I control. If I let the model fill owner, it will eventually pick a name that looks senior and is currently asleep. Would you want that name in the timeline tomorrow morning?
The owner map is the first command
My first command is not kubectl get and it is not a restart sketch. My first command is lookup_owner, and it reads a committed YAML file that on-call can argue about in daylight. The file is small, mean, and reviewable, which is the point.
# owners.yaml — the only place an unfreeze can happen
services:
checkout-api:
owner: "priya"
backup: "ops-payments"
blast_budget: 2
payments-gw:
owner: "diego"
backup: "ops-payments"
blast_budget: 1
cart-cache:
owner: "sam"
backup: "ops-storefront"
blast_budget: 3
blast_budget is the number of related services the bot may even mention in a proposed command. Why a hard number instead of "be careful"? Because models treat vibes as permission, and vibes do not page anyone. If checkout-api pages with four downstream names attached, a budget of two means inspect checkout-api plus one neighbor, then escalate.
Decision table I actually print
| Owner lookup | Related-service blast | Allowed lane | First human action |
|---|---|---|---|
| Hit, budget holds | <= blast_budget |
Propose read-only checks | Ack in the incident channel |
| Hit, budget exceeded | > blast_budget |
Escalate, no cluster talk | Owner confirms the radius |
| Miss | any | Freeze mutating tools | Merge an owners.yaml change |
| Conflicting owners | any | Freeze, page both backups | Incident commander picks |
If the table feels like bureaucracy, ask a better question before you throw it out. Would you rather debate YAML at eleven in the morning, or debate a surprise restart at three?
Replay last week's pages, not last week's prompt
I do not trust a prompt that I wrote while caffeinated and optimistic. I replay a JSONL file of last week's alerts through a classifier that cannot execute anything on a cluster. The output is a receipt, and the receipt is what I read on Monday with the actual owner sitting nearby.
# replay_pager.py — proposed harness, not a production controller
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any
import yaml
MUTATING_VERBS = ("restart", "rollout", "delete", "scale", "apply", "drain")
@dataclass
class Verdict:
alert_id: str
owner: str | None
lane: str
proposals: list[str]
freeze: bool
reason: str
def load_owners(path: Path) -> dict[str, Any]:
data = yaml.safe_load(path.read_text())
return data["services"]
def score_blast(alert: dict[str, Any], owners: dict[str, Any]) -> int:
related = [alert["service"], *alert.get("related_services", [])]
known = [name for name in related if name in owners]
return len(set(known))
def lookup(alert: dict[str, Any], owners: dict[str, Any]) -> dict[str, Any] | None:
return owners.get(alert["service"])
def propose_readonly(alert: dict[str, Any]) -> list[str]:
svc = alert["service"]
return [
f"metrics.query service={svc} window=15m",
f"logs.search service={svc} level=error deploy={alert.get('deploy_sha')}",
f"deploy.info service={svc} sha={alert.get('deploy_sha')}",
f"channel.post {alert.get('page_channel')} owner-lookup-complete",
]
def classify(alert: dict[str, Any], owners: dict[str, Any]) -> Verdict:
rec = lookup(alert, owners)
blast = score_blast(alert, owners)
if rec is None:
return Verdict(
alert_id=alert["alert_id"],
owner=None,
lane="freeze",
proposals=[],
freeze=True,
reason="owner_map_miss",
)
if blast > rec["blast_budget"]:
return Verdict(
alert_id=alert["alert_id"],
owner=rec["owner"],
lane="escalate",
proposals=[f"page.owner {rec['owner']}", f"page.backup {rec['backup']}"],
freeze=True,
reason="blast_over_budget",
)
return Verdict(
alert_id=alert["alert_id"],
owner=rec["owner"],
lane="propose",
proposals=propose_readonly(alert),
freeze=False,
reason="owner_and_budget_ok",
)
def reject_if_mutating(text: str) -> None:
lowered = text.lower()
for verb in MUTATING_VERBS:
if verb in lowered:
raise ValueError(f"mutating verb {verb!r} is not allowed in shadow replay")
def replay(alerts_path: Path, owners_path: Path, out_path: Path) -> None:
owners = load_owners(owners_path)
receipts: list[dict[str, Any]] = []
for line in alerts_path.read_text().splitlines():
if not line.strip():
continue
alert = json.loads(line)
verdict = classify(alert, owners)
for item in verdict.proposals:
reject_if_mutating(item)
receipts.append(verdict.__dict__)
out_path.write_text(json.dumps(receipts, indent=2))
print(f"wrote {len(receipts)} receipts to {out_path}")
if __name__ == "__main__":
replay(Path("alerts.jsonl"), Path("owners.yaml"), Path("receipts.json"))
Keep a tiny fixture next to the script so Monday you is not tempted to improvise.
{"alert_id":"page-001","service":"checkout-api","deploy_sha":"a1b2c3d","related_services":["payments-gw"],"page_channel":"#inc-checkout"}
{"alert_id":"page-002","service":"mystery-batch","deploy_sha":"deadbeef","related_services":["warehouse-etl"],"page_channel":"#inc-unknown"}
Run the harness when you are not in an incident, then read only the frozen rows first.
python3 replay_pager.py
jq '.[] | select(.freeze==true)' receipts.json
If a receipt contains restart, the harness should throw before the file is written. That is the whole point of shadow mode, and it is not optional. The model can still draft a rationale later, but the verbs are a gate in Python, not a wish inside a system prompt.
Freeze and unfreeze, the boring version
Here is the freeze rule for this lane, and it is not a Slack emoji from a tired primary. When owner_map_miss or blast_over_budget fires, mutating tool credentials stay sealed for that service. Unfreeze is a merged change to owners.yaml, or an explicit budget bump reviewed by the backup owner in the same pull request.
A chat message from the bot does not count as unfreeze, and "looks fine" does not count either. Escalation order is equally boring, which I like, because boring is searchable tomorrow.
- Page the named owner from the map, not a cluster nickname.
- Page the backup alias if there is no ack inside ten minutes.
- Page the incident commander only after both of those paths fail.
- Never page a node pool, a Terraform path, or "whoever last touched prod."
Would I still restart a stuck deployment after that sequence? Sometimes I would, after a human with a name on the map says the blast radius is theirs. The bot does not get to skip that sentence, and neither do I when I am the one holding the pager.
Where a free model actually belongs
I still want a model in the loop, just not in the loop that holds kubectl or cloud keys. The useful job is classification: given a filled packet and the owner map, draft a short rationale and a propose-only checklist. That work can live on a throwaway box that never sees production tokens.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
For that shadow lane I point the classifier at MonkeyCode, which is open source, using its free model access and free server option so the remote box only rewrites redacted packets into receipts. If you try the same split, keep credentials on your side of the wall and let the remote host do language, not actions. Delete that paragraph and the runbook still works, which is how I want the tooling to sit.
A Monday test plan, not a vibe
- Dump last week's pages into
alerts.jsonlwith service, region, and related services filled in. - Commit
owners.yamlwith budgets that feel uncomfortably small on purpose. - Run
replay_pager.pyand count freezes versus proposals without changing verbs. - Read every
blast_over_budgetrow with the actual owner, not with the model. - Only then consider exposing a read-only metrics tool, still without mutate verbs.
If the freeze rate is high, your map is lying, not the script. Fix the map in a normal review. Do not loosen MUTATING_VERBS just to make a dashboard look friendly before the next rotation.
Who should not use this
Skip this approach if you have no service catalog, no named humans, and no appetite for YAML arguments during business hours. Skip it if your compliance team has not approved sending even redacted alert text to a hosted model. Skip it if you need sub-minute auto-remediation, because this workflow is deliberately slow. Skip it if you were hoping the agent would replace the pager. It will not replace the pager. It will make the pager more annoying in a way you can audit.
Limitations I will not wave away
Replay is not production, and last week's alert shape will not match the novel failure you get on a holiday weekend. Owner maps rot when people change teams and nobody updates YAML, which is a social failure the script cannot see. Blast budgets are crude, and two related services can still be more dangerous than ten distant ones. The verb denylist is bypassable with creative phrasing, so keep mutating tools out of the process entirely while you are in shadow mode.
I am not giving you latency numbers, pass rates, or a customer story, because those would be invented for this page. The conclusion stays the same when the tools change next quarter. If you cannot name the owner, you do not run the command. Everything else is a receipt you can read when you are awake enough to argue with YAML.
Top comments (0)