T+0 Is Observe-Only: Bind Alert Classes to a Clock and a Command Budget

The on-call bot should not improvise while the pager is screaming at three in the morning. I treat every page as a clock, an alert class, and a shrinking command budget. Until a human acknowledges the incident, the write lane stays closed on every mutating command. Why would we let a model negotiate production access during the noisiest fifteen minutes of the week?

This runbook is a state machine I can diff in git, not a prompt I hope the model remembers. Agents that just figure it out will assume a root cause, then reach for restart as the shortest verb. Have you watched an assistant invent confidence while the graphs are still loading in another tab? I would rather ship a boring incident clock tonight than debug a clever shell tomorrow morning.

What this runbook actually decides

I answer four questions before any process on a box is touched tonight.

1. Which alert class opened the incident, and is that class even in the allowlist?
2. What time is it on the incident clock, measured from the first accepted page?
3. Which read-only probes are still inside the remaining command budget?
4. Is the write lane frozen by clock expiry, budget exhaustion, or a shift handoff?

If any answer is missing, the bot pages a human and waits without improvising a fix. The bot does not get a consolation prize of just one restart against production hosts. Escalation is a first-class action in this runbook file, not a failure of nerve. Would you bet the checkout path on a model that still has not read deploy history?

The incident clock

I open a clock at T+0 when the first well-formed page is accepted, not when a model starts talking. T+0 through T+5 is observe-only, which means metrics, logs, and deploy history, never restarts. T+5 through T+10 allows the remaining read-only probes that the alert class declared in advance. After T+10 without a human ACK, I escalate and freeze writes even if someone unfroze them earlier.

Does a loud P1 severity badge change the incident clock for mutating commands at all? I stay stubborn here, because severity tells me who to wake, not which kill switch to flip during the storm. A screaming pager is a reason to read faster, not a reason to grant kubectl delete. The badge still matters for who gets the phone call, and that mapping lives beside the clock, not inside a prompt.

Clock states I actually encode

• observe: T+0 to T+5, read-only probes only, and any unfreeze request is ignored
• diagnose: T+5 to T+10, still read-only, while the budget ticks down per command
• escalate: T+10 with no ACK, page the secondary, and freeze the write lane
• acked: a human owns the incident, so a scoped unfreeze may be requested
• frozen: clock expired, budget spent, or the pager changed hands mid-incident
• closed: incident ended, receipts archived, and the clock is destroyed on purpose

Bind the page to an alert class first

I refuse to let the model classify the outage in real time under pager noise. Each page must carry alert_class, service, correlation_id, and a timestamp I can parse without guessing. Missing fields do not get imputed from the title of the Slack thread during the storm. If the class is unknown, the bot stops, because unknown is not a synonym for try restart.

Deduping the storm before T+0 starts

Alert storms lie, and they will lie to an agent with a shell faster than they lie to you. I group pages that share correlation_id and alert_class inside a two-minute window into one incident. The clock starts once, on the first accepted member of that group, not on every echo. If a later page arrives with a new class, I open a sibling incident instead of widening the first one.

Why merge a node failure into a checkout latency clock just because they landed in the same minute? Sibling incidents can share a Slack thread and still keep separate budgets, clocks, and write lists. The bot is allowed to mention the sibling in a receipt, not to borrow its commands. That keeps a noisy kube event from unlocking a checkout flag flip.

Here is the YAML I keep next to the service, not inside a chat prompt that can drift.

# runbook/incident_clock.yaml
version: 1
clock:
  observe_until_s: 300
  diagnose_until_s: 600
  auto_escalate_s: 600
  unfreeze_ttl_s: 900
budget:
  max_reads: 6
  max_writes: 1
  max_retries_per_command: 1
shift_handoff:
  freeze_writes: true
  require_incoming_ack: true
alert_classes:
  checkout_latency:
    service: checkout
    first_commands:
      - id: p95
        argv: ["promql", "histogram_quantile(0.95, checkout_latency_bucket)"]
        mutates: false
      - id: deploys
        argv: ["deploylog", "checkout", "--since", "90m"]
        mutates: false
      - id: errors
        argv: ["logql", "count_over_time({app=checkout} |= `5xx` [15m])"]
        mutates: false
    escalate:
      page: ["checkout-secondary", "payments-owner"]
      reason: "clock_expired_or_budget_spent"
    writes_after_ack:
      - id: canary_toggle
        argv: ["featureflag", "checkout", "canary", "off"]
        mutates: true
        requires_unfreeze: true
  kube_node_not_ready:
    service: platform
    first_commands:
      - id: node_status
        argv: ["kubectl", "get", "nodes", "--no-headers"]
        mutates: false
      - id: last_cordon
        argv: ["auditlog", "search", "cordon", "--since", "2h"]
        mutates: false
    writes_after_ack: []

Notice the kube class has an empty write list on purpose, and I want that emptiness to be loud. Not every page deserves a mutation, even after a human finally shows up. If the class cannot name a write, the bot cannot invent one later during the call. That is the whole point of binding classes before the incident starts.

First commands are probes, not fixes

The first commands are boring on purpose, and I like them that way at three. They read p95, recent deploys, error counts, node status, and audit history without touching traffic. They never restart, never cordon, never flip a flag, and never drain a node. If a command is not in first_commands for that class, the executor returns denied and spends nothing.

I also stamp a receipt after every probe, because memory is not an audit log I can hand to the secondary. The receipt carries the clock time, the remaining budget, and the exact argv that ran. When the secondary asks what the bot already did, I paste the receipt file instead of a vibe. Would you rather debug from that file or from a model summarizing its own courage?

# /var/oncall/receipts/inc-20260909-0142.jsonl
{"t_plus_s": 12, "cmd": "p95", "mutates": false, "exit": 0, "budget_reads_left": 5}
{"t_plus_s": 41, "cmd": "deploys", "mutates": false, "exit": 0, "budget_reads_left": 4}
{"t_plus_s": 88, "cmd": "errors", "mutates": false, "exit": 0, "budget_reads_left": 3}

The executor compares argv byte for byte against the YAML, and it rejects paraphrases from the model. Equivalent kubectl is how nodes disappear while everybody is still reading graphs in another window. If the assistant wants a new probe, it opens a pull request against the runbook, not a shell. Can your current bot prove it did not rewrite the command under pressure?

Escalation is cheaper than a clever write

At T+10 without ACK, I page the secondary and freeze writes, even if graphs look obvious. The bot may keep reading if budget remains, but it may not spend the write slot tonight. Escalation is not an insult to the primary sitting in the channel. It is how I keep a tired person from rubber-stamping a restart the model already drafted.

I escalate on three triggers, and I log which one fired in the receipt stream.

1. Clock hit auto_escalate_s with no human ACK in the incident record.
2. Read budget hit zero before a coherent picture showed up in receipts.
3. The pager rotated, because a new human has not accepted write risk yet.

That third trigger is the one people forget when they celebrate handoff GIFs in the channel. Shift change is a freeze event, not a courtesy Slack message in the incident thread. The incoming engineer inherits the clock and the receipts, not an open write lane. Do you currently freeze writes when the rotation bot posts that Bob is up?

Freeze and unfreeze without a chat vote

Unfreeze is not a thumbs-up emoji in the incident channel, and I will not parse emoji. I require a scoped record: incident id, service, single command id, expiry, and the on-call identity. The unfreeze dies when the TTL ends, when the budget's one write is spent, or when the shift hands off. A freeze always wins a race with an unfreeze, and the executor checks freeze first on purpose.

The unfreeze record

{
  "incident_id": "inc-20260909-0142",
  "service": "checkout",
  "command_id": "canary_toggle",
  "expires_at": 1780000000,
  "actor": "emery.oncall"
}

That record cannot say command_id: "*" and it cannot outlive unfreeze_ttl_s in the YAML. If the human wants a second write, they spend a new ACK path and a new budget, which this file refuses. I would rather reopen the discussion than silently refill a kill switch after a canary toggle. Is your unfreeze currently a channel mood, or a record another engineer can replay?

# incident_clock.py — proposal / policy sketch, not production-hardened
from __future__ import annotations
import time
from dataclasses import dataclass
from typing import Literal

State = Literal["observe", "diagnose", "escalate", "acked", "frozen", "closed"]

@dataclass
class Unfreeze:
    incident_id: str
    service: str
    command_id: str
    expires_at: float
    actor: str

@dataclass
class Incident:
    id: str
    alert_class: str
    service: str
    t0: float
    acked_by: str | None
    frozen: bool
    reads_used: int
    writes_used: int
    shift_id: str

def clock_state(inc: Incident, now: float, cfg: dict) -> State:
    if inc.frozen:
        return "frozen"
    elapsed = now - inc.t0
    if inc.acked_by:
        return "acked"
    if elapsed >= cfg["clock"]["auto_escalate_s"]:
        return "escalate"
    if elapsed >= cfg["clock"]["observe_until_s"]:
        return "diagnose"
    return "observe"

def first_ids(cfg: dict, alert_class: str) -> set[str]:
    return {c["id"] for c in cfg["alert_classes"][alert_class]["first_commands"]}

def freeze_reasons(inc: Incident, now: float, cfg: dict, current_shift: str) -> list[str]:
    reasons = []
    st = clock_state(inc, now, cfg)
    if st in {"escalate", "frozen"}:
        reasons.append("clock_or_flag")
    if inc.reads_used >= cfg["budget"]["max_reads"]:
        reasons.append("read_budget")
    if inc.writes_used >= cfg["budget"]["max_writes"]:
        reasons.append("write_budget")
    if current_shift != inc.shift_id:
        reasons.append("shift_handoff")
    return reasons

def unfreeze_ok(inc: Incident, cmd: dict, now: float, u: Unfreeze | None) -> bool:
    if u is None:
        return False
    return (
        u.incident_id == inc.id
        and u.service == inc.service
        and u.command_id == cmd["id"]
        and u.expires_at > now
        and bool(u.actor)
    )

def allow(inc, cmd, now, cfg, unfreeze, current_shift) -> tuple[bool, str]:
    reasons = freeze_reasons(inc, now, cfg, current_shift)
    if reasons and cmd["mutates"]:
        return False, f"frozen:{','.join(reasons)}"
    st = clock_state(inc, now, cfg)
    if not cmd["mutates"]:
        if inc.reads_used >= cfg["budget"]["max_reads"]:
            return False, "read_budget"
        if st == "observe" and cmd["id"] not in first_ids(cfg, inc.alert_class):
            return False, "not_in_first_commands"
        return True, "read_ok"
    if st != "acked":
        return False, "writes_require_ack"
    if not unfreeze_ok(inc, cmd, now, unfreeze):
        return False, "unfreeze_missing_or_expired"
    return True, "write_ok"

Is this glamorous agentic AI with a memory of the company wiki? No, and that is the feature I want. Most of the danger is a model assuming a cause, then picking a verb that sounds like ownership. The clock does not care how confident the summary sounded in Slack at T+3.

CLI I wrap around the same policy

python incident_clock.py accept --file /var/oncall/page.json
python incident_clock.py probe  --incident inc-20260909-0142 --cmd p95
python incident_clock.py status --incident inc-20260909-0142
python incident_clock.py ack    --incident inc-20260909-0142 --actor emery.oncall
python incident_clock.py unfreeze --incident inc-20260909-0142 --cmd canary_toggle --ttl 900
python incident_clock.py handoff --incident inc-20260909-0142 --from emery.oncall --to bob.oncall

Every subcommand prints a receipt line or a deny reason, never a fuzzy paragraph. If handoff runs, writes are frozen until Bob ACKs and issues a fresh unfreeze. I keep kube credentials out of this process, because a policy engine with extra secrets becomes an incident of its own. Can you run status right now and see the clock without asking a model to narrate it?

A decision table I keep in the repo

Clock / condition	Reads	Writes	Escalate	Notes
T+0–T+5, no ACK	allowlisted probes	deny	no	observe-only, ignore unfreeze
T+5–T+10, no ACK	remaining budget	deny	no	still no personality
T+10, no ACK	optional	deny	yes	freeze writes
ACK, unfreeze scoped	yes	one matching id	no	TTL and service must match
Budget spent	deny	deny	yes	freeze wins
Shift handoff	yes	deny	page incoming	new ACK required
Unknown alert class	deny	deny	yes	do not classify live

Test plan I want in the pipeline before the pager is live

I do not need a cluster to prove the policy holds under fake time. I freeze now, feed the executor incidents, and assert deny reasons by string. These tests are a proposal you can copy, not production metrics I pretend already exist. If you skip them, the YAML is just literature sitting next to a dangerous bot.

# test_incident_clock.py — proposal
def test_observe_denies_write_even_with_unfreeze():
    ...

def test_t10_auto_escalates_and_freezes_writes():
    ...

def test_unfreeze_expires_and_cannot_be_reused():
    ...

def test_shift_handoff_freezes_write_lane():
    ...

def test_unknown_class_pages_human():
    ...

def test_empty_write_list_cannot_gain_a_restart():
    ...

def test_sibling_class_does_not_share_write_budget():
    ...

Run them like this, and fail the pipeline if any policy test is skipped.

python -m pytest test_incident_clock.py -q
python -c "import yaml; yaml.safe_load(open('runbook/incident_clock.yaml'))"

If the YAML cannot parse, the bot does not start, and that is intentional friction. A broken runbook is not an invitation to free-style against production with a helpful model. Would you merge a payment service that cannot parse its own config at boot?

Where a coding assistant actually helps

I still want help writing the YAML and the tests, just not help SSHing into prod during the incident. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode, an open-source coding assistant with free model access and a free server option, to draft alert-class stubs and the pytest file above. The assistant proposes probes from my service README, and I keep every mutating argv behind ACK plus unfreeze.

It does not get a production kube context, and I do not paste secrets into the prompt. If you already maintain runbooks in git, try generating one alert class and its negative tests there, then delete anything that is not in your allowlist. The clock stays in your repo either way, which is the part I actually care about during a page.

Limitations, and who should not use this

This design assumes you can identify alert classes before the outage, which baby systems sometimes cannot honestly do. It also assumes your executor is the only path to kubectl and flag flips during an incident. A human with a raw shell can bypass every freeze, and I do not pretend otherwise in this writeup. The clock will look slow if your real ACK time is ninety seconds and you already trust a tiny reviewed catalog.

Do not use this if you want an agent to own the incident end to end without a human ACK. Do not use it if legal requires a two-person unfreeze and this file only stores one actor. Do not use it to hide missing observability behind a budget of six reads that never quite explain the graph. And please do not point a free coding assistant at live credentials just because the runbook is sitting in the same repo.

The core conclusion does not change when the model gets smarter again next month. T+0 stays observe-only, first commands stay probes, escalation stays cheaper than a guessed restart, and freeze still wins at shift change. Can your current bot explain which of those rules it followed, with receipts you could paste? If the answer is a shrug, start with the clock, not with a bigger prompt.