Incomplete Alerts Are a Trap: Make the On-Call Bot Fail Closed

If a paging alert is missing a host, service, or freeze flag, your on-call bot should stop, not improvise. I keep seeing agents fill those holes with guesses that look confident and still page the wrong people. The safer pattern is a typed alert contract, a read-only first command list, and a freeze latch only a human can lift. Everything below is a proposed workflow you can run locally; treat the code as a starting kit, not production gospel.

Have you watched a model invent a hostname because the payload looked almost complete? That is the failure I want to kill this week. Current agent writeups keep circling the same wound: tools assume missing context, then act. On-call is the worst place for that habit, because a missing field is not a creative prompt. It is a reason to refuse the page.

Why incomplete pages hurt more than late pages

A late page still points at a real service, a real host, and a real owner. An incomplete page sends someone into a fog with a critical severity sticker glued on top. I would rather the bot reply INCOMPLETE_ALERT in Slack than SSH into a guessed box. Does that feel slow during a real incident? Yes, and that slowness is the point of a contract.

Here is the rule I want at the top of the runbook, before any model sees the payload:

• Required fields must exist, parse, and match an allowlist.
• Missing fields fail closed; the bot does not complete them.
• First commands are read-only probes, never restarts or deploys.
• Freeze is a latch on disk; unfreeze is a human action with a reason.
• Escalation follows a table of severity, completeness, and freeze state.

If those five bullets already feel strict, good. On-call bots do not need more imagination. They need fewer chances to lie.

A typed alert contract, not a friendlier prompt

I do not start with a system prompt that says “be careful.” I start with a schema the process can reject before any model call. The contract below is proposed Python, not a war story from a specific outage, and you should swap the enums for your own services.

# alert_contract.py — proposed local validator, not a hosted service
from __future__ import annotations

from datetime import datetime, timezone
from enum import Enum
from typing import Optional

from pydantic import BaseModel, Field, ValidationError, field_validator

class Severity(str, Enum):
    INFO = "info"
    WARN = "warn"
    CRITICAL = "critical"

class Service(str, Enum):
    CHECKOUT = "checkout"
    PAYMENTS = "payments"
    INGRESS = "ingress"

ALLOWED_HOST_SUFFIXES = (".prod.internal", ".staging.internal")

class AlertContract(BaseModel):
    alert_id: str = Field(min_length=8, max_length=64)
    service: Service
    severity: Severity
    host: str
    runbook_id: str
    freeze_hint: bool
    received_at: datetime
    page_owner: str
    extra: Optional[dict] = None

    @field_validator("host")
    @classmethod
    def host_must_be_ours(cls, value: str) -> str:
        if not value.endswith(ALLOWED_HOST_SUFFIXES):
            raise ValueError("host is not in the allowlist")
        if any(ch.isspace() for ch in value):
            raise ValueError("host cannot contain whitespace")
        return value

    @field_validator("received_at")
    @classmethod
    def must_be_aware(cls, value: datetime) -> datetime:
        if value.tzinfo is None:
            raise ValueError("received_at must include a timezone")
        return value


def parse_alert(payload: dict) -> AlertContract:
    try:
        return AlertContract.model_validate(payload)
    except ValidationError as exc:
        raise ValueError(f"INCOMPLETE_ALERT: {exc.error_count()} field issues") from exc

Notice what the model never gets to do here. It never invents payments-2.prod.internal because the JSON key was blank. It never upgrades warn to critical because the description sounded scary. If the contract fails, the bot stops. Why would we even call a model before that gate?

First commands that cannot mutate anything

After the contract passes, I still do not want a shell that can restart a fleet. I want a tiny allowlist of read-only probes keyed by service. Anything else is a human decision, including the tempting one-liner that “just bounces the pod.”

# first_commands.py — proposed allowlist
READ_ONLY = {
    "checkout": [
        "kubectl -n checkout get pods -o wide",
        "curl -fsS https://checkout.prod.internal/healthz",
        "journalctl -u checkout --since -10m --no-pager | tail -n 80",
    ],
    "payments": [
        "kubectl -n payments get pods -o wide",
        "curl -fsS https://payments.prod.internal/healthz",
        "redis-cli -h payments-redis.prod.internal ping",
    ],
    "ingress": [
        "kubectl -n ingress get pods -o wide",
        "curl -fsS https://ingress.prod.internal/healthz",
        "ss -ltnp | grep -E ':443|:80'",
    ],
}

FORBIDDEN_TOKENS = ("restart", "delete", "scale", "rollout undo", "drain", "reboot")


def first_commands(service: str) -> list[str]:
    cmds = READ_ONLY[service]
    for cmd in cmds:
        lowered = cmd.lower()
        if any(token in lowered for token in FORBIDDEN_TOKENS):
            raise RuntimeError(f"mutating token in allowlist: {cmd}")
    return cmds

I like that this file can fail CI if somebody sneaks rollout restart into the probe list. Would you rather debate that in a postmortem, or in a unit test on a Tuesday afternoon? I know my answer.

Freeze is a latch; unfreeze is a human act

A freeze window is not a sentence in the prompt. It is a latch the bot can read and cannot lift. I keep it as a small JSON file so a human, or a locked change ticket, is the only writer. The bot may notice freeze, and it may refuse mutating advice, but it does not get to declare the freeze over because error rates dropped.

# freeze_latch.py — proposed local latch
import json
from pathlib import Path

LATCH = Path("/var/oncall/freeze.json")


def freeze_state() -> dict:
    if not LATCH.exists():
        return {"frozen": False, "reason": "no-latch", "by": None}
    data = json.loads(LATCH.read_text())
    return {
        "frozen": bool(data.get("frozen")),
        "reason": data.get("reason") or "unspecified",
        "by": data.get("by"),
    }


def assert_can_page(severity: str) -> None:
    state = freeze_state()
    if state["frozen"] and severity != "critical":
        raise PermissionError(
            f"FROZEN: {state['reason']} (set by {state['by']}); non-critical pages are suppressed"
        )


def human_unfreeze(actor: str, reason: str) -> None:
    if not actor or not reason or len(reason) < 12:
        raise ValueError("unfreeze requires an actor and a real reason")
    LATCH.write_text(json.dumps({
        "frozen": False,
        "reason": reason,
        "by": actor,
    }, indent=2))

Critical alerts can still page during freeze, because a payments outage does not wait for the marketing freeze to end. Everything else waits. Who should hold the unfreeze key? A human on the change calendar, not the same process that wants to look helpful.

Escalation is a table, not a vibe

I do not want the bot to “use its judgment” about who to wake. I want a boring table that a new hire can read at 03:00. Completeness is a first-class input, sitting beside severity and freeze, because an incomplete critical alert is an investigation, not a page storm.

Severity	Contract	Freeze	Action
info	valid	any	ticket only, no page
warn	valid	frozen	ticket only
warn	valid	open	page secondary after 15 minutes
critical	valid	any	page primary now
any	invalid	any	do not page; post INCOMPLETE_ALERT and wake the on-call only if it repeats

# escalate.py — proposed decision table
from dataclasses import dataclass

@dataclass
class Decision:
    page_primary: bool
    page_secondary: bool
    ticket_only: bool
    note: str


def decide(severity: str, valid: bool, frozen: bool) -> Decision:
    if not valid:
        return Decision(False, False, True, "INCOMPLETE_ALERT")
    if severity == "info":
        return Decision(False, False, True, "info stays on the ticket board")
    if severity == "warn" and frozen:
        return Decision(False, False, True, "warn suppressed during freeze")
    if severity == "warn":
        return Decision(False, True, True, "warn pages secondary after delay")
    return Decision(True, False, True, "critical pages primary even when frozen")

Ask yourself the uncomfortable question before you wire this to a pager. If the alert is critical and the host field is empty, do you still want a 03:00 phone call? I do not, unless the same incomplete fingerprint repeats twice. Repeats are a signal. One broken exporter is noise.

A reproducible test plan, not a vibe check

I want tests that fail on a laptop, without a cluster and without a model bill. Label these as unexecuted examples until you run them. They encode the contract, not my personal production metrics, because I am not inventing a customer outage to sell the pattern.

# test_oncall_contract.py
from datetime import datetime, timezone
import pytest

from alert_contract import parse_alert
from escalate import decide
from first_commands import first_commands
from freeze_latch import assert_can_page

NOW = datetime.now(timezone.utc)

BASE = {
    "alert_id": "alt-9f3c12ab",
    "service": "payments",
    "severity": "critical",
    "host": "pay-1.prod.internal",
    "runbook_id": "rb-payments-latency",
    "freeze_hint": False,
    "received_at": NOW.isoformat(),
    "page_owner": "payments-primary",
}

def test_rejects_blank_host():
    payload = dict(BASE, host="")
    with pytest.raises(ValueError, match="INCOMPLETE_ALERT"):
        parse_alert(payload)

def test_rejects_unknown_suffix():
    payload = dict(BASE, host="pay-1.random.cloud")
    with pytest.raises(ValueError, match="INCOMPLETE_ALERT"):
        parse_alert(payload)

def test_first_commands_stay_read_only():
    for cmd in first_commands("payments"):
        assert "restart" not in cmd.lower()

def test_incomplete_never_pages_primary():
    d = decide("critical", valid=False, frozen=False)
    assert d.page_primary is False
    assert d.note == "INCOMPLETE_ALERT"

def test_warn_does_not_page_when_frozen(tmp_path, monkeypatch):
    latch = tmp_path / "freeze.json"
    latch.write_text('{"frozen": true, "reason": "release freeze", "by": "sre"}')
    monkeypatch.setattr("freeze_latch.LATCH", latch)
    with pytest.raises(PermissionError, match="FROZEN"):
        assert_can_page("warn")

Run it like this, then paste a broken payload from last week’s alert tool and watch it fail closed:

python -m pip install pydantic pytest
python -m pytest -q test_oncall_contract.py

If a test fails because your real hostnames use a different suffix, that is a gift. Change the allowlist. Do not loosen the parser until a model can “figure it out.”

Where a free coding workspace actually fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a measured speedup, a model name, or a quota. I am saying the boring part of this workflow is rewriting the contract and the tests until they hurt in the right places. MonkeyCode’s free model access and free server option are useful there, because you can iterate the validator on a scratch box without pointing it at production kubeconfigs. Keep secrets off that box. Keep the latch file in a repo your on-call humans already trust.

The product is not the runbook. The contract is the runbook. If you strip every product name out of this article, you should still have a schema, an allowlist, a latch, and a table you can argue about in a review.

Limitations, and who should not do this

This approach is wrong for several honest reasons, and I want those on the same page as the code. A fail-closed contract will drop pages when exporters omit fields, which is safer than guessing and still dangerous if your monitoring is sloppy. Read-only first commands will not restore a dead shard. A file-based freeze latch is not a distributed lock, and two writers can still race if you skip real change control.

Do not use this if you lack paging authority, if you cannot name an allowlist of hosts, or if a regulator expects every action to go through an already approved runbook tool. Do not point a free server at production credentials. Do not let a model write the unfreeze reason after the fact. And do not copy this table into a bank or hospital pager without a human review that I cannot do for you in a blog post.

I also will not pretend this matches every stack. If your alerts arrive as free-form email, you need a parser in front of the contract, and that parser should fail closed too. If your freeze calendar lives in a vendor, read it through an API you already audit, not through a scraped screenshot.

What I want you to try on the next noisy afternoon

Take one real alert JSON from staging, strip the host, and run the tests. Then put the host back, set the freeze latch, and confirm a warn does not page. After that, print the first commands and ask a teammate if any line could mutate state. If the answer is yes, you do not need a smarter agent. You need a smaller allowlist.

If you want a scratch place to hammer on those tests without waking a pager, MonkeyCode’s free server option is enough for a dry run. Keep the disclosure in mind, keep production keys off the box, and let the contract stay stricter than the model. That is the whole lesson, and it still holds if you never touch the product at all.