Escalation Is a State Machine, Not a Phone Tree

The worst on-call hour starts with a missed fact, not a missed phone call. You see the alert, skim three paragraphs of a runbook, and run the wrong command first—or skip the freeze window entirely—and the incident gets worse. The fix is to model escalation as a state machine and let each alert carry the context needed to move to the next state.

Escalation isn't a sequence of phone numbers; it's a transition between states. In this post I'll show you how to encode alerts, first commands, escalation steps, and a freeze/unfreeze rule into a small runbook that runs on a free server. No PDFs, no heroics—just a deterministic path from pager to page-out.

The Phone Tree Fallacy

Most runbooks describe escalation as a list: "Call Bob, then Alice, then the on-call manager." That works until Bob is asleep, Alice is on call, and the manager's page goes to an old phone. The real problem is missing a decision: which state are we in, and what moves us out of it?

A state machine forces you to answer that question for every possible alert. The state transitions remain predictable even when humans are slow, tired, or unavailable.

A Runbook State Machine in YAML

Start by writing the runbook as data, not prose. Here's a simplified version for a checkout service:

version: 1
name: checkout-service

freeze_window:
  timezone: "UTC"
  windows:
    - days: ["Mon", "Tue", "Wed", "Thu"]
      start: "09:00"
      end: "17:00"
    - days: ["Fri"]
      start: "09:00"
      end: "12:00"

states:
  initial: AwaitingContext
  transition_rules:
    AwaitingContext:
      timeout_minutes: 5
      on_timeout: Escalated
    Escalated:
      timeout_minutes: 15
      on_timeout: Paging
    Paging:
      timeout_minutes: 60
      on_timeout: Incident

alerts:
  - matcher: "high_5xx_ratio"
    first_commands:
      - "kb logs --service checkout --since 10m | grep 5xx | head"
      - "kb metrics latency --service checkout --window 1h"
    target_state: AwaitingContext
  - matcher: "payment_timeout"
    first_commands:
      - "kb traces --service checkout --span PaymentService --timeout | top"
      - "kb status --dependency payment-gateway"
    target_state: AwaitingContext

Notice the freeze_window block. It's not a comment; it's a guard. When an alert arrives during a freeze window, the runbook says: acknowledge, log, but do not run remediation commands. That's the rule your past self would have thanked you for.

Freeze Rules as a Guard, Not a Suggestion

A freeze window only works if it's enforced by the same system that runs the runbook. If the freeze check lives in a human's head, it gets skipped exactly when a scary alert fires at 4 PM on a Friday.

Here's the freeze check in Python, using zoneinfo to avoid UTC mistakes:

from zoneinfo import ZoneInfo
from datetime import datetime

def in_freeze(event_time, freeze_config):
    tz = ZoneInfo(freeze_config["timezone"])
    local = event_time.astimezone(tz)
    weekday = local.strftime("%a")
    hhmm = local.strftime("%H:%M")
    for window in freeze_config["windows"]:
        if weekday in window["days"] and window["start"] <= hhmm <= window["end"]:
            return True
    return False

Use this function as the first branch in your alert handler. If the freeze is active, the state machine never advances past acknowledgment.

From Alert to State: The Handler

Now the fun part: turning an incoming webhook into a state transition. Below is a compact handler that matches an alert, checks the freeze, and returns the first commands to execute.

import re
import yaml
from datetime import datetime, timezone

runbook = yaml.safe_load(open("runbook.yaml"))

def handle_alert(alert_data):
    if in_freeze(datetime.now(timezone.utc), runbook["freeze_window"]):
        return "FREEZE", "acknowledged, no remediation permitted"
    for rule in runbook["alerts"]:
        if re.search(rule["matcher"], alert_data["title"], re.I):
            return rule["target_state"], rule["first_commands"]
    return "UNMATCHED", "runbook has no entry; escalate manually"

That's it—the core logic fits in a few dozen lines. The state machine's timeout transitions live in a cron job or a background worker that checks updated_at on each incident record.

Running It on a Free Server

You don't need a paid VM for a tiny HTTP endpoint. I exposed this handler as a webhook and deployed it to MonkeyCode's free server option. The setup took five minutes: paste the Python file, set the runbook YAML as an environment variable, and get a public URL. For alert traffic volume of a small team, it's plenty.

For the AI piece, MonkeyCode's free model access lets me send the alert title and recent log lines to a model and get back a one-line summary. That summary becomes a field in the incident state, so the on-call engineer doesn't have to read eleven pages of logs before deciding. I keep the prompt tiny and explicitly tell the model: "If you don't know, say so." Hallucinations are a real risk, which is why the freeze rule and the deterministic path still have the final say.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Who Should Not Use This

Let's be honest about boundaries. If your service is safety-critical, runs in a regulated industry, or handles customer PII, a free-tier webhook plus an LLM summary is not sufficient for automated decisions. Use this pattern as a decision aid, not a decision-maker. The state machine should always end in a human page unless you've independently verified the model's summary against a trusted log store.

Also, verify your freeze windows against the team's actual on-call calendar. A freeze rule that's wrong by one timezone is worse than no freeze rule at all because it gives false confidence.

The Takeaway

Escalation is a state machine. Define your states, encode your alerts and first commands, and let the freeze rule be a concrete guard instead of a memory. When the pager goes off at 3 AM, you'll already know exactly what the first command is, because the machine told you before you even opened your laptop.

If you're building a similar runbook bot, MonkeyCode's free server and 10-million-token allowance are a reasonable place to start—but build the state machine first, and treat the AI summary as a nice-to-have. What does your escalation state machine look like? I'd love to see a YAML that beats mine.