On-Call Alert Triage: Build a Free AI Runbook Bot That Respects Freeze Windows

On-call breaks down not when the pager screams, but when the runbook you read at 2 a.m. is wrong or missing. I've spent too many hours decoding alerts with stale YAML and half-remembered commands, so I started experimenting with AI to generate the first response. The surprising win wasn't a model that knows everything; it was a small bot that pairs a free model with an explicit freeze rule, so the AI can suggest commands without ever accidentally shipping a change during a release blackout. Here's the complete workflow I built using MonkeyCode's free model access and the free server option, and why I wouldn't run a production triage bot without a hard freeze/unfreeze boundary.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why Your AI Runbook Needs a Freeze Rule

A plain AI runbook generator will happily suggest kubectl delete pod or systemctl restart without asking whether you're inside a change freeze. During a freeze, those commands can violate your release policy and turn a small incident into a postmortem. The trick is to let the model do what it's good at — parsing the alert, extracting the service name, and suggesting a list of candidate commands — and then gate every command through a deterministic policy function that knows your freeze calendar. That way the AI stays creative, but the final authority is a boring piece of Python that never hallucinates.

The Architecture: Webhook, Model, Policy, Notify

I built this as a small Python service that runs as a webhook endpoint on MonkeyCode's free server. It receives an alert payload from any monitoring tool, sends the alert text to the free model endpoint, gets back a JSON list of suggested commands, checks each command against the freeze policy, and finally posts the filtered suggestions to Slack. The whole service is under 150 lines, and because the model access and the server are both free, the recurring cost is zero. Let me show you the core pieces, starting with the freeze check.

1. The Freeze/Unfreeze Rule (Pseudocode)

from datetime import datetime, timezone
import re

# Maintain a list of freeze windows, e.g. [("2026-12-24", "2026-12-26"), ...]
FREEZE_WINDOWS = []

READONLY_PATTERN = re.compile(r"^(kubectl get|kubectl describe|journalctl|tail|curl -I|psql -c "SELECT)")

WRITE_PATTERN = re.compile(r"^(kubectl delete|kubectl apply|systemctl restart|rm |mv |DROP )")

def is_in_freeze_window(now=None):
    now = now or datetime.now(timezone.utc)
    for start, end in FREEZE_WINDOWS:
        if start <= now <= end:
            return True
    return False

def is_command_allowed(command, is_freeze=False):
    if not is_freeze:
        return True
    if READONLY_PATTERN.match(command):
        return True
    if WRITE_PATTERN.match(command):
        return False
    # Unknown commands are denied during freeze
    return False

This is deliberately simple and intentionally conservative: during a freeze, only commands that are obviously read-only are allowed. Everything else returns False, and your on-call engineer gets a message like "Command blocked by freeze policy — manual escalation required." You can extend this with your own regexes or move to an allowlist, but the core idea stays the same: the model proposes, the policy disposes.

2. Turning an Alert into Model Suggestions

The next step is the function that sends the alert to the model and asks for a JSON array of commands. I use MonkeyCode's free model endpoint through a simple requests.post call, and I tell the model to respond with nothing except valid JSON. I also add the current freeze state to the prompt so the model knows the constraints, even though the policy function will still veto dangerous commands.

import requests

def get_model_commands(alert_text, freeze_state):
    prompt = f"""
You are an on-call assistant. An alert just fired.
Alert: {alert_text}
Freeze state: {"ACTIVE" if freeze_state else "NOT ACTIVE"}

Return a JSON list of up to 3 diagnostic commands to run first.
If freeze is active, prefer read-only commands.
Output strictly JSON: ["cmd1", "cmd2", "cmd3"]
"""
    # This is a simplified example; use the actual MonkeyCode free model endpoint.
    resp = requests.post(
        "https://free-model.monkeycode.example/v1/chat/completions",
        json={"model": "free-model", "messages": [{"role": "user", "content": prompt}]},
        timeout=15,
    )
    resp.raise_for_status()
    # Parse the JSON array from the response
    import json
    text = resp.json()["choices"][0]["message"]["content"]
    return json.loads(text)

I've written this example using a placeholder endpoint because the exact URL and request format depend on how you authenticate with the free tier. The important part is the pattern: ask for a structured response, parse it defensively, and never treat the model's output as executable instructions until it passes the policy check.

3. The Triage Entrypoint

Every alert hits one function that ties everything together. It fetches the alert, checks the freeze window, calls the model, filters the commands, and posts the result to your on-call channel.

def triage_alert(alert_payload):
    alert_text = alert_payload.get("text", "")
    freeze_active = is_in_freeze_window()

    try:
        commands = get_model_commands(alert_text, freeze_active)
    except Exception as exc:
        commands = []
        # fallback: always show the raw alert to humans

    allowed = []
    blocked = []
    for cmd in commands:
        if is_command_allowed(cmd, freeze_active):
            allowed.append(cmd)
        else:
            blocked.append(cmd)

    message = {
        "alert": alert_text,
        "freeze": freeze_active,
        "suggested_allowed": allowed,
        "suggested_blocked": blocked,
    }
    # Send message to Slack / Teams / email...
    post_to_oncall_channel(message)
    return message

Deploying to the Free Server

MonkeyCode's free server is where I run this webhook. The flow is straightforward: I push the code to a repository, use the MonkeyCode CLI to link the repo to a free server instance, and set the environment variables for the model endpoint and Slack webhook. The server gives me a public URL, which I configure as the alert destination in my monitoring tool. I won't list exact CLI commands here because the interface changes often, and I don't want you to run a stale command. Instead, check the official MonkeyCode docs for the current deployment steps for a Python service.

One thing I learned the hard way: keep the freeze windows themselves in version control. I store them as a JSON file in the repo, so every change to the freeze schedule goes through a pull request and leaves an audit trail. Your freeze rule is only as trustworthy as its review process.

Decision Table: What the Bot Does During a Freeze

Alert Type	Example Command	Freeze Active?	Action
High CPU	kubectl top pods	Yes	Allowed (read-only)
Pod CrashLoop	kubectl logs pod	Yes	Allowed (read-only)
DB Slowness	kubectl delete pod db-0	Yes	Blocked + show manual escalation path
Config Wrong	kubectl apply -f config.yaml	Yes	Blocked + show manual escalation path
Any critical	systemctl restart service	No	Allowed, but still logged

Limitations and Who Should Not Use This

The model's suggestions are unverified by any test suite, so every command that passes the freeze policy should still be reviewed by a human before execution. The regex-based policy is brittle; it won't catch a clever command like kubectl scale deployment app --replicas=0 during a freeze, so you need to keep your patterns current. This approach is also not for teams that operate in regulated industries with strict change-management workflows — a regex allowlist is not a compliance tool. Finally, if your on-call process doesn't have a clear escalation path, adding AI suggestions will make the chaos faster, not better. You need a runbook baseline before you automate parts of it.

Give It a Try Before You Trust It

The best way to judge this pattern is to run it against a week of old alerts from your own monitoring system. I did that, and it immediately caught three commands that the previous human-only runbook would have missed. MonkeyCode's free model access and free server let you run this experiment without opening your wallet. Clone a simple webhook template, plug in your freeze windows, and spend one weekend feeding it historical incidents. Then decide whether the bot earns a place on your real on-call rotation.