The 3 AM pager is a decision problem, not a code problem, and most teams lose the night because the runbook exists only in someone's head. I've been prototyping a lightweight alert-triage loop with MonkeyCode's free model access and free server option, and the lesson is consistent: the model is only as good as the rules you freeze around it. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Why On-Call Got Harder Without a New Alert
AI coding agents changed what breaks, and they broke the old assumptions. My last two debugging stories were a silent config change that produced a 503 after every test passed and a cold start that returned a 200 with an empty body, and both were invisible to the standard dashboards. When a trending DEV discussion asks "What do you do while AI codes?", the honest answer for on-call engineers is: you debug the code the agent wrote at 3 AM. That means the alert text is no longer enough; you need first commands, an escalation ladder, and a freeze rule written before the pager fires.
The Runbook Is a State Machine
A runbook that nobody can execute under stress is a wish. I structure mine as a state machine with four states: alert received, first commands, escalation, and freeze.
Alerts: Severity Before Panic
- P1 — user-facing outage or data loss; page the secondary immediately.
- P2 — degraded but serving; one engineer, no page.
- P3 — cosmetic or latent; log it, review tomorrow.
The first question is not "what is wrong" but "is this real?" Half the alerts I get are stale metrics or deploy noise.
First Commands: The Five You Type Before Reading Anything
curl -sS -o /dev/null -w '%{time_total}' https://api.example.com/healthjournalctl -u api --since '10 min ago' | grep -i error | tail -20git log --oneline -5 --since='2 hours ago'redis-cli llen jobs:queuekubectl get pods --field-selector=status.phase!=Running
Run these before you open the trace explorer, because they tell you whether this is code, infra, or a deploy.
Escalation: Time-Boxed, Not Emotional
| Time | Action |
|---|---|
| 0-5 min | Run first commands, classify P1/P2/P3 |
| 5-15 min | P1: page secondary; P2: keep working alone |
| 15+ min | P1 or stubborn P2: FREEZE deploys, call senior |
| 30+ min | Any unresolved P1: involve the agent author |
The Freeze/Unfreeze Rule
- FREEZE: no deploys, no config changes, no agent runs; AI becomes read-only.
- UNFREEZE: 15 clean minutes of the primary metric, or a confirmed rollback, plus a human saying "unfreeze."
- A model never unfreezes anything. Ever.
The Artifact: A Triage Loop on a Free Server
The free server option is enough to host a small triage service, and the free model allowance (10M tokens as I write this, subject to change) covers a month of alert summaries for a small team. Here is the core script:
# triage.py — triggered by your alert webhook
import os, subprocess, json, requests
MODEL_URL = os.getenv("MODEL_URL") # OpenAI-compatible endpoint
MODEL_KEY = os.getenv("MODEL_KEY")
MODEL_NAME = os.getenv("MODEL_NAME", "free-model")
FIRST_COMMANDS = {
"latency": "curl -sS -o /dev/null -w '%{time_total}' https://api.example.com/health",
"errors": "journalctl -u api --since '10 min ago' | grep -i error | tail -20",
"deploys": "git log --oneline -5 --since='2 hours ago'",
"queue": "redis-cli llen jobs:queue",
}
def run_first_commands():
return {k: subprocess.run(v, shell=True, capture_output=True, text=True).stdout.strip()
for k, v in FIRST_COMMANDS.items()}
def triage(alert, outputs):
prompt = f"""You are an on-call triage assistant.
Classify this alert as P1, P2, or P3 and suggest exactly one next command.
Never propose a rollback, config change, or unfreeze; only a human decides that.
Alert: {json.dumps(alert)}
Command outputs: {json.dumps(outputs)}"""
r = requests.post(MODEL_URL,
headers={"Authorization": f"Bearer {MODEL_KEY}"},
json={"model": MODEL_NAME,
"messages": [{"role": "user", "content": prompt}]})
return r.json()["choices"][0]["message"]["content"]
if __name__ == "__main__":
alert = json.loads(os.environ["ALERT_PAYLOAD"])
print(triage(alert, run_first_commands()))
The script does not auto-remediate, and that is the point. It summarizes the state, classifies severity, and suggests the next command, so the human spends less time reading logs and more time deciding.
Where This Breaks
Free tiers have limits, and you should treat them as a constraint, not a promise. Token allowances change, the free server may have cold starts, and a model can confidently misclassify a P1 as a P3; I have seen it happen with a truncated log. Do not use this approach if you operate under strict compliance rules, if alerts carry customer data that cannot leave your network, or if your team treats the model's suggestion as a decision. The runbook, not the model, is what keeps the night boring.
If you want to try this pattern, the free tier is enough to prototype it this weekend: write the runbook, deploy the script to the free server, and point one test alert at it. The pager will still ring, but at least you will know what to type first.
Top comments (0)