We are building an on-call triage agent that consumes raw JSON alerts from your existing monitoring stack and returns structured incident reports with severity, root-cause analysis, and remediation steps. It is designed to slot into your current PagerDuty, Slack, or custom webhook pipeline without replacing any existing tooling. If you have ever been paged by a 500-line stack trace at 3 a.m., this tool is for you.
What you'll need
- Python 3.10 or newer.
- An Oxlo.ai API key from https://portal.oxlo.ai.
- The OpenAI SDK:
pip install openai. - A sample alert JSON file. We will create one in Step 3.
Step 1: Test the Oxlo.ai client
Before we write any logic, we verify that the client can reach Oxlo.ai and that our key works. I like to do a quick health check with a trivial completion to catch networking or auth issues early.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Say 'Oxlo.ai is up' and nothing else."},
],
)
print(response.choices[0].message.content)
Step 2: Write the system prompt
The system prompt is the only part of the agent that is not code, so I keep it in a dedicated constant that the team can edit without touching the logic. It defines the output schema and the tone.
SYSTEM_PROMPT = """You are an on-call triage engineer. Your job is to read a raw system alert and produce a structured incident report.
Follow these rules exactly:
1. Output valid Markdown.
2. Include a Severity header (Critical, High, Medium, or Low).
3. Include a Root Cause header with a one-sentence summary.
4. Include a Remediation header with ordered steps.
5. Include a Dependencies header listing any services mentioned.
6. Be concise. Do not hallucinate logs or metrics that are not in the input.
"""
Step 3: Ingest and normalize alerts
Real engineering systems emit noisy JSON. We will write a small normalizer that accepts a file path, validates that required fields exist, and flattens the payload into a single string for the LLM. This keeps the prompt clean and prevents token bloat from nested metadata.
import json
def normalize_alert(path: str) -> str:
with open(path, "r") as f:
payload = json.load(f)
required = {"service", "timestamp", "message"}
if not required.issubset(payload.keys()):
raise ValueError(f"Missing required fields: {required - payload.keys()}")
return json.dumps(payload, indent=2)
Step 4: Build the triage core
This is the function that calls Oxlo.ai. We pass the normalized alert as the user message and let the model reason through the report. I use llama-3.3-70b here because it is reliable for structured generation and follows system instructions tightly. Because Oxlo.ai uses flat per-request pricing, sending a large JSON payload does not increase the cost.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
def triage_alert(alert_json: str) -> str:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": alert_json},
],
)
return response.choices[0].message.content
Step 5: Add a deterministic severity gate
LLMs are great at reasoning, but I still enforce a hard rule. If the alert contains the words "payment" and "timeout", we auto-escalate to Critical before the LLM sees it. This hybrid approach keeps the agent grounded in business rules your team already trusts.
def apply_severity_gate(payload: dict) -> dict:
message = payload.get("message", "").lower()
if "payment" in message and "timeout" in message:
payload["severity_override"] = "Critical"
return payload
def build_user_message(payload: dict) -> str:
gated = apply_severity_gate(payload)
return json.dumps(gated, indent=2)
Step 6: Wire the CLI
We package everything into a small script that reads a JSON file path and prints the report. This is the shape you would drop into a cron job, a webhook handler, or a CI step.
import sys
import json
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are an on-call triage engineer. Your job is to read a raw system alert and produce a structured incident report.
Follow these rules exactly:
1. Output valid Markdown.
2. Include a Severity header (Critical, High, Medium, or Low).
3. Include a Root Cause header with a one-sentence summary.
4. Include a Remediation header with ordered steps.
5. Include a Dependencies header listing any services mentioned.
6. Be concise. Do not hallucinate logs or metrics that are not in the input.
"""
def apply_severity_gate(payload: dict) -> dict:
message = payload.get("message", "").lower()
if "payment" in message and "timeout" in message:
payload["severity_override"] = "Critical"
return payload
def triage_alert(alert_json: str) -> str:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": alert_json},
],
)
return response.choices[0].message.content
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python triage.py alert.json")
sys.exit(1)
with open(sys.argv[1], "r") as f:
payload = json.load(f)
payload = apply_severity_gate(payload)
report = triage_alert(json.dumps(payload, indent=2))
print(report)
Run it
Create a file named alert.json with the following content.
{
"service": "payment-gateway",
"timestamp": "2024-05-21T03:14:00Z",
"message": "payment timeout after 30s, downstream provider unresponsive",
"host": "prod-api-07",
"trace_id": "abc123"
}
Then run the agent.
python triage.py alert.json
You should see output similar to this.
## Severity
Critical
## Root Cause
The payment gateway timed out waiting for the downstream provider, indicating a dependency failure.
## Remediation
1. Check the downstream provider status page and internal health endpoint.
2. Review recent deployments to the payment-gateway service.
3. If the provider is degraded, enable the circuit breaker and queue transactions for retry.
4. Notify the payments on-call channel with trace_id abc123.
## Dependencies
- payment-gateway
- downstream provider
Next steps
Replace the local JSON file with a webhook listener that consumes your actual Alertmanager or PagerDuty events. You can host it as a tiny FastAPI service and return the Markdown report directly to Slack.
If you start correlating multi-service outages, swap llama-3.3-70b for deepseek-v3.2 or kimi-k2.6 on Oxlo.ai. Both models handle longer reasoning chains well, and the flat per-request pricing means you can pass in large trace dumps without watching the meter run.
Top comments (0)