We're building an incident triage agent that reads structured error logs, queries internal runbooks through function calling, and returns a concise remediation checklist. On-call engineers can use it to shrink initial response time from several minutes down to a few seconds.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
- Ten minutes
Oxlo.ai runs all of these models with request-based pricing, so the cost of a long log dump or a multi-turn tool loop does not scale with token count. That makes it practical to ship this as an always-on internal service. See https://oxlo.ai/pricing for details.
Step 1: Test the connection
First, confirm that your Oxlo.ai key and the OpenAI SDK are wired correctly. I always do this before I add any logic.
from openai import OpenAI
import json
import os
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY"),
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[{"role": "user", "content": "Say 'Connection OK' and nothing else."}],
)
print(response.choices[0].message.content)If you see Connection OK, the endpoint is live and you are ready to build.
Step 2: Define the system prompt and tools
The agent needs a tight system prompt so it stays focused on triage, plus two tools: one to look up runbooks and one to page the on-call engineer if the severity warrants it. Here is the prompt I shipped.
SYSTEM_PROMPT = """You are an on-call incident triage agent.
Your job is to analyze error logs, identify the root service, look up the relevant runbook, and produce a numbered remediation checklist.
If the log indicates a severity of 'critical' or 'outage', you must page the on-call engineer using the page_oncall tool.
Do not speculate beyond the runbook content. Keep your final answer under 150 words."""Now register the tool schema. Oxlo.ai exposes function calling through the standard OpenAI chat completions format, so the model can request a tool exactly like it would with any other provider.
TOOLS = [
{
"type": "function",
"function": {
"name": "lookup_runbook",
"description": "Fetch the runbook for a given service.",
"parameters": {
"type": "object",
"properties": {
"service": {
"type": "string",
"description": "Name of the affected service, e.g. postgres, payment-gateway, auth-service.",
}
},
"required": ["service"],
},
},
},
{
"type": "function",
"function": {
"name": "page_oncall",
"description": "Page the on-call engineer for critical incidents.",
"parameters": {
"type": "object",
"properties": {
"reason": {
"type": "string",
"description": "Short reason for the page.",
}
},
"required": ["reason"],
},
},
},
]Step 3: Implement the tool handlers
In production these would hit your internal wiki and PagerDuty. For this tutorial I use a hardcoded runbook dictionary and a print stub so the script stays runnable offline.
RUNBOOKS = {
"postgres": "1. Check connection pool metrics in Datadog. 2. If pool > 80%, restart the pgbouncer sidecars. 3. If recovery fails, fail over to the read replica.",
"payment-gateway": "1. Verify Stripe API status page. 2. Check webhook delivery logs. 3. If latency > 5s, enable circuit breaker and queue transactions.",
"auth-service": "1. Check Redis session store memory usage. 2. Rotate signing keys if JWT validation errors spike. 3. Scale pods if CPU > 70%.",
}
def lookup_runbook(service: str) -> str:
return RUNBOOKS.get(service, f"No runbook found for {service}. Escalate manually.")
def page_oncall(reason: str) -> str:
print(f"[PAGING ONCALL] {reason}")
return "On-call engineer paged successfully."Step 4: Build the agent loop
The loop sends the user log to llama-3.3-70b, handles any tool calls, appends the results, and asks the model again. I cap it at five turns to prevent runaway loops.
def triage_incident(log_text: str, max_turns: int = 5) -> str:
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": log_text},
]
for _ in range(max_turns):
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
tools=TOOLS,
tool_choice="auto",
)
message = response.choices[0].message
if message.tool_calls:
messages.append({
"role": "assistant",
"content": message.content or "",
"tool_calls": [tc.model_dump() for tc in message.tool_calls],
})
for tc in message.tool_calls:
fn_name = tc.function.name
args = json.loads(tc.function.arguments)
if fn_name == "lookup_runbook":
result = lookup_runbook(args["service"])
elif fn_name == "page_oncall":
result = page_oncall(args["reason"])
else:
result = "Unknown tool."
messages.append({
"role": "tool",
"tool_call_id": tc.id,
"content": result,
})
else:
return message.content
return "Agent reached max turns without a final answer."Step 5: Run it
Here is a realistic log snippet. I pipe it through the agent and print the result.
LOG = """[2024-05-21T03:14:00Z] severity=critical service=postgres msg="FATAL: remaining connection slots are reserved for non-replicated superuser connections"""
print(triage_incident(LOG))When I run this, the model first calls lookup_runbook with service "postgres". After receiving the runbook text, it returns a final answer. Typical output looks like this:
[PAGING ONCALL] Critical postgres connection pool exhaustion detected.
1. Check connection pool metrics in Datadog.
2. If pool > 80%, restart the pgbouncer sidecars.
3. If recovery fails, fail over to the read replica.Because Oxlo.ai charges per request rather than per token, running this loop, even with a long log payload and a multi-turn tool exchange, costs the same flat rate each time. That predictability matters when you are shipping an internal tool that on-call engineers will hammer at 3 AM.
Next steps
Swap llama-3.3-70b for qwen-3-32b if your stack emits logs in multiple languages, or switch to deepseek-v3.2 if you want stronger reasoning over cryptic stack traces. After that, consider wiring the page_oncall function to your real paging provider and deploying this as a Slack slash command so the team can invoke it without leaving the incident channel.
Top comments (0)