We are going to build an on-call triage agent that ingests structured application logs and produces a severity-ranked incident summary. It sits downstream of your existing log aggregator and feeds directly into Slack or PagerDuty, so you do not need to replace any infrastructure to get LLM-powered context.
What you'll need
- Python 3.10 or newer
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK:
pip install openai
I also assume you have a Unix-like terminal and a directory where you can create a few files.
Step 1: Verify connectivity
Before we touch any logs, we should confirm that the Python client can reach Oxlo.ai and that our key works. I keep my key in an environment variable called OXLO_API_KEY.
from openai import OpenAI
import os
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key=os.getenv("OXLO_API_KEY"))
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Say 'Connection OK' and nothing else."},
],
)
print(response.choices[0].message.content)Run that once. If you see Connection OK, we can move on.
Step 2: Define the system prompt
The system prompt is the only part of the agent you will tune later, so I keep it in a dedicated constant. It tells the model how to classify logs and what shape to return.
SYSTEM_PROMPT = """You are an on-call triage assistant.
Your job is to read a batch of structured log entries and produce a concise incident summary.
Return ONLY a JSON object with this exact structure:
{
"severity": "critical|high|low",
"affected_service": "name of primary service",
"summary": "One sentence describing the issue.",
"recommended_action": "One concrete step for the on-call engineer."
}
Do not include markdown formatting or explanations outside the JSON."""Step 3: Create a sample log source
Most engineering teams already emit structured logs in JSON Lines format. We will simulate that by writing a file named logs.jsonl with realistic entries from a payment service and an auth service.
import json
LOG_LINES = [
{"ts": "2024-06-01T14:22:10Z", "svc": "payments", "lvl": "ERROR", "msg": "Stripe webhook timeout after 30s", "trace_id": "abc123"},
{"ts": "2024-06-01T14:22:11Z", "svc": "payments", "lvl": "ERROR", "msg": "Stripe webhook timeout after 30s", "trace_id": "abc124"},
{"ts": "2024-06-01T14:23:00Z", "svc": "auth", "lvl": "WARN", "msg": "Elevated 401s from /login", "trace_id": "def456"},
{"ts": "2024-06-01T14:24:05Z", "svc": "payments", "lvl": "INFO", "msg": "Retry succeeded", "trace_id": "abc125"},
]
with open("logs.jsonl", "w") as f:
for line in LOG_LINES:
f.write(json.dumps(line) + "\n")
print("Wrote logs.jsonl")Step 4: Ingest and filter
In production you might pull from S3 or Elasticsearch. Here we read the local file and keep only ERROR and WARN lines, then serialize them into a compact string for the LLM context.
import json
def load_recent_logs(path: str, max_lines: int = 50):
logs = []
with open(path) as f:
for line in f:
entry = json.loads(line)
if entry["lvl"] in ("ERROR", "WARN"):
logs.append(entry)
return logs[:max_lines]
def format_for_llm(logs):
lines = []
for log in logs:
lines.append(f"{log['ts']} [{log['svc']}] {log['lvl']}: {log['msg']} ({log['trace_id']})")
return "\n".join(lines)
raw_logs = load_recent_logs("logs.jsonl")
log_text = format_for_llm(raw_logs)
print(log_text)Step 5: Classify with Oxlo.ai
Now we send the filtered logs to llama-3.3-70b on Oxlo.ai. Because Oxlo.ai charges per request, not per token, we can pass a large chunk of log context without worrying about prompt length driving up cost. The model returns structured JSON we can parse downstream.
from openai import OpenAI
import json, os
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key=os.getenv("OXLO_API_KEY"))
SYSTEM_PROMPT = """You are an on-call triage assistant.
Your job is to read a batch of structured log entries and produce a concise incident summary.
Return ONLY a JSON object with this exact structure:
{
"severity": "critical|high|low",
"affected_service": "name of primary service",
"summary": "One sentence describing the issue.",
"recommended_action": "One concrete step for the on-call engineer."
}
Do not include markdown formatting or explanations outside the JSON."""
def classify_incident(log_text: str):
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": log_text},
],
)
raw = response.choices[0].message.content.strip()
if raw.startswith("
```"):
raw = raw.split("```
")[1].replace("json", "").strip()
return json.loads(raw)
result = classify_incident(log_text)
print(json.dumps(result, indent=2))Step 6: Format for Slack
Your existing paging system probably accepts plain text or Markdown. We will turn the JSON decision into a brief Slack message block so a human can act on it immediately.
def format_slack(result: dict, log_count: int):
emoji = {"critical": ":fire:", "high": ":warning:", "low": ":information_source:"}.get(
result["severity"], ":question:"
)
return f"""{emoji} *{result['severity'].upper()}* incident detected in `{result['affected_service']}`
*Summary:* {result['summary']}
*Recommended action:* {result['recommended_action']}
*Logs analyzed:* {log_count}"""
slack_msg = format_slack(result, len(raw_logs))
print(slack_msg)Run it
Here is the complete script. Save it as triage.py, ensure logs.jsonl exists, and run python triage.py.
from openai import OpenAI
import json, os
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key=os.getenv("OXLO_API_KEY"))
SYSTEM_PROMPT = """You are an on-call triage assistant.
Your job is to read a batch of structured log entries and produce a concise incident summary.
Return ONLY a JSON object with this exact structure:
{
"severity": "critical|high|low",
"affected_service": "name of primary service",
"summary": "One sentence describing the issue.",
"recommended_action": "One concrete step for the on-call engineer."
}
Do not include markdown formatting or explanations outside the JSON."""
def load_recent_logs(path: str, max_lines: int = 50):
logs = []
with open(path) as f:
for line in f:
entry = json.loads(line)
if entry["lvl"] in ("ERROR", "WARN"):
logs.append(entry)
return logs[:max_lines]
def format_for_llm(logs):
lines = []
for log in logs:
lines.append(f"{log['ts']} [{log['svc']}] {log['lvl']}: {log['msg']} ({log['trace_id']})")
return "\n".join(lines)
def classify_incident(log_text: str):
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": log_text},
],
)
raw = response.choices[0].message.content.strip()
if raw.startswith("
```"):
raw = raw.split("```
")[1].replace("json", "").strip()
return json.loads(raw)
def format_slack(result: dict, log_count: int):
emoji = {"critical": ":fire:", "high": ":warning:", "low": ":information_source:"}.get(
result["severity"], ":question:"
)
return f"""{emoji} *{result['severity'].upper()}* incident detected in `{result['affected_service']}`
*Summary:* {result['summary']}
*Recommended action:* {result['recommended_action']}
*Logs analyzed:* {log_count}"""
if __name__ == "__main__":
raw_logs = load_recent_logs("logs.jsonl")
if not raw_logs:
print("No errors or warnings found.")
exit(0)
log_text = format_for_llm(raw_logs)
result = classify_incident(log_text)
print(format_slack(result, len(raw_logs)))When I ran this against the sample logs, the output looked like this:
🔥 *CRITICAL* incident detected in `payments` *Summary:* Multiple consecutive Stripe webhook timeouts indicate a payment processing outage. *Recommended action:* Check Stripe status page and verify webhook endpoint network connectivity. *Logs analyzed:* 3
Next steps
Swap the local file reader for a boto3 pull from your S3 log bucket or a query to Elasticsearch so the agent runs against real data. If volume grows, move the script into a Lambda or Kubernetes CronJob and route the Slack payload to a webhook instead of stdout.
For workloads like this where the prompt grows with log volume, Oxlo.ai's per-request pricing keeps the cost flat no matter how much context you need to include. You can explore plans and model options at https://oxlo.ai/pricing.
Top comments (0)