Building an LLM Operations Audit Agent gives you a single interface to turn raw inference logs into actionable health reports. We will construct a Python agent that reads a batch of API call records, detects anomalies like latency spikes and elevated error rates, and returns a structured JSON summary. This is aimed at engineers who run production LLM workloads and need to surface problems quickly without building a full observability stack.
What you'll need
- Python 3.10 or higher
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK:
pip install openai
Step 1: Bootstrap the client
Import the required modules and instantiate the OpenAI SDK against Oxlo.ai. I keep the API key in an environment variable in practice, but here is the explicit client setup.
import os
import json
from datetime import datetime, timedelta, timezone
import random
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)Step 2: Generate synthetic operations logs
Since we cannot assume you already have a log warehouse, we will manufacture a realistic batch of twenty inference records. I intentionally bias the later entries for deepseek-v3.2 so we have a real anomaly to detect.
MODELS = ["llama-3.3-70b", "qwen-3-32b", "deepseek-v3.2", "kimi-k2.6"]
def generate_logs(n=20):
logs = []
base_time = datetime.now(timezone.utc) - timedelta(hours=1)
for i in range(n):
model = random.choice(MODELS)
if model == "deepseek-v3.2" and i > 10:
latency = random.randint(8000, 12000)
status = random.choice([500, 502, 200])
else:
latency = random.randint(200, 1500)
status = 200
logs.append({
"timestamp": (base_time + timedelta(minutes=i*3)).isoformat(),
"model": model,
"status_code": status,
"latency_ms": latency,
"prompt_tokens": random.randint(100, 2000),
})
return logs
operations_logs = generate_logs()
print(json.dumps(operations_logs[:3], indent=2))Step 3: Write the audit system prompt
The system prompt is the agent's instruction manual. It defines the input schema, the statistics to compute, and the exact JSON output shape we expect.
SYSTEM_PROMPT = """You are an LLM Operations Analyst. Your job is to audit a batch of inference logs and produce a structured health report.
Input format: A JSON array of log objects containing timestamp, model, status_code, latency_ms, and prompt_tokens.
Rules:
- Compute aggregate statistics: total requests, error rate, average latency per model.
- Flag any model with an error rate above 5 percent or average latency above 3000 ms as DEGRADED.
- List the top 3 slowest requests.
- Output strictly as a JSON object with keys: summary, per_model_stats, degraded_models, top_slow_requests, recommendations.
- Do not include markdown formatting around the JSON.
"""Step 4: Send logs to Oxlo.ai for analysis
Here is the core audit function. We serialize the logs to JSON, ship them in a single user message, and ask llama-3.3-70b to reason over the entire batch. Because Oxlo.ai uses flat per-request pricing, stuffing a large log payload into the prompt does not inflate cost the way token-based billing would. For current plan details, see https://oxlo.ai/pricing.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
def audit_logs(logs: list) -> dict:
user_message = json.dumps(logs, indent=2)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
response_format={"type": "json_object"},
)
raw = response.choices[0].message.content
return json.loads(raw)
report = audit_logs(operations_logs)
print(json.dumps(report, indent=2))Step 5: Surface actionable alerts
Structured JSON is great for pipelines, but humans need a concise summary. This lightweight wrapper turns the model's findings into plain text alerts.
def print_alerts(report: dict):
degraded = report.get("degraded_models", [])
if not degraded:
print("All models healthy.")
return
for item in degraded:
name = item.get("model", "unknown")
reason = item.get("reason", "unknown issue")
print(f"ALERT: {name} is degraded - {reason}")
print_alerts(report)Run it
Putting it all together, the main block generates thirty logs, audits them, and prints both the full report and the alert summary. Here is the driver script and an example of what the output looks like when the anomaly injection hits.
if __name__ == "__main__":
logs = generate_logs(30)
health_report = audit_logs(logs)
print("=== LLM Ops Report ===")
print(json.dumps(health_report, indent=2))
print("\n=== Alerts ===")
print_alerts(health_report)Example output:
=== LLM Ops Report ===
{
"summary": {
"total_requests": 30,
"overall_error_rate": 0.1
},
"per_model_stats": {
"llama-3.3-70b": {"requests": 8, "avg_latency_ms": 850, "error_rate": 0.0},
"qwen-3-32b": {"requests": 7, "avg_latency_ms": 920, "error_rate": 0.0},
"deepseek-v3.2": {"requests": 8, "avg_latency_ms": 9800, "error_rate": 0.375},
"kimi-k2.6": {"requests": 7, "avg_latency_ms": 780, "error_rate": 0.0}
},
"degraded_models": [
{
"model": "deepseek-v3.2",
"reason": "average latency 9800 ms exceeds 3000 ms threshold, error rate 37.5 percent exceeds 5 percent"
}
],
"top_slow_requests": [
{"timestamp": "2026-01-12T10:33:00", "model": "deepseek-v3.2", "latency_ms": 12000},
{"timestamp": "2026-01-12T10:30:00", "model": "deepseek-v3.2", "latency_ms": 11500},
{"timestamp": "2026-01-12T10:27:00", "model": "deepseek-v3.2", "latency_ms": 11000}
],
"recommendations": [
"Investigate deepseek-v3.2 deployment for capacity constraints or upstream dependency failures."
]
}
=== Alerts ===
ALERT: deepseek-v3.2 is degraded - average latency 9800 ms exceeds 3000 ms threshold, error rate 37.5 percent exceeds 5 percentWrap-up
This agent gives you a working foundation for LLM operations monitoring. Two concrete next steps: schedule this script to run every fifteen minutes against real request logs from your Oxlo.ai usage history or application database, or swap the model to deepseek-r1-671b or kimi-k2.6 when you need deeper chain-of-thought reasoning over longer multi-hour windows. Because Oxlo.ai charges per request rather than per token, those larger context windows stay cost-predictable as your log volume grows.
Top comments (0)