LLMOps: The Future of DevOps with Large Language Models

#engineering #oxlo #ai

We are going to build an on-call incident triage agent on Oxlo.ai that reads synthetic service telemetry and returns a structured severity assessment with remediation steps. It helps platform teams automate the first five minutes of an incident investigation and cut down alert fatigue.

What you'll need

Before we start, make sure you have the following ready:

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Set up the Oxlo.ai client

I configure the client once at the module level and reuse it for every request. Oxlo.ai exposes a fully OpenAI-compatible endpoint, so the standard SDK works without any adapter code.

from openai import OpenAI
import json

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

Step 2: Define the system prompt

The system prompt is the agent's job description. I keep it explicit about output format so I can parse the response programmatically downstream.

SYSTEM_PROMPT = """You are an expert Site Reliability Engineer. Your job is to triage incoming incidents based on the provided telemetry.

Analyze the service metrics, error logs, and deployment history. Respond with a single JSON object containing exactly these keys:
- severity: one of "critical", "high", "medium", "low"
- summary: a one-sentence description of the issue
- root_cause: your best guess at the underlying cause
- remediation_steps: an ordered list of actionable strings
- affected_services: a list of service names that might be impacted

Be concise. Do not include markdown formatting or explanation outside the JSON."""

Step 3: Simulate telemetry ingestion

In production, this helper would query Prometheus, Datadog, or Splunk. For this tutorial, I wrote a small function that returns realistic telemetry for a fictional microservice.

def fetch_telemetry(service: str):
    # Simulated data source. Replace with real API calls to your observability stack.
    return {
        "service": service,
        "timestamp": "2025-01-15T14:32:00Z",
        "cpu_percent": 94.2,
        "memory_percent": 87.5,
        "error_rate_5m": 12.4,
        "p99_latency_ms": 2300,
        "recent_deploy": "payment-service:v2.3.1 deployed 14 minutes ago",
        "error_logs": [
            "Connection timeout to inventory-db after 3000ms",
            "Retry exhaustion on /api/v1/charge"
        ]
    }

Step 4: Build the user message

I format the telemetry into a plain text report before handing it to the model. Keeping the message structured but human-readable improves parsing reliability.

def build_user_message(telemetry: dict) -> str:
    lines = [
        f"Service: {telemetry['service']}",
        f"Time: {telemetry['timestamp']}",
        f"CPU: {telemetry['cpu_percent']}%",
        f"Memory: {telemetry['memory_percent']}%",
        f"5m Error Rate: {telemetry['error_rate_5m']}%",
        f"P99 Latency: {telemetry['p99_latency_ms']}ms",
        f"Recent Deploy: {telemetry['recent_deploy']}",
        "Recent Error Logs:",
    ]
    for log in telemetry["error_logs"]:
        lines.append(f"  - {log}")
    return "\n".join(lines)

Step 5: Run the triage agent

Now I wire everything together. I send the formatted telemetry to Llama 3.3 70B on Oxlo.ai and parse the result as JSON. Because the system prompt locks the output format, I can treat the response like an API contract.

def triage(service: str):
    telemetry = fetch_telemetry(service)
    user_message = build_user_message(telemetry)

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )

    content = response.choices[0].message.content.strip()
    return json.loads(content)

if __name__ == "__main__":
    result = triage("payment-api")
    print(json.dumps(result, indent=2))

Run it

Executing the script against the simulated payment-api incident produces a structured report immediately. There are no cold starts on Llama 3.3 70B through Oxlo.ai, so the first request returns just as fast as the tenth.

Example output:

{
  "severity": "critical",
  "summary": "Payment API is experiencing connection timeouts and retry exhaustion following a recent deployment.",
  "root_cause": "The v2.3.1 deploy likely introduced a regression in database connection pooling or timeout configuration.",
  "remediation_steps": [
    "Check connection pool settings in payment-service:v2.3.1",
    "Verify inventory-db health and network latency",
    "Consider rolling back to payment-service:v2.3.0",
    "Scale payment-api replicas horizontally if db connections are exhausted"
  ],
  "affected_services": [
    "payment-api",
    "inventory-db"
  ]
}

Wrap up

That is the core of an LLMOps incident triage pipeline. A solid next step is to wrap this function in a FastAPI handler and connect it to your PagerDuty webhooks so every high-priority page triggers an automatic pre-analysis. You could also store past incident reports in a vector database and prepend the three most similar historical cases to the system prompt for few-shot context on Oxlo.ai.

DEV Community