shashank ms

Posted on Aug 7

Implementing LLM in Production: A Step-by-Step Guide

#engineering #oxlo #ai

We are going to build a production incident triage agent that ingests raw application logs, classifies severity, and returns a structured JSON report. This saves on-call engineers from scrolling through thousands of lines of noise during an outage. Because the agent handles large log dumps in a single request, Oxlo.ai's flat per-request pricing keeps costs predictable regardless of input length.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Configure the Oxlo.ai client

I start every project by configuring the client so the base URL points to Oxlo.ai instead of OpenAI. This is a drop-in replacement.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Lock the system prompt

The system prompt is the only part of the agent I treat as config. It forces JSON output and defines the schema the rest of the pipeline expects.

SYSTEM_PROMPT = """You are a production incident triage agent. Analyze the provided application logs and produce a structured JSON incident report with exactly these keys:
- summary: a one-sentence description of the issue
- severity: one of critical, warning, or info
- root_cause: a short explanation of what likely caused the issue
- next_steps: a list of concrete remediation steps

Respond ONLY with valid JSON. Do not include markdown formatting or explanations outside the JSON."""

Step 3: Build the triage function

Now I wire the call to Oxlo.ai. I use llama-3.3-70b as the default because it handles structured instruction well, and I keep temperature low to reduce hallucinated keys.

import json

def triage_logs(log_blob: str, model: str = "llama-3.3-70b") -> dict:
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Application logs:\n\n{log_blob}"},
        ],
        temperature=0.1,
    )
    content = response.choices[0].message.content
    return json.loads(content)

Step 4: Harden with validation

Production code cannot crash because a model added a markdown fence. I wrap the parser in validation that falls back to a safe dict if JSON decoding fails or keys are missing.

def safe_triage(log_blob: str, model: str = "llama-3.3-70b") -> dict:
    try:
        report = triage_logs(log_blob, model)
        required = {"summary", "severity", "root_cause", "next_steps"}
        if not required.issubset(report.keys()):
            raise ValueError(f"Missing keys: {required - set(report.keys())}")
        return report
    except Exception as exc:
        return {
            "summary": "Failed to parse model output",
            "severity": "warning",
            "root_cause": str(exc),
            "next_steps": ["Review logs manually"],
        }

Step 5: Add severity routing

I add a small router so critical issues print to stderr and can be piped to PagerDuty or Slack. The stdout stream stays clean for the JSON report.

import sys

def route(report: dict) -> None:
    severity = report.get("severity", "info")
    if severity == "critical":
        print("ALERT: paging on-call engineer", file=sys.stderr)
    elif severity == "warning":
        print("NOTICE: filing low-priority ticket", file=sys.stderr)
    print(json.dumps(report, indent=2))

Step 6: Wire the CLI

Finally, I wire a minimal CLI that reads a log file and runs the pipeline. This is the entrypoint I deploy as a container in my cluster.

if __name__ == "__main__":
    import argparse

    parser = argparse.ArgumentParser(description="Triage production logs via Oxlo.ai")
    parser.add_argument("logfile", help="Path to raw log file")
    args = parser.parse_args()

    with open(args.logfile, "r", encoding="utf-8") as f:
        logs = f.read()

    report = safe_triage(logs)
    route(report)

Run it

Create a file named app.log with a stack trace or error noise, then run the script. The agent returns structured data you can feed into runbooks.

$ python triage.py app.log
ALERT: paging on-call engineer
{
  "summary": "Database connection pool exhausted after spike in checkout requests",
  "severity": "critical",
  "root_cause": "The checkout service opened more connections than the pool limit without releasing them.",
  "next_steps": [
    "Restart checkout pods to release connections",
    "Increase max pool size in config",
    "Add connection timeout retry logic"
  ]
}

Wrap-up

Two concrete next steps. First, expose the agent as a FastAPI endpoint and stream responses back to your internal dashboard using Oxlo.ai's streaming support. Second, switch the model to qwen-3-32b if you need to triage multilingual logs, or to deepseek-v3.2 if you want deeper reasoning on complex stack traces. Check Oxlo.ai's pricing page to see how flat per-request billing keeps long-context triage affordable.

DEV Community