shashank ms

Posted on Sep 3

Deploying LLM on a Server

#engineering #oxlo #ai

I needed a private API that could accept raw server logs and return structured incident analysis without sending data to a managed chat UI. In this tutorial, we will build exactly that: a containerized FastAPI service that forwards log analysis to Oxlo.ai. Because Oxlo.ai charges per request instead of per token, dumping multi-thousand-line logs into the prompt does not explode the cost.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK installed via pip install openai
Docker for the containerization step

Step 1: Scaffold the project

Create a virtual environment and install the dependencies. Then store your Oxlo.ai API key in a .env file.

mkdir log-analysis-api && cd log-analysis-api
python3 -m venv venv && source venv/bin/activate
pip install openai fastapi uvicorn pydantic python-dotenv

OXLO_API_KEY=YOUR_OXLO_API_KEY

Step 2: Define the system prompt

The agent needs strict instructions to return valid JSON. I keep the prompt in its own variable so it is easy to tweak without touching the route logic.

SYSTEM_PROMPT = """You are a senior site reliability engineer. 
Analyze the provided server logs and return a structured JSON object with three fields:
- summary: a one-sentence description of the issue
- severity: one of low, medium, high, or critical
- root_cause: the most likely root cause
Be concise and factual. Do not speculate beyond what the logs show."""

Step 3: Build the FastAPI server

I wire everything together in main.py. The Oxlo.ai client uses the OpenAI-compatible base URL, and the endpoint calls llama-3.3-70b with the system prompt and log payload.

import os
import json
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"],
)

SYSTEM_PROMPT = """You are a senior site reliability engineer. 
Analyze the provided server logs and return a structured JSON object with three fields:
- summary: a one-sentence description of the issue
- severity: one of low, medium, high, or critical
- root_cause: the most likely root cause
Be concise and factual. Do not speculate beyond what the logs show."""

app = FastAPI()

class LogRequest(BaseModel):
    logs: str

class AnalysisResponse(BaseModel):
    summary: str
    severity: str
    root_cause: str

@app.post("/analyze", response_model=AnalysisResponse)
async def analyze_logs(req: LogRequest):
    if not req.logs.strip():
        raise HTTPException(status_code=400, detail="Empty log payload")
    
    try:
        response = client.chat.completions.create(
            model="llama-3.3-70b",
            messages=[
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": req.logs},
            ],
        )
        content = response.choices[0].message.content
        parsed = json.loads(content)
        return AnalysisResponse(**parsed)
    except json.JSONDecodeError:
        raise HTTPException(status_code=502, detail="Invalid JSON from model")
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

Step 4: Containerize for deployment

I containerize the service so it can deploy to any server without managing GPUs or model weights. The Dockerfile uses a slim Python image and passes the API key at runtime.

Create requirements.txt:

openai
fastapi
uvicorn[standard]
pydantic

Create a Dockerfile:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

ENV PYTHONUNBUFFERED=1
EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Build and run:

docker build -t log-analysis-api .
docker run -p 8000:8000 -e OXLO_API_KEY=$OXLO_API_KEY log-analysis-api

Run it

With the container running, send a test payload. I trimmed real nginx error logs for the demo.

curl -X POST http://localhost:8000/analyze \
  -H "Content-Type: application/json" \
  -d '{"logs": "2024-05-20T14:32:10Z ERROR upstream timed out (110: Connection timed out)\n2024-05-20T14:32:11Z ERROR connect() failed (111: Connection refused)\n2024-05-20T14:32:15Z WARN upstream server temporarily disabled"}'

The endpoint should return structured JSON immediately.

Example output:

{
  "summary": "Upstream backend is unreachable, causing connection timeouts and nginx failovers.",
  "severity": "high",
  "root_cause": "Backend service is either down or rejecting connections on the expected port."
}

Next steps

Add an async Redis cache in front of the /analyze endpoint to avoid re-analyzing identical log batches. If your team handles multilingual logs, swap the model string to qwen-3-32b or kimi-k2.6 with no other code changes. You can review Oxlo.ai request-based pricing at https://oxlo.ai/pricing to estimate cost as you scale traffic.

DEV Community