Real-Time Applications with LLMs on Oxlo

#learnai #oxlo #ai

We are going to build a real-time support ticket triage agent that reads incoming messages, scores urgency, and streams a draft reply back to the terminal. If you run a support queue, this cuts first-response time from minutes to seconds.

What you'll need

Python 3.10 or newer.
An Oxlo.ai API key from https://portal.oxlo.ai.
The OpenAI SDK: pip install openai.

Step 1: Configure the Oxlo.ai client

Create a file named triage.py and initialize the client. Oxlo.ai is fully OpenAI SDK compatible, so the only difference is the base_url.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

Step 2: Define the system prompt

The system prompt keeps classification consistent and replies on-brand. Treat this as a config file you iterate on.

SYSTEM_PROMPT = """You are a support triage agent.
For each incoming ticket, output strictly valid JSON with these keys:
- urgency: one of Low, Medium, or High
- category: one of Billing, Technical, or General
- draft_reply: a concise, empathetic first response to the customer
"""

Step 3: Extract structured triage data

We use JSON mode to get machine-readable labels before we stream anything. This keeps the downstream logic deterministic.

import json

def extract_triage(ticket_text: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": ticket_text},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
    )
    return json.loads(response.choices[0].message.content)

Step 4: Stream the draft reply in real time

For the real-time experience, we stream the draft reply token by token. Because Oxlo.ai uses flat request-based pricing, long streaming outputs do not inflate cost the way token-based billing would. You can see exact pricing at https://oxlo.ai/pricing.

def stream_reply(ticket_text: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": "You are a helpful support agent. Draft a concise, empathetic first reply."},
            {"role": "user", "content": ticket_text},
        ],
        stream=True,
        temperature=0.7,
    )
    print("Draft reply: ", end="", flush=True)
    for chunk in response:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)
    print()

Step 5: Wire the real-time loop

This loop simulates a live queue. In production you would replace the list with a webhook or message broker.

if __name__ == "__main__":
    tickets = [
        "I was charged twice for my subscription this month. Please fix this immediately.",
        "How do I reset my API key?",
        "The dashboard is completely blank after I log in. This is blocking our deploy."
    ]

    for ticket in tickets:
        print(f"\n--- New Ticket ---\n{ticket}")
        triage = extract_triage(ticket)
        print(f"Urgency: {triage['urgency']} | Category: {triage['category']}")

        if triage["urgency"] == "High":
            print("ALERT: High urgency detected. Streaming draft reply...")

        stream_reply(ticket)

Run it

Export your key and run the script.

export OXLO_API_KEY="your_key_here"
python triage.py

Example output:

--- New Ticket ---
I was charged twice for my subscription this month. Please fix this immediately.
Urgency: High | Category: Billing
ALERT: High urgency detected. Streaming draft reply...
Draft reply: I am sorry to see the double charge. I have flagged this with our billing team and you will see a refund within 24 hours.

--- New Ticket ---
How do I reset my API key?
Urgency: Low | Category: Technical
Draft reply: You can reset your API key from the Settings page under Security. Let me know if you need a direct link.

--- New Ticket ---
The dashboard is completely blank after I log in. This is blocking our deploy.
Urgency: High | Category: Technical
ALERT: High urgency detected. Streaming draft reply...
Draft reply: I understand this is blocking your deploy. Can you try a hard refresh and confirm your browser version? I am escalating this to our engineering team now.

Wrap-up and next steps

This agent gives you a foundation. Two concrete ways to extend it:

Add function calling to update a CRM or post high-urgency alerts to Slack. Oxlo.ai supports tool use on models like llama-3.3-70b and kimi-k2.6.
If your tickets include screenshots, swap the model to kimi-k2.6 and pass image URLs in the messages array. Vision input works through the same chat completions endpoint.

DEV Community