LLM vs Traditional Machine Learning for NLP: A Comparative Analysis

#learnai #oxlo #ai

We are building a support ticket triage agent that classifies incoming messages, extracts order IDs, detects sentiment, and drafts a first response. For teams still maintaining TF-IDF classifiers, CRF extractors, and canned reply templates, this single LLM-based agent collapses the entire NLP stack into one API call.

What you'll need

Python 3.10 or newer
pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai
A few sample support tickets to test with

Step 1: Set up the Oxlo.ai client

The OpenAI SDK works as a drop-in client for Oxlo.ai. I use it because I do not need a custom library or a separate inference server. Create a file named triage.py and add the following.

from openai import OpenAI
import json

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Encode the triage logic in a system prompt

In a traditional ML pipeline, we would train a classifier on labeled tickets, maintain a regex or CRF model for order IDs, and store reply templates in a database. I replace all three layers with one system prompt that defines categories, extraction rules, and tone.

SYSTEM_PROMPT = """You are a support ticket triage agent. For each user message, produce a JSON object with exactly these keys:
- "department": one of ["billing", "technical", "shipping", "general"]
- "urgency": one of ["low", "medium", "high", "critical"]
- "order_id": the order number if found, otherwise null
- "sentiment": one of ["angry", "frustrated", "neutral", "happy"]
- "draft_response": a brief, empathetic first reply

Rules:
- If the user mentions "charge", "refund", or "invoice", route to "billing".
- If the user mentions "bug", "error", "crash", or "login", route to "technical".
- If the user mentions "delivery", "package", or "tracking", route to "shipping".
- Keep the draft_response under 80 words.
- Output ONLY valid JSON with no markdown fences."""

Step 3: Build the inference function

I wrap the API call in a small function so we can reuse it for single tickets or batch jobs. A classical NLP stack would require separate inference calls for classification, NER, and text generation. With Oxlo.ai, one request returns everything, and flat per-request pricing keeps costs predictable even when tickets include long conversation history.

def triage_ticket(ticket_text: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": ticket_text},
        ],
    )
    
    raw = response.choices[0].message.content
    return json.loads(raw)

# Quick sanity check
sample = (
    "I was charged twice for order #49281 and my package still hasn't arrived. "
    "This is unacceptable. I need a refund immediately."
)

print(json.dumps(triage_ticket(sample), indent=2))

Step 4: Process a batch of tickets

Production tickets arrive continuously. This loop handles multiple messages and writes structured results to a JSONL file. Because Oxlo.ai charges per request rather than per token, passing the full thread context does not inflate cost.

tickets = [
    "My login screen is blank after the latest update. I cleared cache but nothing changed.",
    "Order #88201 says delivered but it is not in my mailbox. Can you resend it?",
    "Just wanted to say your onboarding call was fantastic. Keep it up!",
    "I was charged twice for order #49281 and my package still hasn't arrived. "
    "This is unacceptable. I need a refund immediately.",
]

with open("triage_results.jsonl", "w") as f:
    for ticket in tickets:
        try:
            out = triage_ticket(ticket)
            f.write(json.dumps({"ticket": ticket, "result": out}) + "\n")
            print(f"Routed to {out['department']} | urgency: {out['urgency']}")
        except Exception as e:
            print(f"Failed on ticket: {e}")

Run it

Export your key and run the script.

export OXLO_API_KEY="sk-oxlo.ai-..."
python triage.py

Expected output:

Routed to technical | urgency: high
Routed to shipping | urgency: high
Routed to general | urgency: low
Routed to billing | urgency: critical

The triage_results.jsonl file now contains parsed records ready for a CRM, Slack webhook, or dashboard.

Wrap-up

You now have a single-file agent that replaces three traditional ML models. If you need stronger reasoning for ambiguous tickets, swap the model to kimi-k2.6 or deepseek-v3.2 on Oxlo.ai without changing any client code. For teams handling high volumes of long-context tickets, flat per-request pricing removes the surprise cost spikes common with token-based billing.

DEV Community