LLM vs Traditional Machine Learning Models for NLP

#learnai #oxlo #ai

Last quarter I shipped a hybrid support ticket triage pipeline that routes tickets with a traditional TF-IDF classifier and drafts replies with an LLM. It is a concrete way to see where classical NLP ends and large language models take over. If your team is drowning in tier-1 support volume, you can build and run this in an afternoon.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai scikit-learn

1. Generate sample data

I will build a tiny synthetic dataset so the script is fully runnable without external CSVs. Three categories are enough to show the difference between a statistical classifier and a generative model.

import random

random.seed(42)

billing_templates = [
    "I was charged twice for my subscription this month.",
    "My invoice shows the wrong amount, can you fix it?",
    "I need a refund for my last payment.",
    "Why did my card get charged after I cancelled?",
    "Update my billing address and resend the receipt."
]

technical_templates = [
    "The API returns a 500 error when I post to /v1/users.",
    "My dashboard is blank after logging in.",
    "Webhooks are not firing on order completion.",
    "I cannot export my data to CSV, it times out.",
    "The mobile app crashes when I open settings."
]

account_templates = [
    "I forgot my password and the reset email never arrives.",
    "Please delete my account and all associated data.",
    "I want to change my email address on file.",
    "Add a new team member to our workspace.",
    "My two-factor authentication is broken."
]

tickets = []
labels = []

for _ in range(30):
    tickets.append(random.choice(billing_templates))
    labels.append("billing")
    tickets.append(random.choice(technical_templates))
    labels.append("technical")
    tickets.append(random.choice(account_templates))
    labels.append("account")

2. Train a traditional classifier

A TF-IDF vectorizer plus logistic regression is fast, deterministic, and runs entirely offline. This is the baseline that traditional machine learning still excels at: structured classification with small data.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

traditional_pipeline = make_pipeline(
    TfidfVectorizer(stop_words="english", max_features=100),
    LogisticRegression(max_iter=200)
)

traditional_pipeline.fit(tickets, labels)

sample_ticket = "I got billed twice after upgrading my plan."
predicted_category = traditional_pipeline.predict([sample_ticket])[0]
print(f"Traditional model prediction: {predicted_category}")

3. Define the LLM system prompt

The LLM handles tone, context, and next steps. I keep the prompt explicit so it acts like a senior support rep rather than a chatbot.

SYSTEM_PROMPT = """You are a senior support analyst.
A traditional classifier has already routed this ticket to the '{category}' queue.
Your job is to draft a concise, empathetic first response.
Acknowledge the issue, state one clear next step, and ask any single follow-up question if needed.
Keep the reply under 120 words."""

4. Call the LLM through Oxlo.ai

Now I send the ticket to Oxlo.ai. I use Llama 3.3 70B because it is reliable for structured support copy, and I inject the predicted category into the system prompt so the model knows which playbook to follow.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def draft_llm_reply(ticket_text, category):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT.format(category=category)},
            {"role": "user", "content": f"Ticket: {ticket_text}"},
        ],
        temperature=0.4,
        max_tokens=200,
    )
    return response.choices[0].message.content

llm_reply = draft_llm_reply(sample_ticket, predicted_category)
print(f"LLM reply:\n{llm_reply}")

5. Wrap everything into one pipeline

I combine the deterministic classifier with the generative model. The classifier gives millisecond-level routing, and the LLM gives human-quality text. This is where Oxlo.ai's request-based pricing matters, because even if you pass long ticket threads into the context, the cost stays flat per API call.

def triage_and_respond(ticket_text):
    category = traditional_pipeline.predict([ticket_text])[0]
    reply = draft_llm_reply(ticket_text, category)
    return category, reply

if __name__ == "__main__":
    tickets_to_test = [
        "I was charged twice after I changed my plan. Please help.",
        "The export button spins forever and nothing downloads.",
        "My 2FA codes are not working and I am locked out."
    ]

    for t in tickets_to_test:
        cat, reply = triage_and_respond(t)
        print(f"---\nTicket: {t}\nCategory: {cat}\nReply: {reply}\n")

Run it

Save the script as support_pipeline.py, replace YOUR_OXLO_API_KEY with your key from https://portal.oxlo.ai, then run python support_pipeline.py. You should see output similar to this:

Traditional model prediction: billing
LLM reply:
Hi there,

I'm sorry to see you were charged twice after changing your plan. I've flagged this for our billing team to review and issue a refund if confirmed.

In the meantime, could you share the transaction IDs from your email receipt? That will help us expedite the correction.

Best,
Support Team

---
Ticket: The export button spins forever and nothing downloads.
Category: technical
Reply:
Hi there,

Thanks for reporting this. A spinning export usually points to a timeout on large datasets.

Could you try filtering the date range to the last seven days and attempting the export again? If it still fails, please share your workspace ID so we can check the server logs.

Best,
Support Team

---
Ticket: My 2FA codes are not working and I am locked out.
Category: account
Reply:
Hi there,

I understand how stressful being locked out can be. Our account team can verify your identity and temporarily disable 2FA.

Please reply with a photo of your government-issued ID and the email address on the account. We will prioritize this and get you back in within the hour.

Best,
Support Team

Wrap-up

Swap the traditional classifier for a lightweight embedding model like BGE-Large through Oxlo.ai if you need semantic similarity instead of bag-of-words routing. If your ticket volume grows, the flat per-request pricing on Oxlo.ai keeps long-context reruns predictable while you iterate on the system prompt.

DEV Community