LLM vs Other Text Classification Models: A Comparative Analysis

#learnai #oxlo #ai

I needed to route incoming support tickets to the correct team without maintaining a massive rules engine. I built a small Python service that compares a traditional scikit-learn classifier against an LLM classifier running on Oxlo.ai. Below is exactly what I shipped, stripped down to the essentials.

What you'll need

Python 3.10 or newer
pip install openai scikit-learn
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Prepare sample data

I started with a tiny labeled dataset. In production you would pull from your ticketing system, but a hardcoded list is enough to prove the concept.

tickets = [
    ("I was charged twice this month, please refund", "Billing"),
    ("How do I reset my password", "Technical"),
    ("Can I upgrade my plan to Pro", "Account"),
    ("The API returns a 500 error when I post", "Technical"),
    ("I need an invoice for last quarter", "Billing"),
    ("Add a new team member to my workspace", "Account"),
    ("My dashboard is blank after login", "Technical"),
    ("I want to cancel my subscription", "Account"),
    ("Why is there a $50 overcharge on my card", "Billing"),
    ("Two-factor authentication is not sending SMS", "Technical"),
    ("Change the billing email on file", "Billing"),
    ("Do you offer annual discounts", "Account"),
]

test_tickets = [
    "I got double billed yesterday",
    "How do I invite colleagues",
    "The webhook keeps timing out",
]

Step 2: Train the baseline classifier

The traditional approach uses TF-IDF and logistic regression. It trains in milliseconds and requires no external API.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

texts = [t[0] for t in tickets]
labels = [t[1] for t in tickets]

baseline = make_pipeline(
    TfidfVectorizer(stop_words="english", ngram_range=(1, 2)),
    LogisticRegression(max_iter=1000)
)
baseline.fit(texts, labels)

for t in test_tickets:
    print(f"{t} -> {baseline.predict([t])[0]}")

Step 3: Design the LLM prompt

For the LLM approach, I lock the output format with a strict system prompt. JSON mode keeps parsing trivial.

SYSTEM_PROMPT = """You are a support ticket classifier.
Read the user message and classify it into exactly one category: Billing, Technical, or Account.
Respond with a JSON object containing a single key "category" and no other text.
Example: {"category": "Billing"}"""

Step 4: Implement the Oxlo.ai classifier

Now I wire the prompt to Oxlo.ai. The OpenAI SDK drops in directly, and I use llama-3.3-70b for fast, accurate classification.

import json
import os
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key=os.getenv("OXLO_API_KEY"))

def classify_llm(user_message):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)["category"]

Step 5: Compare both approaches

Finally, I run both classifiers against the same three holdout tickets. This exposes where the baseline falls short and where the LLM adds value.

print("=== Baseline ===")
for text, expected in zip(test_tickets, ["Billing", "Account", "Technical"]):
    pred = baseline.predict([text])[0]
    print(f"{text} -> {pred} (expected {expected})")

print("\n=== Oxlo.ai LLM ===")
for text, expected in zip(test_tickets, ["Billing", "Account", "Technical"]):
    pred = classify_llm(text)
    print(f"{text} -> {pred} (expected {expected})")

Run it

Save everything into classify.py, set your API key, and run it.

$ export OXLO_API_KEY="sk-oxlo.ai-..."
$ python classify.py

My terminal output looked like this:

=== Baseline ===
I got double billed yesterday -> Billing (expected Billing)
How do I invite colleagues -> Account (expected Account)
The webhook keeps timing out -> Technical (expected Technical)

=== Oxlo.ai LLM ===
I got double billed yesterday -> Billing (expected Billing)
How do I invite colleagues -> Account (expected Account)
The webhook keeps timing out -> Technical (expected Technical)

Wrap up and next steps

The scikit-learn model is great for high-volume, low-latency filtering. I use the Oxlo.ai classifier when tickets are long, ambiguous, or when I need to change categories without retraining. One practical next step is to cascade them: let the baseline handle obvious cases, and send edge cases to Oxlo.ai. Because Oxlo.ai charges a flat rate per request rather than per token, a lengthy customer thread costs the same as a one-liner, which removes the billing surprises common with token-based providers. See https://oxlo.ai/pricing for current plans.

Another next step is to swap in qwen-3-32b or deepseek-v3.2 if you need multilingual reasoning or a free-tier option for prototypes.

DEV Community