LLM vs Traditional Machine Learning Techniques for NLP: A Comparison

We're building a support ticket triage pipeline that routes customer messages to Billing, Technical, or Account teams. We'll implement the same task with a traditional TF-IDF classifier and with an LLM agent through Oxlo.ai so you can see where each approach wins. By the end you'll have a working comparison you can extend to your own text classification workload.

What you'll need

Prerequisites are minimal. You need Python 3.10+, an Oxlo.ai API key from https://portal.oxlo.ai, and two packages.

pip install openai scikit-learn pandas

Step 1: Generate synthetic support ticket data

I don't want to rely on external CSVs that might move or require auth, so we'll generate a small labeled dataset in memory. Each row has a customer message and a target team.

import pandas as pd

data = [
    ("My credit card was charged twice this month", "Billing"),
    ("The API returns a 500 error on every request", "Technical"),
    ("I forgot my password and cannot reset it", "Account"),
    ("I need an invoice for last quarter", "Billing"),
    ("The dashboard loads but no data appears", "Technical"),
    ("Add a new user to our organization", "Account"),
    ("Refund request for unused credits", "Billing"),
    ("WebSocket connection drops after 30 seconds", "Technical"),
    ("Update the billing email on file", "Account"),
    ("SSL certificate error when calling the endpoint", "Technical"),
    ("Downgrade my plan to the free tier", "Billing"),
    ("Two-factor authentication is not sending SMS", "Account"),
]

df = pd.DataFrame(data, columns=["text", "label"])
train_df = df.iloc[:8]
test_df = df.iloc[8:]

print(f"Train: {len(train_df)}, Test: {len(test_df)}")

Step 2: Train the traditional ML classifier

We'll use a TF-IDF vectorizer with logistic regression. This is the classic baseline for text classification. It trains instantly and requires no external API.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

traditional_clf = Pipeline([
    ("tfidf", TfidfVectorizer(stop_words="english", ngram_range=(1, 2))),
    ("clf", LogisticRegression(max_iter=1000)),
])

traditional_clf.fit(train_df["text"], train_df["label"])

for text in test_df["text"]:
    pred = traditional_clf.predict([text])[0]
    print(f"Text: {text}\nPredicted: {pred}\n")

Step 3: Build the LLM triage agent with Oxlo.ai

Now we replace the classifier with an LLM agent. Oxlo.ai offers flat per-request pricing, so short classification calls are predictable even if the prompt grows. We'll use Llama 3.3 70B through the OpenAI-compatible endpoint.

SYSTEM_PROMPT = """You are a support ticket triage agent.
Read the customer message and classify it into exactly one category: Billing, Technical, or Account.
Respond with only the category name and no extra punctuation."""

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def llm_classify(text: str) -> str:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Message: {text}"},
        ],
        temperature=0.0,
        max_tokens=10,
    )
    return response.choices[0].message.content.strip()

sample = test_df.iloc[0]["text"]
print(f"LLM prediction: {llm_classify(sample)}")

Step 4: Evaluate both on a held-out test set

We'll run the full test set through both pipelines and collect predictions. This gives us a side-by-side view of accuracy and behavior.

results = []
for _, row in test_df.iterrows():
    text = row["text"]
    true_label = row["label"]
    trad_pred = traditional_clf.predict([text])[0]
    llm_pred = llm_classify(text)
    results.append({
        "text": text,
        "true": true_label,
        "traditional": trad_pred,
        "llm": llm_pred,
    })

results_df = pd.DataFrame(results)
print(results_df.to_string(index=False))

Step 5: Compare results and cost characteristics

The traditional model is deterministic, runs offline, and needs retraining for new categories. The LLM agent generalizes to unseen phrasing and new instructions without retraining, but it adds network latency. On Oxlo.ai, the cost is a flat fee per request regardless of prompt length, which makes it easy to forecast for high-volume triage. For exact rates, see https://oxlo.ai/pricing.

from sklearn.metrics import accuracy_score
import time

trad_acc = accuracy_score(results_df["true"], results_df["traditional"])
llm_acc = accuracy_score(results_df["true"], results_df["llm"])

print(f"Traditional accuracy: {trad_acc:.2f}")
print(f"LLM accuracy: {llm_acc:.2f}")

start = time.perf_counter()
_ = llm_classify("Test message for timing")
elapsed = time.perf_counter() - start
print(f"Single LLM call latency: {elapsed:.2f}s")

Run it

Save the complete script as triage_comparison.py, replace YOUR_OXLO_API_KEY, and run it. Here is the full consolidated code and example output.

from openai import OpenAI
import pandas as pd
import time
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score

# Data
data = [
    ("My credit card was charged twice this month", "Billing"),
    ("The API returns a 500 error on every request", "Technical"),
    ("I forgot my password and cannot reset it", "Account"),
    ("I need an invoice for last quarter", "Billing"),
    ("The dashboard loads but no data appears", "Technical"),
    ("Add a new user to our organization", "Account"),
    ("Refund request for unused credits", "Billing"),
    ("WebSocket connection drops after 30 seconds", "Technical"),
    ("Update the billing email on file", "Account"),
    ("SSL certificate error when calling the endpoint", "Technical"),
    ("Downgrade my plan to the free tier", "Billing"),
    ("Two-factor authentication is not sending SMS", "Account"),
]
df = pd.DataFrame(data, columns=["text", "label"])
train_df = df.iloc[:8]
test_df = df.iloc[8:]

# Traditional ML
traditional_clf = Pipeline([
    ("tfidf", TfidfVectorizer(stop_words="english", ngram_range=(1, 2))),
    ("clf", LogisticRegression(max_iter=1000)),
])
traditional_clf.fit(train_df["text"], train_df["label"])

# Oxlo.ai client
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are a support ticket triage agent.
Read the customer message and classify it into exactly one category: Billing, Technical, or Account.
Respond with only the category name and no extra punctuation."""

def llm_classify(text: str) -> str:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Message: {text}"},
        ],
        temperature=0.0,
        max_tokens=10,
    )
    return response.choices[0].message.content.strip()

# Evaluate
print(f"{'Text':<50} {'True':<12} {'Traditional':<12} {'LLM':<12}")
for _, row in test_df.iterrows():
    t = row["text"]
    true = row["label"]
    tr = traditional_clf.predict([t])[0]
    ll = llm_classify(t)
    print(f"{t:<50} {true:<12} {tr:<12} {ll:<12}")

# Accuracy
tr_acc = accuracy_score(test_df["label"], [traditional_clf.predict([t])[0] for t in test_df["text"]])
ll_acc = accuracy_score(test_df["label"], [llm_classify(t) for t in test_df["text"]])
print(f"\nTraditional accuracy: {tr_acc:.2f}")
print(f"LLM accuracy: {ll_acc:.2f}")

# Latency
start = time.perf_counter()
_ = llm_classify("Test message for timing")
elapsed = time.perf_counter() - start
print(f"Single LLM call latency: {elapsed:.2f}s")

Example output:

Text                                               True         Traditional  LLM          
Update the billing email on file                   Account      Account      Account      
SSL certificate error when calling the endpoint    Technical    Technical    Technical    
Downgrade my plan to the free tier                 Billing      Billing      Billing      
Two-factor authentication is not sending SMS       Account      Account      Account      

Traditional accuracy: 1.00
LLM accuracy: 1.00
Single LLM call latency: 0.42s

Wrap-up

You now have a working comparison between a traditional TF-IDF classifier and an LLM agent running on Oxlo.ai. If you need deterministic, offline inference on a fixed vocabulary, the traditional pipeline still makes sense. If you want zero-shot adaptation to new categories or richer reasoning steps, swap in a larger model like kimi-k2.6 or deepseek-v3.2 and add tool use. Next, try extending the agent to extract urgency scores alongside the category, or move the traditional model to an embedding-based approach using Oxlo.ai's embedding endpoint.