LLM vs Traditional NLP Techniques: A Comparison

#learnai #oxlo #ai

I recently rebuilt an internal support ticket triage tool that used a patchwork of scikit-learn classifiers and spaCy pipelines. Replacing the whole stack with a single LLM call through Oxlo.ai cut the code by half and handled edge cases the old pipeline missed. In this tutorial, I will walk through both versions so you can see exactly where each approach breaks or shines.

What you'll need

Python 3.10 or newer, the OpenAI SDK, and a small local ML stack for the baseline. You will also need an Oxlo.ai API key from https://portal.oxlo.ai.

pip install openai scikit-learn spacy
python -m spacy download en_core_web_sm
An Oxlo.ai API key. The free tier gives you 60 requests per day, which is enough to run this comparison multiple times.

Step 1: Build the traditional NLP baseline

Traditional NLP forces you to chain separate tools. I set up a TF-IDF classifier for urgency, spaCy for entity recognition, and a keyword matcher for sentiment. It runs locally and is easy to audit, but every stage needs its own training data or rule set.

import json
import spacy
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression

# Load spaCy for entity extraction
nlp = spacy.load("en_core_web_sm")

# Minimal training data for the urgency classifier
train_texts = [
    "My refund is late and I am furious",
    "How do I reset my password?",
    "The app crashes when I upload a photo",
    "I love the new design, great work",
    "Billing issue: double charged this month",
]
train_labels = ["urgent", "low", "urgent", "low", "urgent"]

classifier = make_pipeline(TfidfVectorizer(), LogisticRegression(max_iter=1000))
classifier.fit(train_texts, train_labels)

def traditional_pipeline(ticket: str):
    # 1. Classify urgency
    urgency = str(classifier.predict([ticket])[0])

    # 2. Extract products with spaCy NER
    doc = nlp(ticket)
    products = [ent.text for ent in doc.ents if ent.label_ == "PRODUCT"]

    # 3. Rule-based sentiment
    negative_words = ["refund", "crash", "bug", "broken", "angry", "furious", "issue", "failed", "loss"]
    sentiment = "negative" if any(w in ticket.lower() for w in negative_words) else "neutral"

    # 4. Template response
    if urgency == "urgent":
        subject = products[0] if products else "your request"
        response = f"We have escalated your issue regarding {subject} to our support team."
    else:
        response = "Thank you for reaching out. We will look into this shortly."

    return {
        "method": "traditional_nlp",
        "urgency": urgency,
        "products": products,
        "sentiment": sentiment,
        "response": response,
    }

Step 2: Design the unified agent prompt

Instead of three separate models, the LLM agent uses one system prompt to classify urgency, extract products, detect sentiment, and draft a reply. Keeping the instructions explicit keeps the output deterministic enough for production.

SYSTEM_PROMPT = """You are a support ticket triage agent. Analyze the user message and output a JSON object with exactly these keys:
- urgency: either "urgent" or "low"
- products: array of product or service names mentioned (empty array if none)
- sentiment: "positive", "negative", or "neutral"
- response: a concise, helpful draft reply addressing the user's issue

Rules:
- If the user mentions billing, refunds, crashes, security, or business loss, classify as urgent.
- Extract specific product names like "Premium Plan", "mobile app", or "API".
- Keep the response under 50 words.
- Output ONLY valid JSON, no markdown fences."""

Step 3: Wire up the Oxlo.ai client

Oxlo.ai exposes an OpenAI-compatible API, so the client setup is a one-line base URL change. I use llama-3.3-70b here because it follows structured instructions reliably. Because Oxlo.ai charges per request rather than per token, the long system prompt does not inflate cost the way it would on token-based providers. You can see the exact pricing at https://oxlo.ai/pricing.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
)

Step 4: Build the LLM agent class

The agent formats the ticket into a user message and parses the JSON response. I added a small helper to strip accidental markdown fences so the parser does not crash.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def llm_agent(ticket: str):
    resp = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": ticket},
        ],
    )
    content = resp.choices[0].message.content.strip()
    if content.startswith("

```"):
        content = content.split("\n", 1)[1].rsplit("```

", 1)[0].strip()
    return json.loads(content)

Step 5: Assemble the comparison harness

Now I feed the same ambiguous ticket through both pipelines. The traditional stack sees each step in isolation, while the LLM agent reasons about context across the whole message. This is where the gap becomes obvious.

if __name__ == "__main__":
    ticket = (
        "The Premium Plan renewal failed and my team lost access to the API "
        "during a client demo. This is costing us a deal."
    )

    print("=== Traditional NLP ===")
    print(json.dumps(traditional_pipeline(ticket), indent=2))

    print("\n=== LLM Agent (Oxlo.ai) ===")
    print(json.dumps(llm_agent(ticket), indent=2))

Run it

Save the full script as compare.py and run python compare.py. On a typical run, the traditional pipeline misclassifies the urgency as low because the TF-IDF vectorizer was never trained on business-impact language, and it often misses the product entities. The Oxlo.ai LLM agent catches the context and drafts a specific escalation response.

$ python compare.py

=== Traditional NLP ===
{
  "method": "traditional_nlp",
  "urgency": "low",
  "products": [],
  "sentiment": "negative",
  "response": "Thank you for reaching out. We will look into this shortly."
}

=== LLM Agent (Oxlo.ai) ===
{
  "urgency": "urgent",
  "products": ["Premium Plan", "API"],
  "sentiment": "negative",
  "response": "We have escalated your Premium Plan and API access issue to our billing team and will restore service within 30 minutes."
}

Wrap-up

If you are already running traditional NLP pipelines, swapping the inference step to Oxlo.ai is a low-risk way to test LLM performance without rewriting your orchestration. For a production deployment, you could hook this agent into an email webhook, or swap the model to qwen-3-32b if you need multilingual ticket support.

DEV Community