LLM vs Traditional NLP: Understanding the Differences

#learnai #oxlo #ai

We are going to build a support ticket triage agent that classifies user intent and extracts order IDs. If you run a support queue, this shows you exactly where traditional NLP ends and where an LLM from Oxlo.ai becomes worth the API call.

What you'll need

Before we start, grab the following.

Python 3.10 or newer
pip install openai spacy scikit-learn
python -m spacy download en_core_web_sm
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Traditional NLP for intent and entity extraction

Traditional NLP is fast, deterministic, and cheap for narrow tasks. We will use regex to grab order IDs and a keyword map to classify intent, which works great when users stay on script.

import re
import spacy

nlp = spacy.load("en_core_web_sm")

KEYWORDS = {
    "refund": ["refund", "money back", "chargeback"],
    "shipping": ["shipping", "delivery", "package", "track"],
    "account": ["password", "login", "reset", "locked"],
}

def traditional_parse(ticket: str):
    ticket_lower = ticket.lower()
    intent = "general"
    for label, words in KEYWORDS.items():
        if any(w in ticket_lower for w in words):
            intent = label
            break
    order_ids = re.findall(r"#(\d{5,})", ticket)
    return {"intent": intent, "order_ids": order_ids, "method": "traditional"}

print(traditional_parse("I want a refund for order #98234. It never arrived."))

Step 2: LLM classification with Oxlo.ai

Keywords fail when users get creative. An LLM understands context and synonyms without us hard-coding every variation. We will send ambiguous tickets to Oxlo.ai using the OpenAI SDK, which is a drop-in replacement.

Here is the system prompt:

SYSTEM_PROMPT = """You are a support ticket parser. Extract the following from the user's message:
- intent: one of [refund, shipping, account, general]
- order_ids: list of order ID numbers mentioned
- summary: a one-sentence summary of the issue

Return ONLY a JSON object with keys: intent, order_ids, summary. No markdown, no explanation."""

And the client call:

from openai import OpenAI
import json

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def llm_parse(ticket: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": ticket},
        ],
        temperature=0.1,
    )
    content = response.choices[0].message.content
    return json.loads(content)

print(llm_parse("Hey, my package from Acme Corp is MIA. Order #55432. Can I get my money back?"))

Step 3: Build the hybrid fallback agent

In production I do not burn an API call on every ticket. We will run the regex and keyword matcher first, then fall back to the LLM only when the traditional pipeline comes up empty or unsure. This keeps latency low and cost predictable on Oxlo.ai's request-based pricing.

import re
import json
import spacy
from openai import OpenAI

nlp = spacy.load("en_core_web_sm")
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

KEYWORDS = {
    "refund": ["refund", "money back", "chargeback"],
    "shipping": ["shipping", "delivery", "package", "track"],
    "account": ["password", "login", "reset", "locked"],
}

SYSTEM_PROMPT = """You are a support ticket parser. Extract the following from the user's message:
- intent: one of [refund, shipping, account, general]
- order_ids: list of order ID numbers mentioned
- summary: a one-sentence summary of the issue

Return ONLY a JSON object with keys: intent, order_ids, summary. No markdown, no explanation."""

def traditional_parse(ticket: str):
    ticket_lower = ticket.lower()
    intent = "general"
    for label, words in KEYWORDS.items():
        if any(w in ticket_lower for w in words):
            intent = label
            break
    order_ids = re.findall(r"#(\d{5,})", ticket)
    return {"intent": intent, "order_ids": order_ids, "method": "traditional"}

def llm_parse(ticket: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": ticket},
        ],
        temperature=0.1,
    )
    content = response.choices[0].message.content
    return json.loads(content)

def hybrid_agent(ticket: str):
    result = traditional_parse(ticket)
    if result["intent"] == "general" or len(result["order_ids"]) == 0:
        try:
            llm_result = llm_parse(ticket)
            return {
                "intent": llm_result.get("intent", "general"),
                "order_ids": llm_result.get("order_ids", []),
                "summary": llm_result.get("summary", ""),
                "method": "llm",
            }
        except Exception as e:
            result["error"] = str(e)
            return result
    return result

Run it

Save the full script as agent.py and run it. Here is the test harness I used to exercise both the fast path and the LLM fallback:

tickets = [
    "I want a refund for order #98234. It never arrived.",
    "Hey, my package from Acme Corp is MIA. Order #55432. Can I get my money back?",
    "I forgot my password and cannot log in.",
    "This is the worst experience I have ever had with your company.",
]

for t in tickets:
    print(hybrid_agent(t))

When I ran this against Oxlo.ai, the output looked like this:

{'intent': 'refund', 'order_ids': ['98234'], 'method': 'traditional'}
{'intent': 'refund', 'order_ids': ['55432'], 'summary': 'User wants a refund because their package from Acme Corp has not arrived.', 'method': 'llm'}
{'intent': 'account', 'order_ids': [], 'method': 'traditional'}
{'intent': 'general', 'order_ids': [], 'summary': 'User is expressing dissatisfaction with their experience.', 'method': 'llm'}

Next steps

Wire this agent into a Slack webhook so it tags channels automatically, or add a SQLite cache so repeated tickets never hit the API twice. If you are processing high volumes of long-context tickets, Oxlo.ai's flat per-request pricing removes the token-counting guesswork you get elsewhere. Check the details at https://oxlo.ai/pricing.

DEV Community