LLM vs Traditional NLP Techniques: A Comprehensive Guide

#learnai #oxlo #ai

Last week I rewrote an old support ticket triage script. The original pipeline used classic NLP techniques. The new version keeps the fast local logic for obvious cases and routes ambiguous, long, or nuanced tickets to an LLM. In this guide, I will show you exactly where traditional NLP ends and where an LLM starts to pay off by building a hybrid support ticket agent.

What you'll need

You need Python 3.10 or newer, an Oxlo.ai API key from https://portal.oxlo.ai, and a few Python packages.

pip install openai spacy scikit-learn
python -m spacy download en_core_web_sm

Grab your API key from the Oxlo.ai portal and export it.

export OXLO_API_KEY="oxlo_..."

Step 1: Scaffold the traditional NLP layer

Traditional NLP is still unbeatable for speed and cost on narrow tasks. We will use spaCy to extract named entities and a lightweight keyword scanner to detect urgency. This runs entirely on your machine.

import spacy
import re

nlp = spacy.load("en_core_web_sm")

URGENCY_PATTERNS = re.compile(
    r"\b(urgent|asap|immediately|down|outage|critical|broken)\b",
    re.IGNORECASE
)

def traditional_nlp(ticket: str):
    doc = nlp(ticket)
    entities = [
        {"text": ent.text, "label": ent.label_}
        for ent in doc.ents
    ]
    urgency_score = len(URGENCY_PATTERNS.findall(ticket))
    is_urgent = urgency_score >= 2
    return {
        "entities": entities,
        "urgency_score": urgency_score,
        "is_urgent": is_urgent,
        "category_guess": "unknown"
    }

Step 2: Add the LLM reasoning layer with Oxlo.ai

When a ticket is long, sarcastic, or mixes multiple issues, regex and small models fall over. That is where the LLM takes over. We will use Oxlo.ai because the flat per-request pricing means a 2,000 word customer rant costs the same as a one-liner. Token-based billing would penalize us for that context.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def llm_classify(ticket: str, system_prompt: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": ticket},
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

Step 3: Define the system prompt

The system prompt is the contract between our code and the model. It asks for strict JSON so we can parse it deterministically.

SYSTEM_PROMPT = """You are a support ticket triage agent.
Analyze the ticket and return a single JSON object with these exact keys:
- category: one of [billing, technical, account_access, feature_request, complaint]
- urgency: one of [low, medium, high, critical]
- sentiment: one of [angry, frustrated, neutral, happy]
- entities: array of objects with keys "name" and "type" (company, product, date, person)
- draft_reply: a short, professional response under 50 words
- reasoning: one sentence explaining why you chose the category

Rules:
- Do not output markdown fences.
- Output only the raw JSON object."""

Step 4: Build the hybrid router

Now we wire both layers together. If the traditional layer finds multiple entities and high urgency, we trust it for routing. If the ticket is ambiguous, we call Oxlo.ai.

import json

def triage_ticket(ticket: str):
    # Fast path: traditional NLP
    fast = traditional_nlp(ticket)

    # Heuristic: if we see mixed signals or no clear urgency, use the LLM
    needs_llm = (
        fast["category_guess"] == "unknown"
        or len(fast["entities"]) > 3
        or fast["urgency_score"] == 0
        or len(ticket.split()) > 100
    )

    if not needs_llm:
        return {
            "source": "traditional_nlp",
            "entities": fast["entities"],
            "urgency": "high" if fast["is_urgent"] else "low",
            "category": "technical"
        }

    # Slow path: LLM reasoning
    raw = llm_classify(ticket, SYSTEM_PROMPT)
    parsed = json.loads(raw)
    parsed["source"] = "oxlo_llm"
    return parsed

if __name__ == "__main__":
    # Example 1: short, obvious outage
    short_ticket = "Server is down. Critical outage. ASAP."
    print("=== Short ticket ===")
    print(json.dumps(triage_ticket(short_ticket), indent=2))

    # Example 2: long, ambiguous complaint
    long_ticket = (
        "I have been trying to export my data for three weeks now. "
        "Your billing page charged me twice in January but the invoice shows once. "
        "Also, the new dashboard is slow and I cannot invite my colleague who started last Monday. "
        "This is affecting our Q2 reporting. Please fix all of this immediately."
    )
    print("\n=== Long ticket ===")
    print(json.dumps(triage_ticket(long_ticket), indent=2))

Run it

Save the full script as triage.py and run it.

python triage.py

You should see two blocks of JSON. The short ticket returns instantly from the traditional layer. The long ticket routes to Oxlo.ai and returns structured reasoning, a category, and a draft reply.

=== Short ticket ===
{
  "source": "traditional_nlp",
  "entities": [],
  "urgency": "high",
  "category": "technical"
}

=== Long ticket ===
{
  "source": "oxlo_llm",
  "category": "billing",
  "urgency": "high",
  "sentiment": "frustrated",
  "entities": [
    {"name": "January", "type": "date"},
    {"name": "dashboard", "type": "product"}
  ],
  "draft_reply": "We are sorry for the trouble. I have escalated the billing and performance issues to our team and will update you within the hour.",
  "reasoning": "The ticket mixes a billing dispute with a performance complaint, but the double charge is the most actionable issue."
}

Wrap-up

This hybrid pattern keeps costs near zero for high-volume, simple tickets while still giving you LLM-level reasoning on the hard ones. Because Oxlo.ai charges a flat rate per request, you can stuff the entire conversation history into the LLM call without watching token meters spin.

Two concrete next steps. First, add a confidence threshold so the LLM can delegate back to a human when sentiment is angry and urgency is critical. Second, swap the model to deepseek-r1-671b or qwen-3-32b on Oxlo.ai when you need deeper reasoning for multi-step troubleshooting tickets. You can browse the full model catalog and flat pricing at https://oxlo.ai/pricing.

DEV Community