LLM vs Traditional Rule-Based Systems for NLP: A Comparative Analysis

We are going to build a support ticket triage agent that reads customer messages, classifies intent, and flags urgency. I will build the same logic twice: once with traditional regex and keyword rules, then with an LLM via Oxlo.ai. The comparison shows exactly where each approach breaks and why I switched to the LLM for production.

What you'll need

You need Python 3.10 or newer, the OpenAI SDK, and an Oxlo.ai API key from https://portal.oxlo.ai. Install the SDK with pip.

pip install openai

Step 1: Define the task and test cases

I started by writing five realistic support messages that mix intents and tones. These will be the benchmark for both systems.

TICKETS = [
    "I was charged twice last month and I need my money back immediately.",
    "The dashboard is completely blank after I login, can someone help?",
    "Hey, I think I want a refund? Not sure if I like this.",
    "URGENT: our API integration is down and customers are complaining.",
    "I would like to cancel my subscription because I found a better tool.",
]

Step 2: Build a rule-based intent extractor

The classic approach uses regex patterns for intent and a keyword bag for urgency. It is fast and deterministic, but every new pattern requires a deploy.

import re

INTENT_PATTERNS = {
    "refund": re.compile(r"\brefund\b|\bmoney back\b|\bcharged twice\b"),
    "bug": re.compile(r"\bblank\b|\bdown\b|\bbug\b|\berror\b"),
    "billing": re.compile(r"\bcharge\b|\bsubscription\b|\bcancel\b"),
}

URGENCY_KEYWORDS = ["urgent", "immediately", "down", "asap", "emergency"]

def classify_rule_based(text: str):
    text_lower = text.lower()
    intents = [name for name, pat in INTENT_PATTERNS.items() if pat.search(text_lower)]
    urgency = any(word in text_lower for word in URGENCY_KEYWORDS)
    return {
        "intents": intents if intents else ["unknown"],
        "urgency": "high" if urgency else "low",
    }

Step 3: Test the rule-based engine

I ran the five tickets through the classifier. It caught the obvious keywords, but it missed implied urgency in the blank dashboard and miscategorized the cancellation as billing instead of a refund request.

for t in TICKETS:
    print(t[:50], "...", classify_rule_based(t))

# Output:
# I was charged twice last month and I need my ... {'intents': ['refund'], 'urgency': 'low'}
# The dashboard is completely blank after I logi ... {'intents': ['bug'], 'urgency': 'low'}
# Hey, I think I want a refund? Not sure if I li ... {'intents': ['refund'], 'urgency': 'low'}
# URGENT: our API integration is down and custome ... {'intents': ['bug'], 'urgency': 'high'}
# I would like to cancel my subscription because ... {'intents': ['billing'], 'urgency': 'low'}

Step 4: Build the LLM classifier on Oxlo.ai

Next I swapped the regex engine for a single call to Llama 3.3 70B on Oxlo.ai. I used JSON mode so the response is machine readable without fragile parsing. The system prompt acts as the new rulebook.

SYSTEM_PROMPT = """You are a support triage agent.
Analyze the user message and return a JSON object with exactly these keys:
- intents: list of relevant intents from [refund, bug, billing, sales]
- urgency: either low, medium, or high
- reasoning: one sentence explaining why

Rules:
- If the user mentions wanting money back or being charged incorrectly, include refund.
- If the user describes broken functionality, include bug.
- If the user mentions subscriptions, cancellations, or invoices, include billing.
- If the user is comparing tools or asking about features, include sales.
- urgency is high if the problem blocks work or the user uses strong frustration."""

from openai import OpenAI
import json

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def classify_llm(text: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
    )
    return json.loads(response.choices[0].message.content)

Step 5: Compare rule-based and LLM outputs

I fed the same five tickets to the LLM. It correctly flagged the blank dashboard as high urgency because it inferred business impact, and it tagged the cancellation message with both billing and refund intents. The rule-based system never would have caught the overlap without a major rewrite.

for t in TICKETS:
    print(classify_llm(t))

# Example output:
# {'intents': ['refund'], 'urgency': 'high', 'reasoning': 'User was charged twice and demands money back immediately.'}
# {'intents': ['bug'], 'urgency': 'high', 'reasoning': 'Dashboard is completely blank after login, blocking access.'}
# {'intents': ['refund'], 'urgency': 'low', 'reasoning': 'User is uncertain and asking about a refund.'}
# {'intents': ['bug'], 'urgency': 'high', 'reasoning': 'API integration is down and affecting customers.'}
# {'intents': ['billing', 'refund'], 'urgency': 'low', 'reasoning': 'User wants to cancel subscription due to finding a better tool, implying refund request.'}

Run it

Here is the full script. Save it as triage.py, export your Oxlo.ai key, and run it.

import os
import json
import re
from openai import OpenAI

TICKETS = [
    "I was charged twice last month and I need my money back immediately.",
    "The dashboard is completely blank after I login, can someone help?",
    "Hey, I think I want a refund? Not sure if I like this.",
    "URGENT: our API integration is down and customers are complaining.",
    "I would like to cancel my subscription because I found a better tool.",
]

INTENT_PATTERNS = {
    "refund": re.compile(r"\brefund\b|\bmoney back\b|\bcharged twice\b"),
    "bug": re.compile(r"\bblank\b|\bdown\b|\bbug\b|\berror\b"),
    "billing": re.compile(r"\bcharge\b|\bsubscription\b|\bcancel\b"),
}

URGENCY_KEYWORDS = ["urgent", "immediately", "down", "asap", "emergency"]

def classify_rule_based(text: str):
    text_lower = text.lower()
    intents = [name for name, pat in INTENT_PATTERNS.items() if pat.search(text_lower)]
    urgency = any(word in text_lower for word in URGENCY_KEYWORDS)
    return {
        "intents": intents if intents else ["unknown"],
        "urgency": "high" if urgency else "low",
    }

SYSTEM_PROMPT = """You are a support triage agent.
Analyze the user message and return a JSON object with exactly these keys:
- intents: list of relevant intents from [refund, bug, billing, sales]
- urgency: either low, medium, or high
- reasoning: one sentence explaining why

Rules:
- If the user mentions wanting money back or being charged incorrectly, include refund.
- If the user describes broken functionality, include bug.
- If the user mentions subscriptions, cancellations, or invoices, include billing.
- If the user is comparing tools or asking about features, include sales.
- urgency is high if the problem blocks work or the user uses strong frustration."""

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key=os.environ.get("OXLO_API_KEY"))

def classify_llm(text: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
    )
    return json.loads(response.choices[0].message.content)

if __name__ == "__main__":
    print("=== Rule-based results ===")
    for t in TICKETS:
        print(classify_rule_based(t))

    print("\n=== LLM results via Oxlo.ai ===")
    for t in TICKETS:
        print(classify_llm(t))

Export your key and execute.

export OXLO_API_KEY="sk-..."
python triage.py

Wrap-up and next steps

The rule-based system is fine for a frozen vocabulary, but in practice customer language drifts constantly. The LLM version on Oxlo.ai handles that drift without redeploying code, and the flat per-request pricing keeps costs predictable even when the prompt grows. See https://oxlo.ai/pricing for details.

As a next step, wire the classifier to Oxlo.ai function calling to create a Jira ticket automatically, or swap in qwen-3-32b if you need to triage messages in multiple languages.