LLM vs Traditional Rule-Based Systems for NLP

#learnai #oxlo #ai

We are building a support ticket classifier that routes unstructured customer messages to the correct department and priority queue. I will start with a traditional rule-based baseline in pure Python, then replace it with a single LLM call through Oxlo.ai so you can see exactly where brittle regex falls over and a model keeps working. Because Oxlo.ai uses flat per-request pricing, you can pass long log dumps or email threads without your cost scaling with input length. See https://oxlo.ai/pricing for plan details.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Define the rule-based baseline

I will begin with the kind of keyword and regex script most teams accumulate over time. It looks reasonable until users phrase things creatively.

import re

class RuleClassifier:
    def __init__(self):
        self.urgency_keywords = ["urgent", "asap", "immediately", "down", "broken"]
        self.billing_keywords = ["invoice", "payment", "refund", "charged"]
        self.technical_keywords = ["bug", "error", "crash", "login", "api"]
    
    def classify(self, text: str):
        text_lower = text.lower()
        urgency = any(k in text_lower for k in self.urgency_keywords)
        department = "general"
        if any(k in text_lower for k in self.billing_keywords):
            department = "billing"
        elif any(k in text_lower for k in self.technical_keywords):
            department = "technical"
        return {
            "department": department,
            "urgency": "high" if urgency else "low",
            "confidence": "medium"
        }

classifier = RuleClassifier()

Step 2: Design the LLM classifier system prompt

Instead of maintaining growing keyword lists, I will give a model concise instructions and ask for strict JSON. This prompt is the only rules file you will need.

SYSTEM_PROMPT = """You are a support ticket classifier. Analyze the user message and return a JSON object with exactly these keys:
- department: one of billing, technical, sales, general
- urgency: one of low, medium, high
- reason: a one-sentence explanation

Respond with only the JSON object, no markdown fences."""

Step 3: Wire up the Oxlo.ai client

Now I will swap the regex engine for an LLM call through Oxlo.ai. The client setup is identical to the OpenAI SDK because Oxlo.ai exposes a fully compatible base URL.

import json
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

def llm_classify(user_message: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    raw = response.choices[0].message.content.strip()
    return json.loads(raw)

Step 4: Build a comparison harness

To make the difference concrete, I will run both classifiers against a batch of edge cases that I have seen break keyword systems in production.

test_messages = [
    "I was charged twice last Tuesday and now my dashboard is blank.",
    "The API returns a 500 whenever I send a request with Unicode characters.",
    "We are evaluating your enterprise plan for a 500-seat rollout next quarter.",
    "Everything is broken and I need this fixed yesterday.",
    "My invoice says $0 but my card was billed $299.",
]

for msg in test_messages:
    rule_result = classifier.classify(msg)
    try:
        llm_result = llm_classify(msg)
    except Exception as e:
        llm_result = {"department": "error", "urgency": "unknown", "reason": str(e)}
    
    print(f"Message: {msg}")
    print(f"  Rule-based -> dept: {rule_result['department']}, urgency: {rule_result['urgency']}")
    print(f"  LLM        -> dept: {llm_result.get('department')}, urgency: {llm_result.get('urgency')}, reason: {llm_result.get('reason')}")
    print()

Run it

Export your key and run the script.

export OXLO_API_KEY="sk-oxlo.ai-..."
python classifier_demo.py

Typical output looks like this. Notice how the rule-based system either misroutes ambiguous messages or defaults to general, while the Oxlo.ai LLM call parses intent correctly.

Message: I was charged twice last Tuesday and now my dashboard is blank.
  Rule-based -> dept: billing, urgency: low
  LLM        -> dept: billing, urgency: high, reason: Duplicate charge and critical UI failure reported

Message: The API returns a 500 whenever I send a request with Unicode characters.
  Rule-based -> dept: technical, urgency: low
  LLM        -> dept: technical, urgency: high, reason: Service error blocking API usage

Message: We are evaluating your enterprise plan for a 500-seat rollout next quarter.
  Rule-based -> dept: general, urgency: low
  LLM        -> dept: sales, urgency: medium, reason: Enterprise evaluation and expansion inquiry

Message: Everything is broken and I need this fixed yesterday.
  Rule-based -> dept: technical, urgency: high
  LLM        -> dept: technical, urgency: high, reason: Total system outage with immediate deadline

Message: My invoice says $0 but my card was billed $299.
  Rule-based -> dept: billing, urgency: low
  LLM        -> dept: billing, urgency: medium, reason: Discrepancy between stated and actual charge

Next steps

Try batch-processing your historical ticket backlog through Oxlo.ai to audit how often the old keyword system misrouted messages. You can also extend the same script into an agent by adding function calling so the model returns structured JSON and then immediately opens a Jira issue or Slack alert.

DEV Community