We are going to build a support ticket triage agent that runs a traditional NLP baseline side by side with an LLM parser. The goal is to see exactly where regex and keyword matching break down, and where a model like Llama 3.3 70B on Oxlo.ai takes over. If you process unstructured customer text, you will walk away with a concrete hybrid you can ship today.
What you'll need
Python 3.10+, the OpenAI SDK, and an Oxlo.ai API key from https://portal.oxlo.ai. Oxlo.ai uses flat per-request pricing, so a ticket with ten pages of conversation history costs the same as a one-liner. Install the SDK with pip install openai.
Step 1: Build the traditional NLP baseline
I always start with the cheapest possible solution. A regex and keyword pipeline is fast and deterministic, but it misses nuance. Here is the baseline we will beat.
import re
import json
URGENCY_RE = re.compile(
r'\b(urgent|asap|immediately|down|broken|outage|frustrated)\b',
re.IGNORECASE,
)
CATEGORY_MAP = {
"billing": re.compile(
r'\b(invoice|payment|charged|refund|subscription|renewal)\b',
re.IGNORECASE,
),
"api": re.compile(
r'\b(endpoint|api|rate.limit|timeout|401|403|500|error)\b',
re.IGNORECASE,
),
"account": re.compile(
r'\b(login|password|2fa|mfa|locked|signin)\b',
re.IGNORECASE,
),
}
def traditional_parse(text: str):
urgency = "high" if URGENCY_RE.search(text) else "low"
category = "general"
for cat, pattern in CATEGORY_MAP.items():
if pattern.search(text):
category = cat
break
return {
"parser": "traditional",
"urgency": urgency,
"category": category,
"sentiment": None,
"summary": None,
}Step 2: Define the LLM system prompt
The LLM needs to return structured data we can compare against the baseline. I keep the prompt strict and ask for raw JSON only.
SYSTEM_PROMPT = """You are a support ticket triage parser.
Analyze the customer message and extract these fields:
- urgency: "high", "medium", or "low"
- category: one of "billing", "api", "account", "general"
- sentiment: "frustrated", "neutral", or "satisfied"
- summary: one sentence describing the core issue
Respond with ONLY a JSON object containing these keys. No markdown, no explanation."""Step 3: Wire up the Oxlo.ai LLM parser
Now we add the LLM. I am running Llama 3.3 70B through Oxlo.ai because it follows instructions well, and Oxlo.ai's flat per-request pricing keeps costs predictable even when tickets contain long conversation history. You can swap in Qwen 3 32B or Kimi K2.6 later if you need multilingual reasoning or vision.
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY",
)
def llm_parse(text: str):
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": text},
],
)
raw = response.choices[0].message.content.strip()
# Strip accidental markdown fences
if raw.startswith("
```"):
raw = raw.split("\n", 1)[1].rsplit("```
", 1)[0].strip()
return json.loads(raw)Step 4: Assemble the hybrid triage agent
The final agent runs both parsers. When the traditional pipeline and the LLM disagree, we trust the LLM. In production, you could skip the LLM call entirely when the regex match is unambiguous to save money, but for this demo we will run both so you can see the gap.
def triage_agent(ticket: str):
trad = traditional_parse(ticket)
llm = llm_parse(ticket)
# Trust the LLM when the baseline misses implicit signals
if trad["urgency"] != llm["urgency"] or trad["category"] != llm["category"]:
final = dict(llm)
final["parser"] = "llm (override)"
else:
final = dict(trad)
final["sentiment"] = llm.get("sentiment")
final["summary"] = llm.get("summary")
return {
"ticket": ticket,
"traditional": trad,
"llm": llm,
"final": final,
}Run it
Here is the driver I used to test against four real-world-style tickets. Three expose flaws in the regex baseline.
if __name__ == "__main__":
tickets = [
"I was charged twice this month and I need a refund immediately.",
"The /v1/batch endpoint is returning 500s for the last hour.",
"Hey, just wanted to say the new dashboard is slick. Great work!",
"I cannot log in and it is really frustrating. Help?",
]
for t in tickets:
out = triage_agent(t)
print(json.dumps(out, indent=2))
print("-" * 40)Example output:
{
"ticket": "The /v1/batch endpoint is returning 500s for the last hour.",
"traditional": {
"parser": "traditional",
"urgency": "low",
"category": "api",
"sentiment": null,
"summary": null
},
"llm": {
"urgency": "high",
"category": "api",
"sentiment": "frustrated",
"summary": "Batch API endpoint returning 500 errors for the past hour."
},
"final": {
"urgency": "high",
"category": "api",
"sentiment": "frustrated",
"summary": "Batch API endpoint returning 500 errors for the past hour.",
"parser": "llm (override)"
}
}
----------------------------------------
{
"ticket": "Hey, just wanted to say the new dashboard is slick. Great work!",
"traditional": {
"parser": "traditional",
"urgency": "low",
"category": "general",
"sentiment": null,
"summary": null
},
"llm": {
"urgency": "low",
"category": "general",
"sentiment": "satisfied",
"summary": "Positive feedback about the new dashboard."
},
"final": {
"parser": "traditional",
"urgency": "low",
"category": "general",
"sentiment": "satisfied",
"summary": "Positive feedback about the new dashboard."
}
}
----------------------------------------
Wrap-up and next steps
The regex baseline is fast and free, but it fails on implicit urgency and cannot produce sentiment or summaries. The LLM on Oxlo.ai fills those gaps with a single flat-cost request, so a ten-message thread costs the same as a single sentence. Two concrete next steps: wire this agent into your email or Slack ingestion pipeline, and experiment with DeepSeek V3.2 on Oxlo.ai for the same task if you want a lighter model that is currently on the free tier. See https://oxlo.ai/pricing for plan details.
Top comments (0)