We are going to build a support ticket classifier that analyzes customer sentiment and assigns a category in a single LLM call. This helps support teams route issues and prioritize escalations without maintaining separate NLP pipelines. I am using Oxlo.ai because its request-based pricing keeps the cost flat even when I pass long customer messages with full thread history.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Set up the client
First, I verify that the SDK can reach Oxlo.ai. I point the OpenAI client at the Oxlo.ai base URL and make a quick health check call to Llama 3.3 70B.
from openai import OpenAI
import os
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "user", "content": "Reply with OK if you are online."}
],
max_tokens=10
)
print(response.choices[0].message.content)Step 2: Write the system prompt
I keep the labels in a closed set so the model cannot hallucinate categories. The prompt asks for a JSON object with sentiment, category, confidence, and a short reasoning string.
SYSTEM_PROMPT = """You are a support ticket classifier.
Analyze the customer message and return a JSON object with exactly these keys:
- sentiment: one of [positive, neutral, negative]
- category: one of [billing, technical, feature_request, account]
- confidence: a float between 0.0 and 1.0 representing your certainty
- reasoning: one sentence explaining why you chose these labels
Do not include any text outside the JSON object."""Step 3: Build the classifier function
I use JSON mode to force valid output, then parse the result with the standard library. I also add a small wrapper that returns a clean dictionary or raises an error.
import json
def classify_text(text: str) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": text},
],
response_format={"type": "json_object"},
)
raw = response.choices[0].message.content
return json.loads(raw)Step 4: Process a batch of tickets
Now I run the classifier over a realistic set of tickets. I collect the results in a list so I can inspect them or dump them to a database later.
tickets = [
"I was charged twice this month and I need a refund immediately.",
"How do I reset my password? The link is not working.",
"Love the new dashboard. Great work.",
"Your API returned a 500 error for the last hour and it is blocking our deploy.",
]
results = []
for ticket in tickets:
try:
label = classify_text(ticket)
results.append({"text": ticket, "label": label})
except Exception as exc:
results.append({"text": ticket, "error": str(exc)})
for row in results:
print(row)Step 5: Add confidence thresholds
Not every prediction should be trusted automatically. I flag anything below 0.8 for human review so the team can focus on edge cases.
flagged = []
for row in results:
if "error" in row:
continue
confidence = row["label"].get("confidence", 1.0)
if confidence < 0.8:
row["flag"] = "human_review"
flagged.append(row)
print(f"Auto-approved: {len(results) - len(flagged)}")
print(f"Flagged for review: {len(flagged)}")
for row in flagged:
print(" -", row["text"])Run it
Here is the complete script. When I execute it, the output looks like this.
from openai import OpenAI
import os, json
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)
SYSTEM_PROMPT = """You are a support ticket classifier.
Analyze the customer message and return a JSON object with exactly these keys:
- sentiment: one of [positive, neutral, negative]
- category: one of [billing, technical, feature_request, account]
- confidence: a float between 0.0 and 1.0 representing your certainty
- reasoning: one sentence explaining why you chose these labels
Do not include any text outside the JSON object."""
def classify_text(text: str) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": text},
],
response_format={"type": "json_object"},
)
return json.loads(response.choices[0].message.content)
tickets = [
"I was charged twice this month and I need a refund immediately.",
"How do I reset my password? The link is not working.",
"Love the new dashboard. Great work.",
"Your API returned a 500 error for the last hour and it is blocking our deploy.",
]
results = []
for ticket in tickets:
try:
results.append({"text": ticket, "label": classify_text(ticket)})
except Exception as exc:
results.append({"text": ticket, "error": str(exc)})
flagged = []
for row in results:
if "error" not in row and row["label"].get("confidence", 1.0) < 0.8:
row["flag"] = "human_review"
flagged.append(row)
for row in results:
print(row)
print(f"Auto-approved: {len(results) - len(flagged)}")
print(f"Flagged for review: {len(flagged)}")Example output:
{'text': 'I was charged twice this month and I need a refund immediately.', 'label': {'sentiment': 'negative', 'category': 'billing', 'confidence': 0.95, 'reasoning': 'The customer is reporting a duplicate charge and demanding a refund.'}}
{'text': 'How do I reset my password? The link is not working.', 'label': {'sentiment': 'neutral', 'category': 'technical', 'confidence': 0.91, 'reasoning': 'The customer is asking for help with a broken password reset link.'}}
{'text': 'Love the new dashboard. Great work.', 'label': {'sentiment': 'positive', 'category': 'feature_request', 'confidence': 0.88, 'reasoning': 'The customer is praising a recent UI update.'}}
{'text': 'Your API returned a 500 error for the last hour and it is blocking our deploy.', 'label': {'sentiment': 'negative', 'category': 'technical', 'confidence': 0.93, 'reasoning': 'The customer is reporting a service outage that is impacting their deployment pipeline.'}}
Auto-approved: 4
Flagged for review: 0Next steps
Wire this classifier into a webhook so new tickets are labeled as they arrive. If you accumulate a few thousand verified labels, you can later train a smaller specialist model on Oxlo.ai to drive latency down even further.
Top comments (0)